You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/06/16 15:57:56 UTC
[PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
the-other-tim-brown opened a new pull request, #13444:
URL: https://github.com/apache/hudi/pull/13444
### Change Logs
- Update spark reader to use HoodieFileGroupReader
- Add logic for easily checking if a row represents a delete operation to avoid converting to and from avro multiple times in the CDC spark reader
- TODO: compaction support
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2994549651
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d73bf8a5fd21d41831c3fe85f54133b30c0c97c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240)
* b4765033b87250ef5654d2d71818ae398fa7e91f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3091373794
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
}, {
"hash" : "73e6ef97620b31d888b37d02733f12937807d513",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851",
"triggerID" : "73e6ef97620b31d888b37d02733f12937807d513",
"triggerType" : "PUSH"
}, {
"hash" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 73e6ef97620b31d888b37d02733f12937807d513 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851)
* c9dc3001cfb9e1e542e3e12062f018f6301b8f04 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2196398503
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/compact/CompactOperator.java:
##########
@@ -163,34 +161,23 @@ private void doCompaction(String instantTime,
// schema evolution
CompactionUtil.setAvroSchema(writeConfig, metaClient);
List<WriteStatus> writeStatuses = compactor.compact(
- flinkTable,
writeConfig,
compactionOperation,
instantTime,
flinkTable.getTaskContextSupplier(),
- readerContextOpt,
+ readerContext,
flinkTable);
compactionMetrics.endCompaction();
collector.collect(new CompactionCommitEvent(instantTime, compactionOperation.getFileId(), writeStatuses, taskID));
}
- private Option<HoodieReaderContext<?>> initReaderContext(HoodieFlinkWriteClient<?> writeClient) {
+ private HoodieReaderContext<?> initReaderContext(HoodieFlinkWriteClient<?> writeClient) {
HoodieTableMetaClient metaClient = flinkTable.getMetaClient();
- boolean useFileGroupReaderBasedCompaction = !metaClient.isMetadataTable()
- && writeClient.getConfig().getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED)
- && writeClient.getConfig().populateMetaFields() // Virtual key support by fg reader is not ready
- && !(metaClient.getTableConfig().isCDCEnabled() && writeClient.getConfig().isYieldingPureLogForMor()); // do not support produce cdc log during fg reader
- if (useFileGroupReaderBasedCompaction) {
- // CAUTION: reuse the meta client so that the timeline is updated
- Supplier<InternalSchemaManager> internalSchemaManager = () -> InternalSchemaManager.get(conf, metaClient);
- // initialize storage conf lazily.
- StorageConfiguration<?> readerConf = writeClient.getEngineContext().getStorageConf();
- return Option.of(new FlinkRowDataReaderContext(readerConf, internalSchemaManager, Collections.emptyList(), metaClient.getTableConfig(), Option.empty()));
- } else {
- // always using avro record merger for legacy compaction since log scanner do not support rowdata reading yet.
- writeClient.getConfig().setRecordMergerClass(HoodieAvroRecordMerger.class.getName());
- }
- return Option.empty();
+ // CAUTION: reuse the meta client so that the timeline is updated
+ Supplier<InternalSchemaManager> internalSchemaManager = () -> InternalSchemaManager.get(conf, metaClient);
+ // initialize storage conf lazily.
Review Comment:
The condition `writeClient.getConfig().getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED)` has been ignored?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2203722193
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/compact/HoodieFlinkMergeOnReadTableCompactor.java:
##########
@@ -69,35 +67,22 @@ public void preCompact(
}
}
- public List<WriteStatus> compact(HoodieCompactionHandler compactionHandler,
- HoodieWriteConfig writeConfig,
+ public List<WriteStatus> compact(HoodieWriteConfig writeConfig,
CompactionOperation operation,
String instantTime,
TaskContextSupplier taskContextSupplier,
- Option<HoodieReaderContext<?>> readerContextOpt,
+ HoodieReaderContext<?> readerContext,
HoodieTable table) throws IOException {
String maxInstantTime = getMaxInstantTime(table.getMetaClient());
Review Comment:
Yes, the plan is to have all reads go through the FileGroupReader as our common read path as part of the 1.1.0 release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2194406117
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
Did I ever mention any keyword about performance here? It's the maintainace and extensibility I've be addressing all the time. Let's keep the major work flow clean and clear, that's my point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2190202688
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
+ // If the record is not the same as the base record, we can emit an update
+ if (isDeleteAndRecord.getRight() != baseRecord) {
+ callbackOption.ifPresent(callback -> {
+ BufferedRecord<T> mergeResult = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
Review Comment:
Yes, it is definitely confusing. I've updated the interface to work off of avros directly for now since that is what the CDC logger requires. We can revisit making this interface simply take in the engine record type as well in the future to avoid these various wrapper objects
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3045352346
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b31c59a60518f198d3d19c734dc378b8fa2ef8b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623)
* 576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2196403849
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -349,11 +385,13 @@ protected boolean hasNextLogRecord() {
while (logRecordIterator.hasNext()) {
BufferedRecord<T> nextRecordInfo = logRecordIterator.next();
if (!nextRecordInfo.isDelete()) {
- nextRecord = nextRecordInfo.getRecord();
+ BufferedRecord<T> convertedBufferedRecord = applyOutputSchemaConversion(nextRecordInfo);
+ nextRecord = convertedBufferedRecord.getRecord();
+ handleBaseFileInsert(nextRecordInfo.getRecordKey(), nextRecord);
readStats.incrementNumInserts();
return true;
} else if (emitDelete) {
- nextRecord = readerContext.getDeleteRow(nextRecordInfo.getRecord(), nextRecordInfo.getRecordKey());
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(nextRecordInfo.getRecord(), nextRecordInfo.getRecordKey()));
Review Comment:
We just ignore the deletes from logs, is this behavior in line with before? Ususally these deletes are invalid or probably because of disorder inputs from Network shufflle, not sure if user still want to see these changes to downstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2203726664
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+
+import java.util.function.UnaryOperator;
+
+/**
+ * Interface used within the {@link HoodieFileGroupReader<T>} for processing updates to records in Merge-on-Read tables.
+ * Note that the updates are always relative to the base file's current state.
+ * @param <T> the engine specific record type
+ */
+public interface UpdateProcessor<T> {
+ /**
+ * Processes the update to the record. If the update should not be returned to the caller, the method should return null.
+ * @param recordKey the key of the record being updated
+ * @param previousRecord the previous version of the record, or null if there is no previous value
+ * @param currentRecord the current version of the record after merging with the existing record, if any exists
+ * @param isDelete a flag indicating whether the merge resulted in a delete operation
+ * @return the processed record, or null if the record should not be returned to the caller
+ */
+ T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete);
+
+ static <T> UpdateProcessor<T> create(HoodieReadStats readStats, HoodieReaderContext<T> readerContext,
+ boolean emitDeletes, Option<BaseFileUpdateCallback> updateCallback) {
+ UpdateProcessor<T> handler = new StandardUpdateProcessor<>(readStats, readerContext, emitDeletes);
+ if (updateCallback.isPresent()) {
+ return new CallbackProcessor<>(updateCallback.get(), handler, readerContext);
+ }
+ return handler;
+ }
+
+ /**
+ * A standard update processor that increments the read stats and returns the record if applicable.
+ * @param <T> the engine specific record type
+ */
+ class StandardUpdateProcessor<T> implements UpdateProcessor<T> {
+ protected final HoodieReadStats readStats;
+ protected final HoodieReaderContext<T> readerContext;
+ protected final boolean emitDeletes;
+
+ public StandardUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext,
+ boolean emitDeletes) {
+ this.readStats = readStats;
+ this.readerContext = readerContext;
+ this.emitDeletes = emitDeletes;
+ }
+
+ @Override
+ public T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete) {
+ if (isDelete) {
+ readStats.incrementNumDeletes();
+ if (emitDeletes) {
+ return readerContext.getDeleteRow(currentRecord, recordKey);
+ }
+ return null;
+ } else {
+ if (previousRecord != null) {
+ readStats.incrementNumUpdates();
+ } else {
+ readStats.incrementNumInserts();
+ }
+ return readerContext.seal(currentRecord);
+ }
+ }
+ }
+
+ /**
+ * A processor that wraps the standard update processor and invokes a customizable callback for each update.
+ * @param <T> the engine specific record type
+ */
+ class CallbackProcessor<T> implements UpdateProcessor<T> {
+ private final BaseFileUpdateCallback callback;
Review Comment:
If we want to avoid another specific interface for this callback, we can also make a more generic `TriConsumer` to take its place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3042916126
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b31c59a60518f198d3d19c734dc378b8fa2ef8b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2196392470
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/plan/generators/HoodieLogCompactionPlanGenerator.java:
##########
@@ -55,7 +54,6 @@ public HoodieLogCompactionPlanGenerator(HoodieTable table, HoodieEngineContext e
super(table, engineContext, writeConfig, executor);
this.compactionStrategy = HoodieCompactionStrategy.newBuilder()
.setStrategyParams(getStrategyParams())
- .setCompactorClassName(LogCompactionExecutionHelper.class.getName())
Review Comment:
do we also need to adjust the `HoodieCompactionStrategy` avsc file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2196395996
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -316,27 +322,57 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
+ if (baseFileUpdateCallback != null && isDeleteAndRecord.getRight() != baseRecord) {
+ // If the record is not the same as the base record, we can emit an update
+ handleBaseFileUpdate(logRecordInfo.getRecordKey(), baseRecord, nextRecord);
+ }
readStats.incrementNumUpdates();
return true;
- } else if (emitDelete) {
- // emit Deletes
- nextRecord = readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey());
- readStats.incrementNumDeletes();
- return nextRecord != null;
} else {
- // not emit Deletes
+ // emit Deletes
+ handleBaseFileDelete(logRecordInfo.getRecordKey(), baseRecord);
readStats.incrementNumDeletes();
- return false;
+ if (emitDelete) {
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey()));
+ return nextRecord != null;
+ } else {
+ return false;
+ }
}
}
// Inserts
- nextRecord = readerContext.seal(baseRecord);
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(baseRecord));
readStats.incrementNumInserts();
return true;
}
+ /**
+ * Applies the final output schema conversion to the buffered record if required. This ensures the records match the requested schema.
+ * @param bufferedRecord the buffered record to convert
+ * @return a new buffered record with the converted record and the proper schema ID set
+ */
+ protected BufferedRecord<T> applyOutputSchemaConversion(BufferedRecord<T> bufferedRecord) {
Review Comment:
I see the `applyOutputSchemaConversion` has been applied in several places with optional condition, this is error-prone, it is okey we project the record two times if the cost is low compared to the cdc logging IO: one for regular write, one for cdc logging.
And actually, there should be no projection for cdc logging path at all, because it either comes from regular COW write or MOR compaction, both does not really have explicit query projection like query side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217070397
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UnmergedFileGroupRecordBuffer.java:
##########
@@ -52,9 +51,8 @@ public UnmergedFileGroupRecordBuffer(
RecordMergeMode recordMergeMode,
PartialUpdateMode partialUpdateMode,
TypedProperties props,
- HoodieReadStats readStats,
- boolean emitDelete) {
- super(readerContext, hoodieTableMetaClient, recordMergeMode, partialUpdateMode, props, readStats, Option.empty(), emitDelete);
+ HoodieReadStats readStats) {
+ super(readerContext, hoodieTableMetaClient, recordMergeMode, partialUpdateMode, props, readStats, Option.empty(), null);
Review Comment:
Does passing `null` as the update processor work here, or `UnmergedFileGroupRecordBuffer` is not really used so we can get away for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3085365369
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 5540b78becd2ea0cfd5c13732d22ee463392663c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716)
* ff24a54cfc19c8a89fc445ba13fab96b8b4085f4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3085693939
@jonvex you plan to review this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3091453507
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
}, {
"hash" : "73e6ef97620b31d888b37d02733f12937807d513",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851",
"triggerID" : "73e6ef97620b31d888b37d02733f12937807d513",
"triggerType" : "PUSH"
}, {
"hash" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6855",
"triggerID" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"triggerType" : "PUSH"
}, {
"hash" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 73e6ef97620b31d888b37d02733f12937807d513 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851)
* c9dc3001cfb9e1e542e3e12062f018f6301b8f04 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6855)
* 05d26ce59cd2eee1864bbcca5d8f93236b3043ca UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3091454317
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
}, {
"hash" : "73e6ef97620b31d888b37d02733f12937807d513",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851",
"triggerID" : "73e6ef97620b31d888b37d02733f12937807d513",
"triggerType" : "PUSH"
}, {
"hash" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6855",
"triggerID" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"triggerType" : "PUSH"
}, {
"hash" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* c9dc3001cfb9e1e542e3e12062f018f6301b8f04 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6855)
* 05d26ce59cd2eee1864bbcca5d8f93236b3043ca UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217058909
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+
+import java.util.function.UnaryOperator;
+
+/**
+ * Interface used within the {@link HoodieFileGroupReader<T>} for processing updates to records in Merge-on-Read tables.
+ * Note that the updates are always relative to the base file's current state.
+ * @param <T> the engine specific record type
+ */
+public interface UpdateProcessor<T> {
+ /**
+ * Processes the update to the record. If the update should not be returned to the caller, the method should return null.
+ * @param recordKey the key of the record being updated
+ * @param previousRecord the previous version of the record, or null if there is no previous value
+ * @param currentRecord the current version of the record after merging with the existing record, if any exists
+ * @param isDelete a flag indicating whether the merge resulted in a delete operation
+ * @return the processed record, or null if the record should not be returned to the caller
+ */
+ T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete);
+
+ static <T> UpdateProcessor<T> create(HoodieReadStats readStats, HoodieReaderContext<T> readerContext,
+ boolean emitDeletes, Option<BaseFileUpdateCallback> updateCallback) {
Review Comment:
Based on my understanding, `updateCallback` is introduced for CDC logger, correct?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+
+import java.util.function.UnaryOperator;
+
+/**
+ * Interface used within the {@link HoodieFileGroupReader<T>} for processing updates to records in Merge-on-Read tables.
+ * Note that the updates are always relative to the base file's current state.
+ * @param <T> the engine specific record type
+ */
+public interface UpdateProcessor<T> {
+ /**
+ * Processes the update to the record. If the update should not be returned to the caller, the method should return null.
+ * @param recordKey the key of the record being updated
+ * @param previousRecord the previous version of the record, or null if there is no previous value
+ * @param currentRecord the current version of the record after merging with the existing record, if any exists
+ * @param isDelete a flag indicating whether the merge resulted in a delete operation
+ * @return the processed record, or null if the record should not be returned to the caller
+ */
+ T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete);
+
+ static <T> UpdateProcessor<T> create(HoodieReadStats readStats, HoodieReaderContext<T> readerContext,
+ boolean emitDeletes, Option<BaseFileUpdateCallback> updateCallback) {
+ UpdateProcessor<T> handler = new StandardUpdateProcessor<>(readStats, readerContext, emitDeletes);
+ if (updateCallback.isPresent()) {
+ return new CallbackProcessor<>(updateCallback.get(), handler, readerContext);
+ }
+ return handler;
+ }
+
+ /**
+ * A standard update processor that increments the read stats and returns the record if applicable.
+ * @param <T> the engine specific record type
+ */
+ class StandardUpdateProcessor<T> implements UpdateProcessor<T> {
+ protected final HoodieReadStats readStats;
+ protected final HoodieReaderContext<T> readerContext;
+ protected final boolean emitDeletes;
+
+ public StandardUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext,
+ boolean emitDeletes) {
+ this.readStats = readStats;
+ this.readerContext = readerContext;
+ this.emitDeletes = emitDeletes;
+ }
+
+ @Override
+ public T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete) {
+ if (isDelete) {
+ readStats.incrementNumDeletes();
+ if (emitDeletes) {
+ return readerContext.getDeleteRow(currentRecord, recordKey);
+ }
+ return null;
+ } else {
+ if (previousRecord != null) {
+ readStats.incrementNumUpdates();
+ } else {
+ readStats.incrementNumInserts();
+ }
+ return readerContext.seal(currentRecord);
+ }
+ }
+ }
+
+ /**
+ * A processor that wraps the standard update processor and invokes a customizable callback for each update.
+ * @param <T> the engine specific record type
+ */
+ class CallbackProcessor<T> implements UpdateProcessor<T> {
+ private final BaseFileUpdateCallback callback;
+ private final UpdateProcessor<T> delegate;
+ private final HoodieReaderContext<T> readerContext;
+ private final Option<UnaryOperator<T>> outputConverter;
+ private final Schema requestedSchema;
+
+ public CallbackProcessor(BaseFileUpdateCallback callback, UpdateProcessor<T> delegate,
+ HoodieReaderContext<T> readerContext) {
+ this.callback = callback;
+ this.delegate = delegate;
+ this.readerContext = readerContext;
+ this.outputConverter = readerContext.getSchemaHandler().getOutputConverter();
+ this.requestedSchema = readerContext.getSchemaHandler().getRequestedSchema();
+ }
+
+ @Override
+ public T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete) {
+ T result = delegate.processUpdate(recordKey, previousRecord, currentRecord, isDelete);
+
+ if (isDelete) {
+ callback.onDelete(recordKey, convertOutput(previousRecord));
+ } else if (previousRecord != null && previousRecord != currentRecord) {
+ callback.onUpdate(recordKey, convertOutput(previousRecord), convertOutput(currentRecord));
+ } else {
+ callback.onInsert(recordKey, convertOutput(currentRecord));
+ }
+ return result;
+ }
+
+ private GenericRecord convertOutput(T record) {
+ T convertedRecord = outputConverter.map(converter -> record == null ? null : converter.apply(record)).orElse(record);
+ return convertedRecord == null ? null : readerContext.convertToAvroRecord(convertedRecord, requestedSchema);
+ }
Review Comment:
Should the `convertOuput` be part of the `CDCCallback` implementation? The logic of `convertOuput` seems unrelated to the processor logic here. To achieve that, `BaseFileUpdateCallback#onUpdate, #onInsert, #onDelete` should also take instance of `T` instead of `GenericRecord`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3092308805
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
}, {
"hash" : "73e6ef97620b31d888b37d02733f12937807d513",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851",
"triggerID" : "73e6ef97620b31d888b37d02733f12937807d513",
"triggerType" : "PUSH"
}, {
"hash" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6855",
"triggerID" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"triggerType" : "PUSH"
}, {
"hash" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6856",
"triggerID" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"triggerType" : "PUSH"
}, {
"hash" : "93425d152b8586d862d0f5fa4a58f69b12628986",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6859",
"triggerID" : "93425d152b8586d862d0f5fa4a58f69b12628986",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 05d26ce59cd2eee1864bbcca5d8f93236b3043ca Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6856)
* 93425d152b8586d862d0f5fa4a58f69b12628986 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6859)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2994734533
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b4765033b87250ef5654d2d71818ae398fa7e91f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404)
* b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2185713172
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -109,18 +112,23 @@ public HoodieFileGroupReader(HoodieReaderContext<T> readerContext, HoodieStorage
long start, long length, boolean shouldUseRecordPosition) {
this(readerContext, storage, tablePath, latestCommitTime, fileSlice, dataSchema,
requestedSchema, internalSchemaOpt, hoodieTableMetaClient, props, start, length,
- shouldUseRecordPosition, false, false, false);
+ shouldUseRecordPosition, false, false, false, Option.empty());
}
private HoodieFileGroupReader(HoodieReaderContext<T> readerContext, HoodieStorage storage, String tablePath,
String latestCommitTime, FileSlice fileSlice, Schema dataSchema, Schema requestedSchema,
Option<InternalSchema> internalSchemaOpt, HoodieTableMetaClient hoodieTableMetaClient, TypedProperties props,
- long start, long length, boolean shouldUseRecordPosition, boolean allowInflightInstants, boolean emitDelete, boolean sortOutput) {
+ long start, long length, boolean shouldUseRecordPosition, boolean allowInflightInstants, boolean emitDelete, boolean sortOutput,
+ Option<FileGroupUpdateCallback<T>> updateCallback) {
this.readerContext = readerContext;
+ this.fileGroupUpdateCallback = updateCallback;
this.metaClient = hoodieTableMetaClient;
this.storage = storage;
this.hoodieBaseFileOption = fileSlice.getBaseFile();
- this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
+ this.logFiles = fileSlice.getLogFiles()
Review Comment:
don't really need to fix in this patch.
but can we add a different api to `FileSlice` only to expose `getCDClogFiles()` and so `getLogFiles()` can only return non cdc files to call callers.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
+ // If the record is not the same as the base record, we can emit an update
+ if (isDeleteAndRecord.getRight() != baseRecord) {
+ callbackOption.ifPresent(callback -> {
+ BufferedRecord<T> mergeResult = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
+ callback.onUpdate(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)), readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(mergeResult)));
+ });
+ }
readStats.incrementNumUpdates();
return true;
- } else if (emitDelete) {
- // emit Deletes
- nextRecord = readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey());
- readStats.incrementNumDeletes();
- return nextRecord != null;
} else {
- // not emit Deletes
+ // emit Deletes
+ callbackOption.ifPresent(callback -> callback.onDelete(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo))));
readStats.incrementNumDeletes();
- return false;
+ if (emitDelete) {
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey()));
Review Comment:
prior to this patch, can you help me understand where are we applying the schema conversion?
looks like w/ this patch, even if callbackOption is empty, here we are doing something extra which was not happening before.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
+ // If the record is not the same as the base record, we can emit an update
+ if (isDeleteAndRecord.getRight() != baseRecord) {
+ callbackOption.ifPresent(callback -> {
+ BufferedRecord<T> mergeResult = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
Review Comment:
orthogonal point.
sometimes, it might be confusing for devs on what is actually stored in BufferedRecord as record type. whether its HoodieRecord or is it the raw record representation. for eg, InternalRow incase of spark, GenericRecord incase of avro etc.
bcoz, many places we have named methods as `getRecord()` which might lead to confusion.
not sure if we can give it any better naming.
just a thought
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/CDCFileGroupIterator.scala:
##########
@@ -402,10 +375,8 @@ class CDCFileGroupIterator(split: HoodieCDCFileGroupSplit,
&& currentCDCFileSplit.getBeforeFileSlice.isPresent)
loadBeforeFileSliceIfNeeded(currentCDCFileSplit.getBeforeFileSlice.get)
val absLogPath = new StoragePath(basePath, currentCDCFileSplit.getCdcFiles.get(0))
- val morSplit = HoodieMergeOnReadFileSplit(None, List(new HoodieLogFile(storage.getPathInfo(absLogPath))))
- val logFileIterator = new LogFileIterator(
Review Comment:
this existing LogFileIterator which is in Iterators.scala, accounts for isDeleteOperation.
can you remind me if FileGroupReader/FileGroupRecordBuffer accounts for it as well. I did not see any explicit checks in
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -598,11 +638,13 @@ protected boolean hasNextLogRecord() {
while (logRecordIterator.hasNext()) {
BufferedRecord<T> nextRecordInfo = logRecordIterator.next();
if (!nextRecordInfo.isDelete()) {
- nextRecord = nextRecordInfo.getRecord();
+ BufferedRecord<T> convertedBufferedRecord = applyOutputSchemaConversion(nextRecordInfo);
+ nextRecord = convertedBufferedRecord.getRecord();
+ callbackOption.ifPresent(callback -> callback.onInsert(readerContext.constructHoodieRecord(convertedBufferedRecord)));
readStats.incrementNumInserts();
return true;
} else if (emitDelete) {
- nextRecord = readerContext.getDeleteRow(nextRecordInfo.getRecord(), nextRecordInfo.getRecordKey());
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(nextRecordInfo.getRecord(), nextRecordInfo.getRecordKey()));
Review Comment:
why no callback invoked here?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/CDCFileGroupIterator.scala:
##########
@@ -482,47 +449,74 @@ class CDCFileGroupIterator(split: HoodieCDCFileGroupSplit,
if (!same) {
// clear up the beforeImageRecords
beforeImageRecords.clear()
- val iter = loadFileSlice(fileSlice)
- iter.foreach { row =>
- val key = getRecordKey(row)
- // Due to the reuse buffer mechanism of Spark serialization,
- // we have to copy the serialized result if we need to retain its reference
- beforeImageRecords.put(key, serialize(row, copy = true))
+ val iter = loadFileSliceWithKeys(fileSlice)
+ iter.foreach { tuple =>
+ beforeImageRecords.put(tuple._1, tuple._2)
}
// reset beforeImageFiles
beforeImageFiles.clear()
beforeImageFiles.append(files: _*)
}
}
+ private def loadFileSliceWithKeys(fileSlice: FileSlice): Iterator[(String, InternalRow)] = {
+ val readerContext = new SparkFileFormatInternalRowReaderContext(parquetReader, Seq.empty, Seq.empty,
+ conf, metaClient.getTableConfig)
+ loadFileSlice(fileSlice, readerContext).map(internalRow => {
+ val recordKey = readerContext.getRecordKey(internalRow, avroSchema)
+ (recordKey, internalRow)
+ })
+ }
+
private def loadFileSlice(fileSlice: FileSlice): Iterator[InternalRow] = {
- val baseFileInfo = storage.getPathInfo(fileSlice.getBaseFile.get().getStoragePath)
- val basePartitionedFile = sparkPartitionedFileUtils.createPartitionedFile(
- InternalRow.empty,
- baseFileInfo.getPath,
- 0,
- baseFileInfo.getLength
- )
- val logFiles = fileSlice.getLogFiles
- .sorted(HoodieLogFile.getLogFileComparator)
- .collect(Collectors.toList[HoodieLogFile])
- .asScala.toList
- .filterNot(_.getFileName.endsWith(HoodieCDCUtils.CDC_LOGFILE_SUFFIX))
-
- if (logFiles.isEmpty) {
- // no log files, just load the base parquet file
- parquetReader(basePartitionedFile)
- } else {
- // use [[RecordMergingFileIterator]] to load both the base file and log files
- val morSplit = HoodieMergeOnReadFileSplit(Some(basePartitionedFile), logFiles)
- new RecordMergingFileIterator(
- morSplit,
- BaseFileReader(parquetReader, originTableSchema.structTypeSchema),
- originTableSchema,
- originTableSchema,
- tableState,
- conf.unwrapAs(classOf[Configuration]))
- }
+ val readerContext = new SparkFileFormatInternalRowReaderContext(parquetReader, Seq.empty, Seq.empty,
+ conf, metaClient.getTableConfig)
+ loadFileSlice(fileSlice, readerContext)
+ }
+
+ private def loadFileSlice(fileSlice: FileSlice, readerContext: SparkFileFormatInternalRowReaderContext): Iterator[InternalRow] = {
+ val fileGroupReader = HoodieFileGroupReader.newBuilder()
+ .withReaderContext(readerContext)
+ .withHoodieTableMetaClient(metaClient)
+ .withFileSlice(fileSlice)
+ .withDataSchema(avroSchema)
+ .withRequestedSchema(avroSchema)
+ .withInternalSchema(toJavaOption(originTableSchema.internalSchema))
+ .withProps(readerProperties)
+ .withLatestCommitTime(split.changes.last.getInstant)
+ .build()
+ CloseableIteratorListener.addListener(fileGroupReader.getClosableIterator).asScala
+ }
+
+ private def loadLogFile(logFile: HoodieLogFile, instant: String): Iterator[BufferedRecord[InternalRow]] = {
+ val partitionPath = FSUtils.getRelativePartitionPath(metaClient.getBasePath, logFile.getPath.getParent)
+ val readerContext = new SparkFileFormatInternalRowReaderContext(parquetReader, Seq.empty, Seq.empty,
+ conf, metaClient.getTableConfig)
+ readerContext.setLatestCommitTime(instant)
+ readerContext.setHasBootstrapBaseFile(false)
+ readerContext.setHasLogFiles(true)
+ readerContext.initRecordMerger(readerProperties)
+ readerContext.setSchemaHandler(
+ new FileGroupReaderSchemaHandler[InternalRow](readerContext, avroSchema, avroSchema,
+ Option.empty(), metaClient.getTableConfig, readerProperties))
+ val recordBuffer = new KeyBasedFileGroupRecordBuffer[InternalRow](readerContext, metaClient,
+ readerContext.getMergeMode, readerProperties, new HoodieReadStats,
+ Option.ofNullable(metaClient.getTableConfig.getPreCombineField), true, Option.empty())
+
+ HoodieMergedLogRecordReader.newBuilder[InternalRow]
+ .withStorage(metaClient.getStorage)
+ .withHoodieReaderContext(readerContext)
+ .withLogFiles(Collections.singletonList(logFile))
+ .withReverseReader(false)
+ .withBufferSize(HoodieMetadataConfig.MAX_READER_BUFFER_SIZE_PROP.defaultValue)
+ .withPartition(partitionPath)
+ .withAllowInflightInstants(true)
+ .withMetaClient(metaClient)
+ .withAllowInflightInstants(true)
Review Comment:
`withAllowInflightInstants` is expected to be used only for MDT record generation for writes
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
+ // If the record is not the same as the base record, we can emit an update
+ if (isDeleteAndRecord.getRight() != baseRecord) {
+ callbackOption.ifPresent(callback -> {
+ BufferedRecord<T> mergeResult = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
+ callback.onUpdate(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)), readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(mergeResult)));
+ });
+ }
readStats.incrementNumUpdates();
return true;
- } else if (emitDelete) {
- // emit Deletes
- nextRecord = readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey());
- readStats.incrementNumDeletes();
- return nextRecord != null;
} else {
- // not emit Deletes
+ // emit Deletes
+ callbackOption.ifPresent(callback -> callback.onDelete(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo))));
Review Comment:
won't we get just HoodieKeys from delete log block for deletes. Trying to understand how does applying output schema conversion works here? we just inject defaults values for entire data cols ?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
+ // If the record is not the same as the base record, we can emit an update
+ if (isDeleteAndRecord.getRight() != baseRecord) {
+ callbackOption.ifPresent(callback -> {
+ BufferedRecord<T> mergeResult = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
+ callback.onUpdate(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)), readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(mergeResult)));
+ });
+ }
readStats.incrementNumUpdates();
return true;
- } else if (emitDelete) {
- // emit Deletes
- nextRecord = readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey());
- readStats.incrementNumDeletes();
- return nextRecord != null;
} else {
- // not emit Deletes
+ // emit Deletes
+ callbackOption.ifPresent(callback -> callback.onDelete(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo))));
readStats.incrementNumDeletes();
- return false;
+ if (emitDelete) {
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey()));
+ return nextRecord != null;
+ } else {
+ return false;
+ }
}
}
// Inserts
- nextRecord = readerContext.seal(baseRecord);
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(baseRecord));
readStats.incrementNumInserts();
return true;
}
+ /**
+ * Applies the final output schema conversion to the buffered record if required. This ensures the records match the requested schema.
+ * @param bufferedRecord the buffered record to convert
+ * @return a new buffered record with the converted record and the proper schema ID set
+ */
+ protected BufferedRecord<T> applyOutputSchemaConversion(BufferedRecord<T> bufferedRecord) {
+ if (bufferedRecord.getRecord() != null && outputConverter.isPresent()) {
+ return new BufferedRecord<>(bufferedRecord.getRecordKey(), bufferedRecord.getOrderingValue(),
+ outputConverter.get().apply(bufferedRecord.getRecord()), readerContext.encodeAvroSchema(readerContext.getSchemaHandler().getRequestedSchema()), bufferedRecord.isDelete());
Review Comment:
minor. `readerContext.encodeAvroSchema(readerContext.getSchemaHandler().getRequestedSchema())` will be same for entire file group. we could initialize once in the constructor and use it here rater than calling `encodeAvroSchema` for every record
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
+ // If the record is not the same as the base record, we can emit an update
+ if (isDeleteAndRecord.getRight() != baseRecord) {
+ callbackOption.ifPresent(callback -> {
+ BufferedRecord<T> mergeResult = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
+ callback.onUpdate(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)), readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(mergeResult)));
+ });
+ }
readStats.incrementNumUpdates();
return true;
- } else if (emitDelete) {
- // emit Deletes
- nextRecord = readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey());
- readStats.incrementNumDeletes();
- return nextRecord != null;
} else {
- // not emit Deletes
+ // emit Deletes
+ callbackOption.ifPresent(callback -> callback.onDelete(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo))));
readStats.incrementNumDeletes();
- return false;
+ if (emitDelete) {
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey()));
Review Comment:
oh, its happening at L 573 and 599 as well
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/CDCFileGroupIterator.scala:
##########
@@ -482,47 +449,74 @@ class CDCFileGroupIterator(split: HoodieCDCFileGroupSplit,
if (!same) {
// clear up the beforeImageRecords
beforeImageRecords.clear()
- val iter = loadFileSlice(fileSlice)
- iter.foreach { row =>
- val key = getRecordKey(row)
- // Due to the reuse buffer mechanism of Spark serialization,
- // we have to copy the serialized result if we need to retain its reference
- beforeImageRecords.put(key, serialize(row, copy = true))
+ val iter = loadFileSliceWithKeys(fileSlice)
+ iter.foreach { tuple =>
+ beforeImageRecords.put(tuple._1, tuple._2)
}
// reset beforeImageFiles
beforeImageFiles.clear()
beforeImageFiles.append(files: _*)
}
}
+ private def loadFileSliceWithKeys(fileSlice: FileSlice): Iterator[(String, InternalRow)] = {
+ val readerContext = new SparkFileFormatInternalRowReaderContext(parquetReader, Seq.empty, Seq.empty,
+ conf, metaClient.getTableConfig)
+ loadFileSlice(fileSlice, readerContext).map(internalRow => {
+ val recordKey = readerContext.getRecordKey(internalRow, avroSchema)
+ (recordKey, internalRow)
+ })
+ }
+
private def loadFileSlice(fileSlice: FileSlice): Iterator[InternalRow] = {
- val baseFileInfo = storage.getPathInfo(fileSlice.getBaseFile.get().getStoragePath)
- val basePartitionedFile = sparkPartitionedFileUtils.createPartitionedFile(
- InternalRow.empty,
- baseFileInfo.getPath,
- 0,
- baseFileInfo.getLength
- )
- val logFiles = fileSlice.getLogFiles
- .sorted(HoodieLogFile.getLogFileComparator)
- .collect(Collectors.toList[HoodieLogFile])
- .asScala.toList
- .filterNot(_.getFileName.endsWith(HoodieCDCUtils.CDC_LOGFILE_SUFFIX))
-
- if (logFiles.isEmpty) {
- // no log files, just load the base parquet file
- parquetReader(basePartitionedFile)
- } else {
- // use [[RecordMergingFileIterator]] to load both the base file and log files
- val morSplit = HoodieMergeOnReadFileSplit(Some(basePartitionedFile), logFiles)
- new RecordMergingFileIterator(
- morSplit,
- BaseFileReader(parquetReader, originTableSchema.structTypeSchema),
- originTableSchema,
- originTableSchema,
- tableState,
- conf.unwrapAs(classOf[Configuration]))
- }
+ val readerContext = new SparkFileFormatInternalRowReaderContext(parquetReader, Seq.empty, Seq.empty,
+ conf, metaClient.getTableConfig)
+ loadFileSlice(fileSlice, readerContext)
+ }
+
+ private def loadFileSlice(fileSlice: FileSlice, readerContext: SparkFileFormatInternalRowReaderContext): Iterator[InternalRow] = {
+ val fileGroupReader = HoodieFileGroupReader.newBuilder()
+ .withReaderContext(readerContext)
+ .withHoodieTableMetaClient(metaClient)
+ .withFileSlice(fileSlice)
+ .withDataSchema(avroSchema)
+ .withRequestedSchema(avroSchema)
+ .withInternalSchema(toJavaOption(originTableSchema.internalSchema))
+ .withProps(readerProperties)
+ .withLatestCommitTime(split.changes.last.getInstant)
+ .build()
+ CloseableIteratorListener.addListener(fileGroupReader.getClosableIterator).asScala
+ }
+
+ private def loadLogFile(logFile: HoodieLogFile, instant: String): Iterator[BufferedRecord[InternalRow]] = {
+ val partitionPath = FSUtils.getRelativePartitionPath(metaClient.getBasePath, logFile.getPath.getParent)
+ val readerContext = new SparkFileFormatInternalRowReaderContext(parquetReader, Seq.empty, Seq.empty,
+ conf, metaClient.getTableConfig)
+ readerContext.setLatestCommitTime(instant)
+ readerContext.setHasBootstrapBaseFile(false)
+ readerContext.setHasLogFiles(true)
+ readerContext.initRecordMerger(readerProperties)
+ readerContext.setSchemaHandler(
+ new FileGroupReaderSchemaHandler[InternalRow](readerContext, avroSchema, avroSchema,
+ Option.empty(), metaClient.getTableConfig, readerProperties))
+ val recordBuffer = new KeyBasedFileGroupRecordBuffer[InternalRow](readerContext, metaClient,
+ readerContext.getMergeMode, readerProperties, new HoodieReadStats,
+ Option.ofNullable(metaClient.getTableConfig.getPreCombineField), true, Option.empty())
+
+ HoodieMergedLogRecordReader.newBuilder[InternalRow]
+ .withStorage(metaClient.getStorage)
+ .withHoodieReaderContext(readerContext)
+ .withLogFiles(Collections.singletonList(logFile))
+ .withReverseReader(false)
+ .withBufferSize(HoodieMetadataConfig.MAX_READER_BUFFER_SIZE_PROP.defaultValue)
+ .withPartition(partitionPath)
+ .withAllowInflightInstants(true)
+ .withMetaClient(metaClient)
+ .withAllowInflightInstants(true)
Review Comment:
repeated calls to `withAllowInflightInstants`
also, why do we need to set this to true?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -598,11 +638,13 @@ protected boolean hasNextLogRecord() {
while (logRecordIterator.hasNext()) {
BufferedRecord<T> nextRecordInfo = logRecordIterator.next();
if (!nextRecordInfo.isDelete()) {
- nextRecord = nextRecordInfo.getRecord();
+ BufferedRecord<T> convertedBufferedRecord = applyOutputSchemaConversion(nextRecordInfo);
+ nextRecord = convertedBufferedRecord.getRecord();
+ callbackOption.ifPresent(callback -> callback.onInsert(readerContext.constructHoodieRecord(convertedBufferedRecord)));
readStats.incrementNumInserts();
return true;
} else if (emitDelete) {
- nextRecord = readerContext.getDeleteRow(nextRecordInfo.getRecord(), nextRecordInfo.getRecordKey());
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(nextRecordInfo.getRecord(), nextRecordInfo.getRecordKey()));
Review Comment:
bcoz, in L 567 to 602, we would have invoked callbacks for updates and deletes. and only pending items are inserts from logs and so just 643 alone would suffice.
is my understanding right?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupUpdateCallback.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.model.HoodieRecord;
+
+public interface FileGroupUpdateCallback<T> {
Review Comment:
java docs please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2181943955
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
we can introduce a new composite merger based on https://github.com/apache/hudi/pull/13511, hook the cdc log write handing in `BufferedRecordMerger#finalMerge`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2977198137
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d73bf8a5fd21d41831c3fe85f54133b30c0c97c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188529773
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
+ // If the record is not the same as the base record, we can emit an update
+ if (isDeleteAndRecord.getRight() != baseRecord) {
+ callbackOption.ifPresent(callback -> {
+ BufferedRecord<T> mergeResult = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
+ callback.onUpdate(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)), readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(mergeResult)));
+ });
+ }
readStats.incrementNumUpdates();
return true;
- } else if (emitDelete) {
- // emit Deletes
- nextRecord = readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey());
- readStats.incrementNumDeletes();
- return nextRecord != null;
} else {
- // not emit Deletes
+ // emit Deletes
+ callbackOption.ifPresent(callback -> callback.onDelete(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo))));
readStats.incrementNumDeletes();
- return false;
+ if (emitDelete) {
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey()));
Review Comment:
It was happening here: https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java#L325
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188801497
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
> The CDC logic is not really part of merging, why should they be coupled?
To make the logic in file group reader buffer clean and more maintainable.
> Please also note that there will be outputs even when there is no merging in the case of log files with entries that are not in the base files.
That's why I saied `BufferedRecordMerger#finalMerge` instead of the other two APIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3042524085
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476)
* bc708ae8514d6e385f7be77c88af33101c488a52 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622)
* 8b31c59a60518f198d3d19c734dc378b8fa2ef8b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188898741
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
> The finalMerge is never called in this case.
The are two cases you plugin in the cdc logging function:
1. `FileGroupRecordBuffer#hasNextBaseRecord`;
2. `FileGroupRecordBuffer#hasNextLogRecord`.
The `finalMerge` plays a role in the 1st case, but you are right, in case2, we also need to trigger the cdc logging.
Let's introduce another two `record buffers` to override the two cases when an update callback is there.
> Can you explain what you mean by this please?
Like the new introduced check per-row:
```java
if (isDeleteAndRecord.getRight() != baseRecord) {
callbackOption.ifPresent(
...
callbackOption.ifPresent(
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3046572185
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626)
* 0d744b09a8221e626e289097a7a63726d7768259 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2191384240
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
we should only need 2, one for `KeyBasedFileGroupRecordBuffer` and another for `PositionBasedFileGroupRecordBuffer`.
https://github.com/apache/hudi/pull/13511 does not change any logic, I didn't get what you are arguing around.
Before your patch, we at lest subclass the merge handle to keep the core merge logic clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2214352748
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1046,40 +1048,14 @@ private static <T> ClosableIterator<BufferedRecord<T>> getLogRecords(List<String
.withMetaClient(datasetMetaClient)
.withAllowInflightInstants(true)
.withRecordBuffer(recordBuffer)
- .build();
- return new CloseableLogRecordsIterator<>(mergedLogRecordReader, recordBuffer);
+ .build()) {
+ // initializes the record buffer with the log records
+ return recordBuffer.getLogRecordIterator();
Review Comment:
This is an existing issue, which we should track separately. `recordBuffer.getLogRecordIterator()` invokes `ExternalSpillableMap#values` which puts all in-memory and disk-spilled records to a in-memory list which can cause OOM.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -394,4 +372,29 @@ static Comparable getOrderingValue(HoodieReaderContext readerContext,
? DEFAULT_ORDERING_VALUE
: readerContext.convertValueToEngineType(deleteRecord.getOrderingValue());
}
+
+ private static class LogRecordIterator<T> implements ClosableIterator<BufferedRecord<T>> {
+ private final FileGroupRecordBuffer<T> fileGroupRecordBuffer;
+ private final Iterator<BufferedRecord<T>> logRecordIterator;
+
+ private LogRecordIterator(FileGroupRecordBuffer<T> fileGroupRecordBuffer) {
+ this.fileGroupRecordBuffer = fileGroupRecordBuffer;
+ this.logRecordIterator = fileGroupRecordBuffer.records.values().iterator();
Review Comment:
Regarding the OOM comment I have on `HoodieTableMetadataUtil`, it can be fixed by using `fileGroupRecordBuffer.records.iterator()` instead of `fileGroupRecordBuffer.records.values().iterator()`.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/plan/generators/HoodieLogCompactionPlanGenerator.java:
##########
@@ -55,7 +54,6 @@ public HoodieLogCompactionPlanGenerator(HoodieTable table, HoodieEngineContext e
super(table, engineContext, writeConfig, executor);
this.compactionStrategy = HoodieCompactionStrategy.newBuilder()
.setStrategyParams(getStrategyParams())
- .setCompactorClassName(LogCompactionExecutionHelper.class.getName())
Review Comment:
This field is used to determine whether log compaction execution should be used by existing releases. So we should just backfill the same class name for backwards compatibility.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java:
##########
@@ -332,7 +332,7 @@ protected static Option<IndexedRecord> toAvroRecord(HoodieRecord record, Schema
try {
return record.toIndexedRecord(writerSchema, props).map(HoodieAvroIndexedRecord::getData);
} catch (IOException e) {
- LOG.error("Fail to get indexRecord from " + record, e);
+ LOG.error("Failed to convert to IndexedRecord", e);
Review Comment:
Does this cause any issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3085643726
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* ff24a54cfc19c8a89fc445ba13fab96b8b4085f4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812)
* 6513ebe37168dc7a9d162963c9aeb246f99ebdb9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217344810
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedAppendHandle.java:
##########
@@ -77,9 +79,11 @@ public void doAppend() {
props.put(HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE.key(), String.valueOf(maxMemoryPerCompaction));
// Initializes the record iterator
try (HoodieFileGroupReader<T> fileGroupReader = HoodieFileGroupReader.<T>newBuilder().withReaderContext(readerContext).withHoodieTableMetaClient(hoodieTable.getMetaClient())
- .withLatestCommitTime(instantTime).withFileSlice(fileSlice).withDataSchema(writeSchemaWithMetaFields).withRequestedSchema(writeSchemaWithMetaFields)
+ .withLatestCommitTime(instantTime).withFileSlice(fileSlice).withDataSchema(writeSchemaWithMetaFields).withRequestedSchema(writeSchemaWithMetaFields).withEnableOptimizedLogBlockScan(true)
.withInternalSchema(internalSchemaOption).withProps(props).withShouldUseRecordPosition(usePosition).withSortOutput(hoodieTable.requireSortedRecords()).build()) {
recordItr = fileGroupReader.getClosableHoodieRecordIterator();
+ header.put(HoodieLogBlock.HeaderMetadataType.COMPACTED_BLOCK_TIMES,
+ StringUtils.join(fileGroupReader.getValidBlockInstants(), ","));
Review Comment:
Is there any test coverage around this? Seems that the log comapction tests pass without the change. We can add validation on the log header to make sure `COMPACTED_BLOCK_TIMES` is added, in an existing log compaction test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217345581
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/cdc/HoodieCDCTestBase.scala:
##########
@@ -57,7 +57,8 @@ abstract class HoodieCDCTestBase extends HoodieSparkClientTestBase {
PRECOMBINE_FIELD.key -> "timestamp",
HoodieWriteConfig.TBL_NAME.key -> "hoodie_test",
HoodieMetadataConfig.COMPACT_NUM_DELTA_COMMITS.key -> "1",
- HoodieCleanConfig.AUTO_CLEAN.key -> "false"
+ HoodieCleanConfig.AUTO_CLEAN.key -> "false",
+ HoodieWriteConfig.MARKERS_TIMELINE_SERVER_BASED_BATCH_INTERVAL_MS.key -> "10"
Review Comment:
Got it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2192633570
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
> > Don't forget SortedKeyBasedFileGroupRecordBuffer
>
> This is used for MDT right? do we need to log cdc in such scenarios?
>
@danny0405 there is already `HoodieSortedMergeHandleWithChangeLog` so there is already support for CDC on with sorted tables and I think we should preserve that.
> > If we really care about a single check that happens for a subset of rows, why isn't the same scrutiny applied to other PRs regardless of whether it is new logic or not
>
> We did it when Lin introduces more merging mode to FG reader, that is why we added the `BufferedRecordMerger` absractions, sub-classing the file group record buffer is just a suggestion, you can think other alternatives to get rid of the new introduced per-row handling and keep the core merging logic clean and clear.
@danny0405 as mentioned before this is not really per-row. It is per-deduped-log record. Like I also mentioned before, this performance can be easily recovered by some other basic improvements like avoiding the auto-boxing and un-boxing that happens in these same cases. When you call a method for another class, this also introduces an implicit null check in java so by moving the merger code into a separate class it has also added "per-row" degradation by your logic. These checks are on the order of 1 nanosecond or less with modern JVMs. So with 1M updates to the base file, you get 1 millisecond of latency added.
We can skip using Option to also get performance improvements. Making these extra objects is just overhead for the JVM and we can use `null` instead but that sacrifices readability and intentionality of the code. There are so many places we can shave a nanosecond or more in this codebase if we want to make these tradeoffs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3049343952
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbb2380688b2279b75557d443d2e238eb32a1056 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3042659965
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc708ae8514d6e385f7be77c88af33101c488a52 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622)
* 8b31c59a60518f198d3d19c734dc378b8fa2ef8b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188837358
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
> > The CDC logic is not really part of merging, why should they be coupled?
>
> To make the logic in file group reader buffer clean and more maintainable.
>
That is a fine goal but we should consider that we will now need the logic in two places, one in the merger and one in the file group reader buffer now due to the point I have already raised.
> > Please also note that there will be outputs even when there is no merging in the case of log files with entries that are not in the base files.
>
> That's why I saied `BufferedRecordMerger#finalMerge` instead of the other two APIs.
The `finalMerge` is never called in this case. `finalMerge` is only used in the case where there is a record in the base file that is merged with some records coming from log files.
> In any case, please stop introducing row-level ramifications continuously if the `cdc` logging is deterministic per-query.
Can you explain what you mean by this please?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
> > The CDC logic is not really part of merging, why should they be coupled?
>
> To make the logic in file group reader buffer clean and more maintainable.
>
That is a fine goal but we should consider that we will now need the logic in two places, one in the merger and one in the file group reader buffer now due to the point I have already raised.
> > Please also note that there will be outputs even when there is no merging in the case of log files with entries that are not in the base files.
>
> That's why I saied `BufferedRecordMerger#finalMerge` instead of the other two APIs.
The `finalMerge` is never called in this case. `finalMerge` is only used in the case where there is a record in the base file that is merged with some records coming from log files.
> In any case, please stop introducing row-level ramifications continuously if the `cdc` logging is deterministic per-query.
Can you explain what you mean by this please?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188566283
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
+ // If the record is not the same as the base record, we can emit an update
+ if (isDeleteAndRecord.getRight() != baseRecord) {
+ callbackOption.ifPresent(callback -> {
+ BufferedRecord<T> mergeResult = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
+ callback.onUpdate(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)), readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(mergeResult)));
+ });
+ }
readStats.incrementNumUpdates();
return true;
- } else if (emitDelete) {
- // emit Deletes
- nextRecord = readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey());
- readStats.incrementNumDeletes();
- return nextRecord != null;
} else {
- // not emit Deletes
+ // emit Deletes
+ callbackOption.ifPresent(callback -> callback.onDelete(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo))));
readStats.incrementNumDeletes();
- return false;
+ if (emitDelete) {
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey()));
+ return nextRecord != null;
+ } else {
+ return false;
+ }
}
}
// Inserts
- nextRecord = readerContext.seal(baseRecord);
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(baseRecord));
readStats.incrementNumInserts();
return true;
}
+ /**
+ * Applies the final output schema conversion to the buffered record if required. This ensures the records match the requested schema.
+ * @param bufferedRecord the buffered record to convert
+ * @return a new buffered record with the converted record and the proper schema ID set
+ */
+ protected BufferedRecord<T> applyOutputSchemaConversion(BufferedRecord<T> bufferedRecord) {
+ if (bufferedRecord.getRecord() != null && outputConverter.isPresent()) {
+ return new BufferedRecord<>(bufferedRecord.getRecordKey(), bufferedRecord.getOrderingValue(),
+ outputConverter.get().apply(bufferedRecord.getRecord()), readerContext.encodeAvroSchema(readerContext.getSchemaHandler().getRequestedSchema()), bufferedRecord.isDelete());
Review Comment:
Yes, will update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2194406117
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
Did I ever mention any keyword about performance here? It's the maintainace and extensibility I've been addressing all the time. Let's keep the major work flow clean and clear, that's my point.
Tell me why we need to care about whether to trigger cdc loggings per-row here for regular read/writer paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2198996346
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -349,11 +385,13 @@ protected boolean hasNextLogRecord() {
while (logRecordIterator.hasNext()) {
BufferedRecord<T> nextRecordInfo = logRecordIterator.next();
if (!nextRecordInfo.isDelete()) {
- nextRecord = nextRecordInfo.getRecord();
+ BufferedRecord<T> convertedBufferedRecord = applyOutputSchemaConversion(nextRecordInfo);
+ nextRecord = convertedBufferedRecord.getRecord();
+ handleBaseFileInsert(nextRecordInfo.getRecordKey(), nextRecord);
readStats.incrementNumInserts();
return true;
} else if (emitDelete) {
- nextRecord = readerContext.getDeleteRow(nextRecordInfo.getRecord(), nextRecordInfo.getRecordKey());
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(nextRecordInfo.getRecord(), nextRecordInfo.getRecordKey()));
Review Comment:
This matches the current logic [here](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieMergeHandleWithChangeLog.java#L106). It seems like the changes within the log files are not output to the CDCLogger, only the changes to the base file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2198999019
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/plan/generators/HoodieLogCompactionPlanGenerator.java:
##########
@@ -55,7 +54,6 @@ public HoodieLogCompactionPlanGenerator(HoodieTable table, HoodieEngineContext e
super(table, engineContext, writeConfig, executor);
this.compactionStrategy = HoodieCompactionStrategy.newBuilder()
.setStrategyParams(getStrategyParams())
- .setCompactorClassName(LogCompactionExecutionHelper.class.getName())
Review Comment:
The field is nullable but it probably makes more sense to keep setting this for now for compatibility between versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2994550968
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d73bf8a5fd21d41831c3fe85f54133b30c0c97c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240)
* b4765033b87250ef5654d2d71818ae398fa7e91f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2994813427
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408)
* c716bd4c30370148bb32494941fe33e50e6ef8c1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2997587539
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 415a6a4bc909295ba21e97600ddf44d00bcf050b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417)
* f75a7f9495ddde683839d777894769af2887c9a6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2997191152
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 415a6a4bc909295ba21e97600ddf44d00bcf050b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217352786
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedAppendHandle.java:
##########
@@ -77,9 +79,11 @@ public void doAppend() {
props.put(HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE.key(), String.valueOf(maxMemoryPerCompaction));
// Initializes the record iterator
try (HoodieFileGroupReader<T> fileGroupReader = HoodieFileGroupReader.<T>newBuilder().withReaderContext(readerContext).withHoodieTableMetaClient(hoodieTable.getMetaClient())
- .withLatestCommitTime(instantTime).withFileSlice(fileSlice).withDataSchema(writeSchemaWithMetaFields).withRequestedSchema(writeSchemaWithMetaFields)
+ .withLatestCommitTime(instantTime).withFileSlice(fileSlice).withDataSchema(writeSchemaWithMetaFields).withRequestedSchema(writeSchemaWithMetaFields).withEnableOptimizedLogBlockScan(true)
.withInternalSchema(internalSchemaOption).withProps(props).withShouldUseRecordPosition(usePosition).withSortOutput(hoodieTable.requireSortedRecords()).build()) {
recordItr = fileGroupReader.getClosableHoodieRecordIterator();
+ header.put(HoodieLogBlock.HeaderMetadataType.COMPACTED_BLOCK_TIMES,
+ StringUtils.join(fileGroupReader.getValidBlockInstants(), ","));
Review Comment:
I'm adding some more testing to the log compaction as part of another [PR](https://github.com/apache/hudi/pull/13556), can we add it there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217080942
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/cdc/HoodieCDCTestBase.scala:
##########
@@ -57,7 +57,8 @@ abstract class HoodieCDCTestBase extends HoodieSparkClientTestBase {
PRECOMBINE_FIELD.key -> "timestamp",
HoodieWriteConfig.TBL_NAME.key -> "hoodie_test",
HoodieMetadataConfig.COMPACT_NUM_DELTA_COMMITS.key -> "1",
- HoodieCleanConfig.AUTO_CLEAN.key -> "false"
+ HoodieCleanConfig.AUTO_CLEAN.key -> "false",
+ HoodieWriteConfig.MARKERS_TIMELINE_SERVER_BASED_BATCH_INTERVAL_MS.key -> "10"
Review Comment:
No it was just speeding up the testing a bit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217118369
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/LogCompactionExecutionHelper.java:
##########
@@ -1,86 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.action.compact;
-
-import org.apache.hudi.client.WriteStatus;
-import org.apache.hudi.common.model.CompactionOperation;
-import org.apache.hudi.common.model.HoodieBaseFile;
-import org.apache.hudi.common.model.HoodieRecordPayload;
-import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
-import org.apache.hudi.common.table.timeline.HoodieInstant;
-import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
-import org.apache.hudi.config.HoodieWriteConfig;
-import org.apache.hudi.table.HoodieCompactionHandler;
-import org.apache.hudi.table.HoodieTable;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-
-/**
- * Strategy class to execute log compaction operations.
- */
-public class LogCompactionExecutionHelper<T extends HoodieRecordPayload, I, K, O>
- extends CompactionExecutionHelper<T, I, K, O> {
-
- private static final Logger LOG = LoggerFactory.getLogger(LogCompactionExecutionHelper.class);
-
- @Override
- protected void transitionRequestedToInflight(HoodieTable table, String logCompactionInstantTime) {
- HoodieActiveTimeline timeline = table.getActiveTimeline();
- HoodieInstant instant = table.getMetaClient().getInstantGenerator().getLogCompactionRequestedInstant(logCompactionInstantTime);
- // Mark instant as compaction inflight
- timeline.transitionLogCompactionRequestedToInflight(instant);
- }
-
- protected String instantTimeToUseForScanning(String logCompactionInstantTime, String maxInstantTime) {
- return logCompactionInstantTime;
- }
-
- protected boolean shouldPreserveCommitMetadata() {
- return true;
- }
-
- @Override
- protected Iterator<List<WriteStatus>> writeFileAndGetWriteStats(HoodieCompactionHandler compactionHandler,
- CompactionOperation operation,
- String instantTime,
- HoodieMergedLogRecordScanner scanner,
- Option<HoodieBaseFile> oldDataFileOpt) throws IOException {
- Map<HoodieLogBlock.HeaderMetadataType, String> header = new HashMap<>();
- header.put(HoodieLogBlock.HeaderMetadataType.COMPACTED_BLOCK_TIMES,
- StringUtils.join(scanner.getValidBlockInstants(), ","));
- // Compacting is very similar to applying updates to existing file
- return compactionHandler.handleInsertsForLogCompaction(instantTime, operation.getPartitionPath(),
- operation.getFileId(), scanner.getRecords(), header);
- }
-
- @Override
- protected boolean enableOptimizedLogBlockScan(HoodieWriteConfig writeConfig) {
- return true;
- }
Review Comment:
It was not, updated
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/LogCompactionExecutionHelper.java:
##########
@@ -1,86 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.action.compact;
-
-import org.apache.hudi.client.WriteStatus;
-import org.apache.hudi.common.model.CompactionOperation;
-import org.apache.hudi.common.model.HoodieBaseFile;
-import org.apache.hudi.common.model.HoodieRecordPayload;
-import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
-import org.apache.hudi.common.table.timeline.HoodieInstant;
-import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
-import org.apache.hudi.config.HoodieWriteConfig;
-import org.apache.hudi.table.HoodieCompactionHandler;
-import org.apache.hudi.table.HoodieTable;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-
-/**
- * Strategy class to execute log compaction operations.
- */
-public class LogCompactionExecutionHelper<T extends HoodieRecordPayload, I, K, O>
- extends CompactionExecutionHelper<T, I, K, O> {
-
- private static final Logger LOG = LoggerFactory.getLogger(LogCompactionExecutionHelper.class);
-
- @Override
- protected void transitionRequestedToInflight(HoodieTable table, String logCompactionInstantTime) {
- HoodieActiveTimeline timeline = table.getActiveTimeline();
- HoodieInstant instant = table.getMetaClient().getInstantGenerator().getLogCompactionRequestedInstant(logCompactionInstantTime);
- // Mark instant as compaction inflight
- timeline.transitionLogCompactionRequestedToInflight(instant);
- }
-
- protected String instantTimeToUseForScanning(String logCompactionInstantTime, String maxInstantTime) {
- return logCompactionInstantTime;
- }
-
- protected boolean shouldPreserveCommitMetadata() {
- return true;
- }
-
- @Override
- protected Iterator<List<WriteStatus>> writeFileAndGetWriteStats(HoodieCompactionHandler compactionHandler,
- CompactionOperation operation,
- String instantTime,
- HoodieMergedLogRecordScanner scanner,
- Option<HoodieBaseFile> oldDataFileOpt) throws IOException {
- Map<HoodieLogBlock.HeaderMetadataType, String> header = new HashMap<>();
- header.put(HoodieLogBlock.HeaderMetadataType.COMPACTED_BLOCK_TIMES,
- StringUtils.join(scanner.getValidBlockInstants(), ","));
Review Comment:
Added in latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217084881
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+
+import java.util.function.UnaryOperator;
+
+/**
+ * Interface used within the {@link HoodieFileGroupReader<T>} for processing updates to records in Merge-on-Read tables.
+ * Note that the updates are always relative to the base file's current state.
+ * @param <T> the engine specific record type
+ */
+public interface UpdateProcessor<T> {
+ /**
+ * Processes the update to the record. If the update should not be returned to the caller, the method should return null.
+ * @param recordKey the key of the record being updated
+ * @param previousRecord the previous version of the record, or null if there is no previous value
+ * @param currentRecord the current version of the record after merging with the existing record, if any exists
+ * @param isDelete a flag indicating whether the merge resulted in a delete operation
+ * @return the processed record, or null if the record should not be returned to the caller
+ */
+ T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete);
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217365794
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedAppendHandle.java:
##########
@@ -77,9 +79,11 @@ public void doAppend() {
props.put(HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE.key(), String.valueOf(maxMemoryPerCompaction));
// Initializes the record iterator
try (HoodieFileGroupReader<T> fileGroupReader = HoodieFileGroupReader.<T>newBuilder().withReaderContext(readerContext).withHoodieTableMetaClient(hoodieTable.getMetaClient())
- .withLatestCommitTime(instantTime).withFileSlice(fileSlice).withDataSchema(writeSchemaWithMetaFields).withRequestedSchema(writeSchemaWithMetaFields)
+ .withLatestCommitTime(instantTime).withFileSlice(fileSlice).withDataSchema(writeSchemaWithMetaFields).withRequestedSchema(writeSchemaWithMetaFields).withEnableOptimizedLogBlockScan(true)
.withInternalSchema(internalSchemaOption).withProps(props).withShouldUseRecordPosition(usePosition).withSortOutput(hoodieTable.requireSortedRecords()).build()) {
recordItr = fileGroupReader.getClosableHoodieRecordIterator();
+ header.put(HoodieLogBlock.HeaderMetadataType.COMPACTED_BLOCK_TIMES,
+ StringUtils.join(fileGroupReader.getValidBlockInstants(), ","));
Review Comment:
Yes, sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #13444:
URL: https://github.com/apache/hudi/pull/13444
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3067555628
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbb2380688b2279b75557d443d2e238eb32a1056 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649)
* 86bcb912ae11da007eb9228660a06a3f22965e27 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714)
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3067767445
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 5540b78becd2ea0cfd5c13732d22ee463392663c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3046949455
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a79b08ffcf0266bdda5e4faea181fc21e3efcab4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2191460353
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
> Don't forget SortedKeyBasedFileGroupRecordBuffer
This is used for MDT right? do we need to log cdc in such scenarios?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188572023
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupUpdateCallback.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.model.HoodieRecord;
+
+public interface FileGroupUpdateCallback<T> {
Review Comment:
Updated the interface name and added javadocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188572284
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/CDCFileGroupIterator.scala:
##########
@@ -402,10 +375,8 @@ class CDCFileGroupIterator(split: HoodieCDCFileGroupSplit,
&& currentCDCFileSplit.getBeforeFileSlice.isPresent)
loadBeforeFileSliceIfNeeded(currentCDCFileSplit.getBeforeFileSlice.get)
val absLogPath = new StoragePath(basePath, currentCDCFileSplit.getCdcFiles.get(0))
- val morSplit = HoodieMergeOnReadFileSplit(None, List(new HoodieLogFile(storage.getPathInfo(absLogPath))))
- val logFileIterator = new LogFileIterator(
Review Comment:
This happens here: https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/read/KeyBasedFileGroupRecordBuffer.java#L83
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188567983
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
Please also note that there will be outputs even when there is no merging in the case of log files with entries that are not in the base files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2997036543
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c716bd4c30370148bb32494941fe33e50e6ef8c1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410)
* 415a6a4bc909295ba21e97600ddf44d00bcf050b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2196396567
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -109,18 +112,23 @@ public HoodieFileGroupReader(HoodieReaderContext<T> readerContext, HoodieStorage
long start, long length, boolean shouldUseRecordPosition) {
this(readerContext, storage, tablePath, latestCommitTime, fileSlice, dataSchema,
requestedSchema, internalSchemaOpt, hoodieTableMetaClient, props, start, length,
- shouldUseRecordPosition, false, false, false);
+ shouldUseRecordPosition, false, false, false, Option.empty());
}
private HoodieFileGroupReader(HoodieReaderContext<T> readerContext, HoodieStorage storage, String tablePath,
String latestCommitTime, FileSlice fileSlice, Schema dataSchema, Schema requestedSchema,
Option<InternalSchema> internalSchemaOpt, HoodieTableMetaClient hoodieTableMetaClient, TypedProperties props,
- long start, long length, boolean shouldUseRecordPosition, boolean allowInflightInstants, boolean emitDelete, boolean sortOutput) {
+ long start, long length, boolean shouldUseRecordPosition, boolean allowInflightInstants, boolean emitDelete, boolean sortOutput,
+ Option<FileGroupUpdateCallback<T>> updateCallback) {
this.readerContext = readerContext;
+ this.fileGroupUpdateCallback = updateCallback;
this.metaClient = hoodieTableMetaClient;
this.storage = storage;
this.hoodieBaseFileOption = fileSlice.getBaseFile();
- this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
+ this.logFiles = fileSlice.getLogFiles()
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2997042578
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c716bd4c30370148bb32494941fe33e50e6ef8c1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410)
* 415a6a4bc909295ba21e97600ddf44d00bcf050b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2214385781
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -394,4 +372,29 @@ static Comparable getOrderingValue(HoodieReaderContext readerContext,
? DEFAULT_ORDERING_VALUE
: readerContext.convertValueToEngineType(deleteRecord.getOrderingValue());
}
+
+ private static class LogRecordIterator<T> implements ClosableIterator<BufferedRecord<T>> {
+ private final FileGroupRecordBuffer<T> fileGroupRecordBuffer;
+ private final Iterator<BufferedRecord<T>> logRecordIterator;
+
+ private LogRecordIterator(FileGroupRecordBuffer<T> fileGroupRecordBuffer) {
+ this.fileGroupRecordBuffer = fileGroupRecordBuffer;
+ this.logRecordIterator = fileGroupRecordBuffer.records.values().iterator();
Review Comment:
Thanks for catching this, I updated it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2214382792
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteHandle.java:
##########
@@ -332,7 +332,7 @@ protected static Option<IndexedRecord> toAvroRecord(HoodieRecord record, Schema
try {
return record.toIndexedRecord(writerSchema, props).map(HoodieAvroIndexedRecord::getData);
} catch (IOException e) {
- LOG.error("Fail to get indexRecord from " + record, e);
+ LOG.error("Failed to convert to IndexedRecord", e);
Review Comment:
This will log out the contents of the record which can leak sensitive information into logs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2194406117
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
Did I ever mention any keyword about performance here? It's the maintainace and extensibility I've been addressing all the time. Let's keep the major work flow clean and clear, that's my point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3052972196
As discussed, I can shepherd this PR once @yihua chimes in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2196391444
##########
hudi-client/hudi-flink-client/src/main/java/org/apache/hudi/table/action/compact/HoodieFlinkMergeOnReadTableCompactor.java:
##########
@@ -69,35 +67,22 @@ public void preCompact(
}
}
- public List<WriteStatus> compact(HoodieCompactionHandler compactionHandler,
- HoodieWriteConfig writeConfig,
+ public List<WriteStatus> compact(HoodieWriteConfig writeConfig,
CompactionOperation operation,
String instantTime,
TaskContextSupplier taskContextSupplier,
- Option<HoodieReaderContext<?>> readerContextOpt,
+ HoodieReaderContext<?> readerContext,
HoodieTable table) throws IOException {
String maxInstantTime = getMaxInstantTime(table.getMetaClient());
Review Comment:
There is a read config `hoodie.file.group.reader.enabled`, it looks like we force to use FG reader for compaction now? Are we planning to remove this option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2193711190
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
Let me empahsize it again, I will definitely give it a -1 if we do not get rid of the new introduced ramifications here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3092339728
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
}, {
"hash" : "73e6ef97620b31d888b37d02733f12937807d513",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851",
"triggerID" : "73e6ef97620b31d888b37d02733f12937807d513",
"triggerType" : "PUSH"
}, {
"hash" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6855",
"triggerID" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"triggerType" : "PUSH"
}, {
"hash" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6856",
"triggerID" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"triggerType" : "PUSH"
}, {
"hash" : "93425d152b8586d862d0f5fa4a58f69b12628986",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6859",
"triggerID" : "93425d152b8586d862d0f5fa4a58f69b12628986",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 93425d152b8586d862d0f5fa4a58f69b12628986 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6859)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217105331
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/LogCompactionExecutionHelper.java:
##########
@@ -1,86 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.action.compact;
-
-import org.apache.hudi.client.WriteStatus;
-import org.apache.hudi.common.model.CompactionOperation;
-import org.apache.hudi.common.model.HoodieBaseFile;
-import org.apache.hudi.common.model.HoodieRecordPayload;
-import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
-import org.apache.hudi.common.table.timeline.HoodieInstant;
-import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
-import org.apache.hudi.config.HoodieWriteConfig;
-import org.apache.hudi.table.HoodieCompactionHandler;
-import org.apache.hudi.table.HoodieTable;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-
-/**
- * Strategy class to execute log compaction operations.
- */
-public class LogCompactionExecutionHelper<T extends HoodieRecordPayload, I, K, O>
- extends CompactionExecutionHelper<T, I, K, O> {
-
- private static final Logger LOG = LoggerFactory.getLogger(LogCompactionExecutionHelper.class);
-
- @Override
- protected void transitionRequestedToInflight(HoodieTable table, String logCompactionInstantTime) {
- HoodieActiveTimeline timeline = table.getActiveTimeline();
- HoodieInstant instant = table.getMetaClient().getInstantGenerator().getLogCompactionRequestedInstant(logCompactionInstantTime);
- // Mark instant as compaction inflight
- timeline.transitionLogCompactionRequestedToInflight(instant);
- }
-
- protected String instantTimeToUseForScanning(String logCompactionInstantTime, String maxInstantTime) {
- return logCompactionInstantTime;
- }
-
- protected boolean shouldPreserveCommitMetadata() {
- return true;
- }
-
- @Override
- protected Iterator<List<WriteStatus>> writeFileAndGetWriteStats(HoodieCompactionHandler compactionHandler,
- CompactionOperation operation,
- String instantTime,
- HoodieMergedLogRecordScanner scanner,
- Option<HoodieBaseFile> oldDataFileOpt) throws IOException {
- Map<HoodieLogBlock.HeaderMetadataType, String> header = new HashMap<>();
- header.put(HoodieLogBlock.HeaderMetadataType.COMPACTED_BLOCK_TIMES,
- StringUtils.join(scanner.getValidBlockInstants(), ","));
- // Compacting is very similar to applying updates to existing file
- return compactionHandler.handleInsertsForLogCompaction(instantTime, operation.getPartitionPath(),
- operation.getFileId(), scanner.getRecords(), header);
- }
-
- @Override
- protected boolean enableOptimizedLogBlockScan(HoodieWriteConfig writeConfig) {
- return true;
- }
Review Comment:
Similarly, let's check if `FileGroupReaderBasedAppendHandle` handles optimized log block scan separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2977193127
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d73bf8a5fd21d41831c3fe85f54133b30c0c97c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2189907001
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
I fundamentally disagree with the idea of adding more record buffers instead of injecting logic. The maintenance of this becomes challenging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3046568594
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626)
* 0d744b09a8221e626e289097a7a63726d7768259 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2191402190
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
Don't forget `SortedKeyBasedFileGroupRecordBuffer`. We already need 3 right now, but what if more merging strategies are added? That is what I mean by maintenance.
Subclassing is not cleaner. There is even a famous OOP principle about this that I am sure you know.
If we really care about a single check that happens for a subset of rows, why isn't the same scrutiny applied to other PRs regardless of whether it is new logic or not? The code changed even if the core logic is preserved. It is an opportunity for improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2191423413
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
> about [auto-boxing performance degradation](https://github.com/apache/hudi/pull/13511/files#diff-08630014159096d27df01401a0fe2178f535d90045d09c1b5dfe8e0cefb7ee66R66)
Do you mean the boxing happened in the return value of `finalMerge` or something else? if it's the former, that's not introduced in #13511.
> [per merge checks](https://github.com/apache/hudi/pull/13511/files#diff-44ea0a598fb8c09126a1dcccad5c3f572c8aadae48c18dcc800e977a4cfb1c5fR313)
Not sure which part are you referring to, could you give more details?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2994736780
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b4765033b87250ef5654d2d71818ae398fa7e91f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404)
* b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2994795039
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b4765033b87250ef5654d2d71818ae398fa7e91f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404)
* b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408)
* c716bd4c30370148bb32494941fe33e50e6ef8c1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217087305
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/CDCFileGroupIterator.scala:
##########
@@ -430,10 +400,7 @@ class CDCFileGroupIterator(split: HoodieCDCFileGroupSplit,
if (currentCDCFileSplit.getBeforeFileSlice.isPresent) {
loadBeforeFileSliceIfNeeded(currentCDCFileSplit.getBeforeFileSlice.get)
}
- recordIter = beforeImageRecords.values.map { record =>
- deserialize(record)
- }.iterator
- beforeImageRecords.clear()
Review Comment:
loadCDCFile still clears the map at the beginning of the method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217087488
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/CDCFileGroupIterator.scala:
##########
@@ -414,11 +385,10 @@ class CDCFileGroupIterator(split: HoodieCDCFileGroupSplit,
}
// load afterFileSlice to afterImageRecords
if (currentCDCFileSplit.getAfterFileSlice.isPresent) {
- val iter = loadFileSlice(currentCDCFileSplit.getAfterFileSlice.get())
+ val iter = loadFileSliceWithKeys(currentCDCFileSplit.getAfterFileSlice.get())
afterImageRecords.clear()
- iter.foreach { row =>
- val key = getRecordKey(row)
- afterImageRecords.put(key, row.copy())
+ iter.foreach { tuple =>
+ afterImageRecords.put(tuple._1, tuple._2.copy())
Review Comment:
It's not, we will do the copy in FGReader now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3091432505
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
}, {
"hash" : "73e6ef97620b31d888b37d02733f12937807d513",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851",
"triggerID" : "73e6ef97620b31d888b37d02733f12937807d513",
"triggerType" : "PUSH"
}, {
"hash" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6855",
"triggerID" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 73e6ef97620b31d888b37d02733f12937807d513 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851)
* c9dc3001cfb9e1e542e3e12062f018f6301b8f04 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6855)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217117719
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UnmergedFileGroupRecordBuffer.java:
##########
@@ -52,9 +51,8 @@ public UnmergedFileGroupRecordBuffer(
RecordMergeMode recordMergeMode,
PartialUpdateMode partialUpdateMode,
TypedProperties props,
- HoodieReadStats readStats,
- boolean emitDelete) {
- super(readerContext, hoodieTableMetaClient, recordMergeMode, partialUpdateMode, props, readStats, Option.empty(), emitDelete);
+ HoodieReadStats readStats) {
+ super(readerContext, hoodieTableMetaClient, recordMergeMode, partialUpdateMode, props, readStats, Option.empty(), null);
Review Comment:
Makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217078762
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/cdc/HoodieCDCTestBase.scala:
##########
@@ -57,7 +57,8 @@ abstract class HoodieCDCTestBase extends HoodieSparkClientTestBase {
PRECOMBINE_FIELD.key -> "timestamp",
HoodieWriteConfig.TBL_NAME.key -> "hoodie_test",
HoodieMetadataConfig.COMPACT_NUM_DELTA_COMMITS.key -> "1",
- HoodieCleanConfig.AUTO_CLEAN.key -> "false"
+ HoodieCleanConfig.AUTO_CLEAN.key -> "false",
+ HoodieWriteConfig.MARKERS_TIMELINE_SERVER_BASED_BATCH_INTERVAL_MS.key -> "10"
Review Comment:
Is the test flaky without this config?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+
+import java.util.function.UnaryOperator;
+
+/**
+ * Interface used within the {@link HoodieFileGroupReader<T>} for processing updates to records in Merge-on-Read tables.
+ * Note that the updates are always relative to the base file's current state.
+ * @param <T> the engine specific record type
+ */
+public interface UpdateProcessor<T> {
+ /**
+ * Processes the update to the record. If the update should not be returned to the caller, the method should return null.
+ * @param recordKey the key of the record being updated
+ * @param previousRecord the previous version of the record, or null if there is no previous value
+ * @param currentRecord the current version of the record after merging with the existing record, if any exists
+ * @param isDelete a flag indicating whether the merge resulted in a delete operation
+ * @return the processed record, or null if the record should not be returned to the caller
+ */
+ T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete);
Review Comment:
Should this be named as `mergedRecord` to avoid confusion? Similar for classes that implement this interface.
```suggestion
T processUpdate(String recordKey, T previousRecord, T mergedRecord, boolean isDelete);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3092303219
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
}, {
"hash" : "73e6ef97620b31d888b37d02733f12937807d513",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851",
"triggerID" : "73e6ef97620b31d888b37d02733f12937807d513",
"triggerType" : "PUSH"
}, {
"hash" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6855",
"triggerID" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"triggerType" : "PUSH"
}, {
"hash" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6856",
"triggerID" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"triggerType" : "PUSH"
}, {
"hash" : "93425d152b8586d862d0f5fa4a58f69b12628986",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "93425d152b8586d862d0f5fa4a58f69b12628986",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 05d26ce59cd2eee1864bbcca5d8f93236b3043ca Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6856)
* 93425d152b8586d862d0f5fa4a58f69b12628986 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217072586
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedMergeHandle.java:
##########
@@ -88,6 +92,19 @@ public FileGroupReaderBasedMergeHandle(HoodieWriteConfig config, String instantT
this.operation = operation;
// If the table is a metadata table or the base file is an HFile, we use AVRO record type, otherwise we use the engine record type.
this.recordType = hoodieTable.isMetadataTable() || HFILE.getFileExtension().equals(hoodieTable.getBaseFileExtension()) ? HoodieRecord.HoodieRecordType.AVRO : enginRecordType;
+ if (hoodieTable.getMetaClient().getTableConfig().isCDCEnabled()) {
+ this.cdcLogger = Option.of(new HoodieCDCLogger(
+ instantTime,
+ config,
+ hoodieTable.getMetaClient().getTableConfig(),
+ partitionPath,
+ getStorage(),
Review Comment:
nit: `storage` is directly available here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3085832561
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 6513ebe37168dc7a9d162963c9aeb246f99ebdb9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3085624193
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* ff24a54cfc19c8a89fc445ba13fab96b8b4085f4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3085369591
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 5540b78becd2ea0cfd5c13732d22ee463392663c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716)
* ff24a54cfc19c8a89fc445ba13fab96b8b4085f4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2191223528
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/CDCFileGroupIterator.scala:
##########
@@ -482,47 +449,74 @@ class CDCFileGroupIterator(split: HoodieCDCFileGroupSplit,
if (!same) {
// clear up the beforeImageRecords
beforeImageRecords.clear()
- val iter = loadFileSlice(fileSlice)
- iter.foreach { row =>
- val key = getRecordKey(row)
- // Due to the reuse buffer mechanism of Spark serialization,
- // we have to copy the serialized result if we need to retain its reference
- beforeImageRecords.put(key, serialize(row, copy = true))
+ val iter = loadFileSliceWithKeys(fileSlice)
+ iter.foreach { tuple =>
+ beforeImageRecords.put(tuple._1, tuple._2)
}
// reset beforeImageFiles
beforeImageFiles.clear()
beforeImageFiles.append(files: _*)
}
}
+ private def loadFileSliceWithKeys(fileSlice: FileSlice): Iterator[(String, InternalRow)] = {
+ val readerContext = new SparkFileFormatInternalRowReaderContext(parquetReader, Seq.empty, Seq.empty,
+ conf, metaClient.getTableConfig)
+ loadFileSlice(fileSlice, readerContext).map(internalRow => {
+ val recordKey = readerContext.getRecordKey(internalRow, avroSchema)
+ (recordKey, internalRow)
+ })
+ }
+
private def loadFileSlice(fileSlice: FileSlice): Iterator[InternalRow] = {
- val baseFileInfo = storage.getPathInfo(fileSlice.getBaseFile.get().getStoragePath)
- val basePartitionedFile = sparkPartitionedFileUtils.createPartitionedFile(
- InternalRow.empty,
- baseFileInfo.getPath,
- 0,
- baseFileInfo.getLength
- )
- val logFiles = fileSlice.getLogFiles
- .sorted(HoodieLogFile.getLogFileComparator)
- .collect(Collectors.toList[HoodieLogFile])
- .asScala.toList
- .filterNot(_.getFileName.endsWith(HoodieCDCUtils.CDC_LOGFILE_SUFFIX))
-
- if (logFiles.isEmpty) {
- // no log files, just load the base parquet file
- parquetReader(basePartitionedFile)
- } else {
- // use [[RecordMergingFileIterator]] to load both the base file and log files
- val morSplit = HoodieMergeOnReadFileSplit(Some(basePartitionedFile), logFiles)
- new RecordMergingFileIterator(
- morSplit,
- BaseFileReader(parquetReader, originTableSchema.structTypeSchema),
- originTableSchema,
- originTableSchema,
- tableState,
- conf.unwrapAs(classOf[Configuration]))
- }
+ val readerContext = new SparkFileFormatInternalRowReaderContext(parquetReader, Seq.empty, Seq.empty,
+ conf, metaClient.getTableConfig)
+ loadFileSlice(fileSlice, readerContext)
+ }
+
+ private def loadFileSlice(fileSlice: FileSlice, readerContext: SparkFileFormatInternalRowReaderContext): Iterator[InternalRow] = {
+ val fileGroupReader = HoodieFileGroupReader.newBuilder()
+ .withReaderContext(readerContext)
+ .withHoodieTableMetaClient(metaClient)
+ .withFileSlice(fileSlice)
+ .withDataSchema(avroSchema)
+ .withRequestedSchema(avroSchema)
+ .withInternalSchema(toJavaOption(originTableSchema.internalSchema))
+ .withProps(readerProperties)
+ .withLatestCommitTime(split.changes.last.getInstant)
+ .build()
+ CloseableIteratorListener.addListener(fileGroupReader.getClosableIterator).asScala
+ }
+
+ private def loadLogFile(logFile: HoodieLogFile, instant: String): Iterator[BufferedRecord[InternalRow]] = {
+ val partitionPath = FSUtils.getRelativePartitionPath(metaClient.getBasePath, logFile.getPath.getParent)
+ val readerContext = new SparkFileFormatInternalRowReaderContext(parquetReader, Seq.empty, Seq.empty,
+ conf, metaClient.getTableConfig)
+ readerContext.setLatestCommitTime(instant)
+ readerContext.setHasBootstrapBaseFile(false)
+ readerContext.setHasLogFiles(true)
+ readerContext.initRecordMerger(readerProperties)
+ readerContext.setSchemaHandler(
+ new FileGroupReaderSchemaHandler[InternalRow](readerContext, avroSchema, avroSchema,
+ Option.empty(), metaClient.getTableConfig, readerProperties))
+ val recordBuffer = new KeyBasedFileGroupRecordBuffer[InternalRow](readerContext, metaClient,
+ readerContext.getMergeMode, readerProperties, new HoodieReadStats,
+ Option.ofNullable(metaClient.getTableConfig.getPreCombineField), true, Option.empty())
+
+ HoodieMergedLogRecordReader.newBuilder[InternalRow]
+ .withStorage(metaClient.getStorage)
+ .withHoodieReaderContext(readerContext)
+ .withLogFiles(Collections.singletonList(logFile))
+ .withReverseReader(false)
+ .withBufferSize(HoodieMetadataConfig.MAX_READER_BUFFER_SIZE_PROP.defaultValue)
+ .withPartition(partitionPath)
+ .withAllowInflightInstants(true)
+ .withMetaClient(metaClient)
+ .withAllowInflightInstants(true)
Review Comment:
Removed this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2191460353
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
> Don't forget SortedKeyBasedFileGroupRecordBuffer
This is used for MDT right? do we need to log cdc in such scenarios?
> If we really care about a single check that happens for a subset of rows, why isn't the same scrutiny applied to other PRs regardless of whether it is new logic or not
We did it when Lin introduces more merging mode to FG reader, that is why we added the `BufferedRecordMerger` absractions, sub-classing the file group record buffer is just a suggestion, you can think other alternatives to get rid of the new introduced per-row handling and keep the core merging logic clean and clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3049126231
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a79b08ffcf0266bdda5e4faea181fc21e3efcab4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642)
* cbb2380688b2279b75557d443d2e238eb32a1056 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2193285530
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
https://gist.github.com/the-other-tim-brown/ae9e1edc153aa11968b2c522606e2c0c
Here's a gist with my approach and findings. We can see that the check is less than .2 nanoseconds (10,000 checks results in ~600 nanoseconds on average (1500 upper bound) increase when compared to the average of similar code without the check. Also benchmarked the use of Hudi's `Option` class which is even slower which is expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2198994019
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -316,27 +322,57 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
+ if (baseFileUpdateCallback != null && isDeleteAndRecord.getRight() != baseRecord) {
+ // If the record is not the same as the base record, we can emit an update
+ handleBaseFileUpdate(logRecordInfo.getRecordKey(), baseRecord, nextRecord);
+ }
readStats.incrementNumUpdates();
return true;
- } else if (emitDelete) {
- // emit Deletes
- nextRecord = readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey());
- readStats.incrementNumDeletes();
- return nextRecord != null;
} else {
- // not emit Deletes
+ // emit Deletes
+ handleBaseFileDelete(logRecordInfo.getRecordKey(), baseRecord);
readStats.incrementNumDeletes();
- return false;
+ if (emitDelete) {
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey()));
+ return nextRecord != null;
+ } else {
+ return false;
+ }
}
}
// Inserts
- nextRecord = readerContext.seal(baseRecord);
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(baseRecord));
readStats.incrementNumInserts();
return true;
}
+ /**
+ * Applies the final output schema conversion to the buffered record if required. This ensures the records match the requested schema.
+ * @param bufferedRecord the buffered record to convert
+ * @return a new buffered record with the converted record and the proper schema ID set
+ */
+ protected BufferedRecord<T> applyOutputSchemaConversion(BufferedRecord<T> bufferedRecord) {
Review Comment:
The projection will remove the record position field if position based merging is used. I will move it directly into the new code that handles the callback though since I agree the risk outweighs the reward here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3067567025
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 86bcb912ae11da007eb9228660a06a3f22965e27 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714)
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 5540b78becd2ea0cfd5c13732d22ee463392663c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2977424352
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5d73bf8a5fd21d41831c3fe85f54133b30c0c97c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188563335
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
The CDC logic is not really part of merging, why should they be coupled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3091366245
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
}, {
"hash" : "73e6ef97620b31d888b37d02733f12937807d513",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851",
"triggerID" : "73e6ef97620b31d888b37d02733f12937807d513",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 6513ebe37168dc7a9d162963c9aeb246f99ebdb9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818)
* 73e6ef97620b31d888b37d02733f12937807d513 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217104220
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/LogCompactionExecutionHelper.java:
##########
@@ -1,86 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.hudi.table.action.compact;
-
-import org.apache.hudi.client.WriteStatus;
-import org.apache.hudi.common.model.CompactionOperation;
-import org.apache.hudi.common.model.HoodieBaseFile;
-import org.apache.hudi.common.model.HoodieRecordPayload;
-import org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner;
-import org.apache.hudi.common.table.log.block.HoodieLogBlock;
-import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
-import org.apache.hudi.common.table.timeline.HoodieInstant;
-import org.apache.hudi.common.util.Option;
-import org.apache.hudi.common.util.StringUtils;
-import org.apache.hudi.config.HoodieWriteConfig;
-import org.apache.hudi.table.HoodieCompactionHandler;
-import org.apache.hudi.table.HoodieTable;
-
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.util.HashMap;
-import java.util.Iterator;
-import java.util.List;
-import java.util.Map;
-
-/**
- * Strategy class to execute log compaction operations.
- */
-public class LogCompactionExecutionHelper<T extends HoodieRecordPayload, I, K, O>
- extends CompactionExecutionHelper<T, I, K, O> {
-
- private static final Logger LOG = LoggerFactory.getLogger(LogCompactionExecutionHelper.class);
-
- @Override
- protected void transitionRequestedToInflight(HoodieTable table, String logCompactionInstantTime) {
- HoodieActiveTimeline timeline = table.getActiveTimeline();
- HoodieInstant instant = table.getMetaClient().getInstantGenerator().getLogCompactionRequestedInstant(logCompactionInstantTime);
- // Mark instant as compaction inflight
- timeline.transitionLogCompactionRequestedToInflight(instant);
- }
-
- protected String instantTimeToUseForScanning(String logCompactionInstantTime, String maxInstantTime) {
- return logCompactionInstantTime;
- }
-
- protected boolean shouldPreserveCommitMetadata() {
- return true;
- }
-
- @Override
- protected Iterator<List<WriteStatus>> writeFileAndGetWriteStats(HoodieCompactionHandler compactionHandler,
- CompactionOperation operation,
- String instantTime,
- HoodieMergedLogRecordScanner scanner,
- Option<HoodieBaseFile> oldDataFileOpt) throws IOException {
- Map<HoodieLogBlock.HeaderMetadataType, String> header = new HashMap<>();
- header.put(HoodieLogBlock.HeaderMetadataType.COMPACTED_BLOCK_TIMES,
- StringUtils.join(scanner.getValidBlockInstants(), ","));
Review Comment:
Is this handled somewhere else? `HoodieLogBlock.HeaderMetadataType.COMPACTED_BLOCK_TIMES` is only used by the reader side after this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3091473174
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
}, {
"hash" : "73e6ef97620b31d888b37d02733f12937807d513",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851",
"triggerID" : "73e6ef97620b31d888b37d02733f12937807d513",
"triggerType" : "PUSH"
}, {
"hash" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6855",
"triggerID" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"triggerType" : "PUSH"
}, {
"hash" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6856",
"triggerID" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* c9dc3001cfb9e1e542e3e12062f018f6301b8f04 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6855)
* 05d26ce59cd2eee1864bbcca5d8f93236b3043ca Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6856)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217081295
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UnmergedFileGroupRecordBuffer.java:
##########
@@ -52,9 +51,8 @@ public UnmergedFileGroupRecordBuffer(
RecordMergeMode recordMergeMode,
PartialUpdateMode partialUpdateMode,
TypedProperties props,
- HoodieReadStats readStats,
- boolean emitDelete) {
- super(readerContext, hoodieTableMetaClient, recordMergeMode, partialUpdateMode, props, readStats, Option.empty(), emitDelete);
+ HoodieReadStats readStats) {
+ super(readerContext, hoodieTableMetaClient, recordMergeMode, partialUpdateMode, props, readStats, Option.empty(), null);
Review Comment:
The `UnmergedFileGroupRecordBuffer` is used but it overrides the methods that use the update processor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217282265
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -99,6 +99,8 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private final boolean allowInflightInstants;
// Callback to run custom logic on updates to the base files for the file group
private final Option<BaseFileUpdateCallback> fileGroupUpdateCallback;
+ private final boolean enableOptimizedLogBlockScan;
+ private List<String> validBlockInstants = Collections.emptyList();
Review Comment:
can we add some doc to address that it is used for log compaction optimized logBlocks scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2994796921
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408)
* c716bd4c30370148bb32494941fe33e50e6ef8c1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2997578875
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 415a6a4bc909295ba21e97600ddf44d00bcf050b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417)
* f75a7f9495ddde683839d777894769af2887c9a6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3045345191
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8b31c59a60518f198d3d19c734dc378b8fa2ef8b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623)
* 576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3045599949
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3046632296
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0d744b09a8221e626e289097a7a63726d7768259 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640)
* a79b08ffcf0266bdda5e4faea181fc21e3efcab4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3046682979
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0d744b09a8221e626e289097a7a63726d7768259 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640)
* a79b08ffcf0266bdda5e4faea181fc21e3efcab4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2196403849
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -349,11 +385,13 @@ protected boolean hasNextLogRecord() {
while (logRecordIterator.hasNext()) {
BufferedRecord<T> nextRecordInfo = logRecordIterator.next();
if (!nextRecordInfo.isDelete()) {
- nextRecord = nextRecordInfo.getRecord();
+ BufferedRecord<T> convertedBufferedRecord = applyOutputSchemaConversion(nextRecordInfo);
+ nextRecord = convertedBufferedRecord.getRecord();
+ handleBaseFileInsert(nextRecordInfo.getRecordKey(), nextRecord);
readStats.incrementNumInserts();
return true;
} else if (emitDelete) {
- nextRecord = readerContext.getDeleteRow(nextRecordInfo.getRecord(), nextRecordInfo.getRecordKey());
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(nextRecordInfo.getRecord(), nextRecordInfo.getRecordKey()));
Review Comment:
We just igore the deletes from logs, is this behavior in line with before? Ususally these deletes are invalid or probably because of disorder inputs from Network shufflle, not sure if use still want to see these changes to downstream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3042507784
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476)
* bc708ae8514d6e385f7be77c88af33101c488a52 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3042505403
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476)
* bc708ae8514d6e385f7be77c88af33101c488a52 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188528982
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
+ // If the record is not the same as the base record, we can emit an update
+ if (isDeleteAndRecord.getRight() != baseRecord) {
+ callbackOption.ifPresent(callback -> {
+ BufferedRecord<T> mergeResult = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
+ callback.onUpdate(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)), readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(mergeResult)));
+ });
+ }
readStats.incrementNumUpdates();
return true;
- } else if (emitDelete) {
- // emit Deletes
- nextRecord = readerContext.getDeleteRow(isDeleteAndRecord.getRight(), baseRecordInfo.getRecordKey());
- readStats.incrementNumDeletes();
- return nextRecord != null;
} else {
- // not emit Deletes
+ // emit Deletes
+ callbackOption.ifPresent(callback -> callback.onDelete(readerContext.constructHoodieRecord(applyOutputSchemaConversion(baseRecordInfo)),
+ readerContext.constructHoodieRecord(applyOutputSchemaConversion(logRecordInfo))));
Review Comment:
The deletes are not always stored in a delete log block, we can also have merging logic that infers a delete based off of a field like we do for Postgres CDC workloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188532999
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -598,11 +638,13 @@ protected boolean hasNextLogRecord() {
while (logRecordIterator.hasNext()) {
BufferedRecord<T> nextRecordInfo = logRecordIterator.next();
if (!nextRecordInfo.isDelete()) {
- nextRecord = nextRecordInfo.getRecord();
+ BufferedRecord<T> convertedBufferedRecord = applyOutputSchemaConversion(nextRecordInfo);
+ nextRecord = convertedBufferedRecord.getRecord();
+ callbackOption.ifPresent(callback -> callback.onInsert(readerContext.constructHoodieRecord(convertedBufferedRecord)));
readStats.incrementNumInserts();
return true;
} else if (emitDelete) {
- nextRecord = readerContext.getDeleteRow(nextRecordInfo.getRecord(), nextRecordInfo.getRecordKey());
+ nextRecord = applyOutputSchemaConversion(readerContext.getDeleteRow(nextRecordInfo.getRecord(), nextRecordInfo.getRecordKey()));
Review Comment:
The CDC only represents changes to the base file currently in Hudi. It does not capture all the changes that happen so in the case of a delete that happened only in the log files, we will have no changes to the base file and don't need to output anything
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3046629011
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626)
* 0d744b09a8221e626e289097a7a63726d7768259 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640)
* a79b08ffcf0266bdda5e4faea181fc21e3efcab4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3049116741
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a79b08ffcf0266bdda5e4faea181fc21e3efcab4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642)
* cbb2380688b2279b75557d443d2e238eb32a1056 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2194119045
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
If the performance is almost the same (here we are talking about nanosecond differences), I think it's OK to keep the current shape of the PR and prioritize the overall goal of unifying the CDC logic through the new FGReader. We should keep the overall goal in mind without getting lost in a particular debate where we already have data points that the current change does not bring regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2191333043
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
The maintenance should become easier and we avoid the per-row ramifications, -1 of course.
I will not continue this review until this refactoring got finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3049322318
@danny0405 Can you please list out your remaining concerns as a checklist..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217081005
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+
+import java.util.function.UnaryOperator;
+
+/**
+ * Interface used within the {@link HoodieFileGroupReader<T>} for processing updates to records in Merge-on-Read tables.
+ * Note that the updates are always relative to the base file's current state.
+ * @param <T> the engine specific record type
+ */
+public interface UpdateProcessor<T> {
+ /**
+ * Processes the update to the record. If the update should not be returned to the caller, the method should return null.
+ * @param recordKey the key of the record being updated
+ * @param previousRecord the previous version of the record, or null if there is no previous value
+ * @param currentRecord the current version of the record after merging with the existing record, if any exists
+ * @param isDelete a flag indicating whether the merge resulted in a delete operation
+ * @return the processed record, or null if the record should not be returned to the caller
+ */
+ T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete);
+
+ static <T> UpdateProcessor<T> create(HoodieReadStats readStats, HoodieReaderContext<T> readerContext,
+ boolean emitDeletes, Option<BaseFileUpdateCallback> updateCallback) {
+ UpdateProcessor<T> handler = new StandardUpdateProcessor<>(readStats, readerContext, emitDeletes);
+ if (updateCallback.isPresent()) {
+ return new CallbackProcessor<>(updateCallback.get(), handler, readerContext);
+ }
+ return handler;
+ }
+
+ /**
+ * A standard update processor that increments the read stats and returns the record if applicable.
+ * @param <T> the engine specific record type
+ */
+ class StandardUpdateProcessor<T> implements UpdateProcessor<T> {
+ protected final HoodieReadStats readStats;
+ protected final HoodieReaderContext<T> readerContext;
+ protected final boolean emitDeletes;
+
+ public StandardUpdateProcessor(HoodieReadStats readStats, HoodieReaderContext<T> readerContext,
+ boolean emitDeletes) {
+ this.readStats = readStats;
+ this.readerContext = readerContext;
+ this.emitDeletes = emitDeletes;
+ }
+
+ @Override
+ public T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete) {
+ if (isDelete) {
+ readStats.incrementNumDeletes();
+ if (emitDeletes) {
+ return readerContext.getDeleteRow(currentRecord, recordKey);
+ }
+ return null;
+ } else {
+ if (previousRecord != null) {
+ readStats.incrementNumUpdates();
+ } else {
+ readStats.incrementNumInserts();
+ }
+ return readerContext.seal(currentRecord);
+ }
+ }
+ }
+
+ /**
+ * A processor that wraps the standard update processor and invokes a customizable callback for each update.
+ * @param <T> the engine specific record type
+ */
+ class CallbackProcessor<T> implements UpdateProcessor<T> {
+ private final BaseFileUpdateCallback callback;
+ private final UpdateProcessor<T> delegate;
+ private final HoodieReaderContext<T> readerContext;
+ private final Option<UnaryOperator<T>> outputConverter;
+ private final Schema requestedSchema;
+
+ public CallbackProcessor(BaseFileUpdateCallback callback, UpdateProcessor<T> delegate,
+ HoodieReaderContext<T> readerContext) {
+ this.callback = callback;
+ this.delegate = delegate;
+ this.readerContext = readerContext;
+ this.outputConverter = readerContext.getSchemaHandler().getOutputConverter();
+ this.requestedSchema = readerContext.getSchemaHandler().getRequestedSchema();
+ }
+
+ @Override
+ public T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete) {
+ T result = delegate.processUpdate(recordKey, previousRecord, currentRecord, isDelete);
+
+ if (isDelete) {
+ callback.onDelete(recordKey, convertOutput(previousRecord));
+ } else if (previousRecord != null && previousRecord != currentRecord) {
+ callback.onUpdate(recordKey, convertOutput(previousRecord), convertOutput(currentRecord));
+ } else {
+ callback.onInsert(recordKey, convertOutput(currentRecord));
+ }
+ return result;
+ }
+
+ private GenericRecord convertOutput(T record) {
+ T convertedRecord = outputConverter.map(converter -> record == null ? null : converter.apply(record)).orElse(record);
+ return convertedRecord == null ? null : readerContext.convertToAvroRecord(convertedRecord, requestedSchema);
+ }
Review Comment:
True, let me move it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3091360832
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
}, {
"hash" : "73e6ef97620b31d888b37d02733f12937807d513",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "73e6ef97620b31d888b37d02733f12937807d513",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 6513ebe37168dc7a9d162963c9aeb246f99ebdb9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818)
* 73e6ef97620b31d888b37d02733f12937807d513 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217081072
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+
+import java.util.function.UnaryOperator;
+
+/**
+ * Interface used within the {@link HoodieFileGroupReader<T>} for processing updates to records in Merge-on-Read tables.
+ * Note that the updates are always relative to the base file's current state.
+ * @param <T> the engine specific record type
+ */
+public interface UpdateProcessor<T> {
+ /**
+ * Processes the update to the record. If the update should not be returned to the caller, the method should return null.
+ * @param recordKey the key of the record being updated
+ * @param previousRecord the previous version of the record, or null if there is no previous value
+ * @param currentRecord the current version of the record after merging with the existing record, if any exists
+ * @param isDelete a flag indicating whether the merge resulted in a delete operation
+ * @return the processed record, or null if the record should not be returned to the caller
+ */
+ T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete);
+
+ static <T> UpdateProcessor<T> create(HoodieReadStats readStats, HoodieReaderContext<T> readerContext,
+ boolean emitDeletes, Option<BaseFileUpdateCallback> updateCallback) {
Review Comment:
that is correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217082534
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UpdateProcessor.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericRecord;
+
+import java.util.function.UnaryOperator;
+
+/**
+ * Interface used within the {@link HoodieFileGroupReader<T>} for processing updates to records in Merge-on-Read tables.
+ * Note that the updates are always relative to the base file's current state.
+ * @param <T> the engine specific record type
+ */
+public interface UpdateProcessor<T> {
+ /**
+ * Processes the update to the record. If the update should not be returned to the caller, the method should return null.
+ * @param recordKey the key of the record being updated
+ * @param previousRecord the previous version of the record, or null if there is no previous value
+ * @param currentRecord the current version of the record after merging with the existing record, if any exists
+ * @param isDelete a flag indicating whether the merge resulted in a delete operation
+ * @return the processed record, or null if the record should not be returned to the caller
+ */
+ T processUpdate(String recordKey, T previousRecord, T currentRecord, boolean isDelete);
+
+ static <T> UpdateProcessor<T> create(HoodieReadStats readStats, HoodieReaderContext<T> readerContext,
+ boolean emitDeletes, Option<BaseFileUpdateCallback> updateCallback) {
Review Comment:
yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217084039
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/CDCFileGroupIterator.scala:
##########
@@ -414,11 +385,10 @@ class CDCFileGroupIterator(split: HoodieCDCFileGroupSplit,
}
// load afterFileSlice to afterImageRecords
if (currentCDCFileSplit.getAfterFileSlice.isPresent) {
- val iter = loadFileSlice(currentCDCFileSplit.getAfterFileSlice.get())
+ val iter = loadFileSliceWithKeys(currentCDCFileSplit.getAfterFileSlice.get())
afterImageRecords.clear()
- iter.foreach { row =>
- val key = getRecordKey(row)
- afterImageRecords.put(key, row.copy())
+ iter.foreach { tuple =>
+ afterImageRecords.put(tuple._1, tuple._2.copy())
Review Comment:
nit: is `tuple._2.copy()` still needed as it goes through FGReader now?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/cdc/CDCFileGroupIterator.scala:
##########
@@ -430,10 +400,7 @@ class CDCFileGroupIterator(split: HoodieCDCFileGroupSplit,
if (currentCDCFileSplit.getBeforeFileSlice.isPresent) {
loadBeforeFileSliceIfNeeded(currentCDCFileSplit.getBeforeFileSlice.get)
}
- recordIter = beforeImageRecords.values.map { record =>
- deserialize(record)
- }.iterator
- beforeImageRecords.clear()
Review Comment:
Is `beforeImageRecords` still cleared somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3085698283
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* ff24a54cfc19c8a89fc445ba13fab96b8b4085f4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812)
* 6513ebe37168dc7a9d162963c9aeb246f99ebdb9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2994920923
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c716bd4c30370148bb32494941fe33e50e6ef8c1 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3067538256
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbb2380688b2279b75557d443d2e238eb32a1056 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649)
* 86bcb912ae11da007eb9228660a06a3f22965e27 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2997754083
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f75a7f9495ddde683839d777894769af2887c9a6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526][DRAFT] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-2994614298
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b4765033b87250ef5654d2d71818ae398fa7e91f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3042529337
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bc708ae8514d6e385f7be77c88af33101c488a52 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622)
* 8b31c59a60518f198d3d19c734dc378b8fa2ef8b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2193861330
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
@danny0405 what is the ramification you are worried about? Is it the less than 1ms increase in total runtime per 5M unique log records or something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2191369244
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
The maintenance is not going to become any easier. Every record buffer implementation will also need a CDC version. How is that easier to maintain?
There will always be some code that needs to run per row when we are merging. Currently this only runs when there are updates, deletes, or new log records. That is not `per-row` exactly. If you are so worried about a null check, why aren't you worried about [auto-boxing performance degradation](https://github.com/apache/hudi/pull/13511/files#diff-08630014159096d27df01401a0fe2178f535d90045d09c1b5dfe8e0cefb7ee66R66) and the [per merge checks](https://github.com/apache/hudi/pull/13511/files#diff-44ea0a598fb8c09126a1dcccad5c3f572c8aadae48c18dcc800e977a4cfb1c5fR313) that you recently approved and merged? Where is the line drawn? It seems arbitrary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188806198
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
In any case, please stop introducing row-level ramifications continuously if the `cdc` logging is deterministic per-query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2188898741
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
> The finalMerge is never called in this case.
The are two cases you plugin in the cdc logging function:
1. `FileGroupRecordBuffer#hasNextBaseRecord`;
2. `FileGroupRecordBuffer#hasNextBaseRecord`.
The `finalMerge` plays a role in the 1st case, but you are right, in case2, we also need to trigger the cdc logging.
Let's introduce another two `record buffers` to override the two cases when an update callback is there.
> Can you explain what you mean by this please?
Like the new introduced check per-row:
```java
if (isDeleteAndRecord.getRight() != baseRecord) {
callbackOption.ifPresent(
...
callbackOption.ifPresent(
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3055162565
> @danny0405 Can you please list out your remaining concerns as a checklist..
Left some comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2198990306
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/compact/CompactOperator.java:
##########
@@ -163,34 +161,23 @@ private void doCompaction(String instantTime,
// schema evolution
CompactionUtil.setAvroSchema(writeConfig, metaClient);
List<WriteStatus> writeStatuses = compactor.compact(
- flinkTable,
writeConfig,
compactionOperation,
instantTime,
flinkTable.getTaskContextSupplier(),
- readerContextOpt,
+ readerContext,
flinkTable);
compactionMetrics.endCompaction();
collector.collect(new CompactionCommitEvent(instantTime, compactionOperation.getFileId(), writeStatuses, taskID));
}
- private Option<HoodieReaderContext<?>> initReaderContext(HoodieFlinkWriteClient<?> writeClient) {
+ private HoodieReaderContext<?> initReaderContext(HoodieFlinkWriteClient<?> writeClient) {
HoodieTableMetaClient metaClient = flinkTable.getMetaClient();
- boolean useFileGroupReaderBasedCompaction = !metaClient.isMetadataTable()
- && writeClient.getConfig().getBooleanOrDefault(HoodieReaderConfig.FILE_GROUP_READER_ENABLED)
- && writeClient.getConfig().populateMetaFields() // Virtual key support by fg reader is not ready
- && !(metaClient.getTableConfig().isCDCEnabled() && writeClient.getConfig().isYieldingPureLogForMor()); // do not support produce cdc log during fg reader
- if (useFileGroupReaderBasedCompaction) {
- // CAUTION: reuse the meta client so that the timeline is updated
- Supplier<InternalSchemaManager> internalSchemaManager = () -> InternalSchemaManager.get(conf, metaClient);
- // initialize storage conf lazily.
- StorageConfiguration<?> readerConf = writeClient.getEngineContext().getStorageConf();
- return Option.of(new FlinkRowDataReaderContext(readerConf, internalSchemaManager, Collections.emptyList(), metaClient.getTableConfig(), Option.empty()));
- } else {
- // always using avro record merger for legacy compaction since log scanner do not support rowdata reading yet.
- writeClient.getConfig().setRecordMergerClass(HoodieAvroRecordMerger.class.getName());
- }
- return Option.empty();
+ // CAUTION: reuse the meta client so that the timeline is updated
+ Supplier<InternalSchemaManager> internalSchemaManager = () -> InternalSchemaManager.get(conf, metaClient);
+ // initialize storage conf lazily.
Review Comment:
We're switching all readers to go through FileGroupReader for consistency as part of the 1.1.0 release goals so that is why I cleaned this up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217081796
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/UnmergedFileGroupRecordBuffer.java:
##########
@@ -52,9 +51,8 @@ public UnmergedFileGroupRecordBuffer(
RecordMergeMode recordMergeMode,
PartialUpdateMode partialUpdateMode,
TypedProperties props,
- HoodieReadStats readStats,
- boolean emitDelete) {
- super(readerContext, hoodieTableMetaClient, recordMergeMode, partialUpdateMode, props, readStats, Option.empty(), emitDelete);
+ HoodieReadStats readStats) {
+ super(readerContext, hoodieTableMetaClient, recordMergeMode, partialUpdateMode, props, readStats, Option.empty(), null);
Review Comment:
yes, unmerged read is not related with cdc logging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3091666530
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
}, {
"hash" : "73e6ef97620b31d888b37d02733f12937807d513",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851",
"triggerID" : "73e6ef97620b31d888b37d02733f12937807d513",
"triggerType" : "PUSH"
}, {
"hash" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6855",
"triggerID" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"triggerType" : "PUSH"
}, {
"hash" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6856",
"triggerID" : "05d26ce59cd2eee1864bbcca5d8f93236b3043ca",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 05d26ce59cd2eee1864bbcca5d8f93236b3043ca Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6856)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2217344810
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/FileGroupReaderBasedAppendHandle.java:
##########
@@ -77,9 +79,11 @@ public void doAppend() {
props.put(HoodieMemoryConfig.MAX_MEMORY_FOR_MERGE.key(), String.valueOf(maxMemoryPerCompaction));
// Initializes the record iterator
try (HoodieFileGroupReader<T> fileGroupReader = HoodieFileGroupReader.<T>newBuilder().withReaderContext(readerContext).withHoodieTableMetaClient(hoodieTable.getMetaClient())
- .withLatestCommitTime(instantTime).withFileSlice(fileSlice).withDataSchema(writeSchemaWithMetaFields).withRequestedSchema(writeSchemaWithMetaFields)
+ .withLatestCommitTime(instantTime).withFileSlice(fileSlice).withDataSchema(writeSchemaWithMetaFields).withRequestedSchema(writeSchemaWithMetaFields).withEnableOptimizedLogBlockScan(true)
.withInternalSchema(internalSchemaOption).withProps(props).withShouldUseRecordPosition(usePosition).withSortOutput(hoodieTable.requireSortedRecords()).build()) {
recordItr = fileGroupReader.getClosableHoodieRecordIterator();
+ header.put(HoodieLogBlock.HeaderMetadataType.COMPACTED_BLOCK_TIMES,
+ StringUtils.join(fileGroupReader.getValidBlockInstants(), ","));
Review Comment:
Is there any test coverage around this? Seems that the log comapction tests pass without the change. We can add validation on the log header to make sure `COMPACTED_BLOCK_TIMES` is added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3091372541
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
}, {
"hash" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6812",
"triggerID" : "ff24a54cfc19c8a89fc445ba13fab96b8b4085f4",
"triggerType" : "PUSH"
}, {
"hash" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818",
"triggerID" : "6513ebe37168dc7a9d162963c9aeb246f99ebdb9",
"triggerType" : "PUSH"
}, {
"hash" : "73e6ef97620b31d888b37d02733f12937807d513",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851",
"triggerID" : "73e6ef97620b31d888b37d02733f12937807d513",
"triggerType" : "PUSH"
}, {
"hash" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c9dc3001cfb9e1e542e3e12062f018f6301b8f04",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 6513ebe37168dc7a9d162963c9aeb246f99ebdb9 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6818)
* 73e6ef97620b31d888b37d02733f12937807d513 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6851)
* c9dc3001cfb9e1e542e3e12062f018f6301b8f04 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3067683331
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
}, {
"hash" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716",
"triggerID" : "5540b78becd2ea0cfd5c13732d22ee463392663c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 86bcb912ae11da007eb9228660a06a3f22965e27 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714)
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
* 5540b78becd2ea0cfd5c13732d22ee463392663c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6716)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #13444:
URL: https://github.com/apache/hudi/pull/13444#discussion_r2192766677
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -565,27 +570,62 @@ protected boolean hasNextBaseRecord(T baseRecord, BufferedRecord<T> logRecordInf
Pair<Boolean, T> isDeleteAndRecord = merge(baseRecordInfo, logRecordInfo);
if (!isDeleteAndRecord.getLeft()) {
// Updates
- nextRecord = readerContext.seal(isDeleteAndRecord.getRight());
+ nextRecord = readerContext.seal(applyOutputSchemaConversion(isDeleteAndRecord.getRight()));
Review Comment:
Let me dig deeper here.. any debate about performance needs some micro benchmarks at the least.. Can you all dump any real test results we have around the points raised here..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3067561776
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
}, {
"hash" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "15a27a8e0312a55fc7e536c4bf91d15edd3294a0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 86bcb912ae11da007eb9228660a06a3f22965e27 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714)
* 15a27a8e0312a55fc7e536c4bf91d15edd3294a0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9526] Use HoodieFileGroupReader throughout the CDC flow [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13444:
URL: https://github.com/apache/hudi/pull/13444#issuecomment-3067542470
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6240",
"triggerID" : "5d73bf8a5fd21d41831c3fe85f54133b30c0c97c",
"triggerType" : "PUSH"
}, {
"hash" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6404",
"triggerID" : "b4765033b87250ef5654d2d71818ae398fa7e91f",
"triggerType" : "PUSH"
}, {
"hash" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6408",
"triggerID" : "b19852ac0998ec6c48a8a66a2ebcd4ff4025c01a",
"triggerType" : "PUSH"
}, {
"hash" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6410",
"triggerID" : "c716bd4c30370148bb32494941fe33e50e6ef8c1",
"triggerType" : "PUSH"
}, {
"hash" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6417",
"triggerID" : "415a6a4bc909295ba21e97600ddf44d00bcf050b",
"triggerType" : "PUSH"
}, {
"hash" : "f75a7f9495ddde683839d777894769af2887c9a6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6419",
"triggerID" : "f75a7f9495ddde683839d777894769af2887c9a6",
"triggerType" : "PUSH"
}, {
"hash" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6476",
"triggerID" : "d5e4fbf3f0af6e01c8aab3911f5d2064bb4bfff1",
"triggerType" : "PUSH"
}, {
"hash" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6622",
"triggerID" : "bc708ae8514d6e385f7be77c88af33101c488a52",
"triggerType" : "PUSH"
}, {
"hash" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6623",
"triggerID" : "8b31c59a60518f198d3d19c734dc378b8fa2ef8b",
"triggerType" : "PUSH"
}, {
"hash" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6626",
"triggerID" : "576a6e1a94b0dc173c70378056d8b9ddf5d9e5e4",
"triggerType" : "PUSH"
}, {
"hash" : "0d744b09a8221e626e289097a7a63726d7768259",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6640",
"triggerID" : "0d744b09a8221e626e289097a7a63726d7768259",
"triggerType" : "PUSH"
}, {
"hash" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6642",
"triggerID" : "a79b08ffcf0266bdda5e4faea181fc21e3efcab4",
"triggerType" : "PUSH"
}, {
"hash" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649",
"triggerID" : "cbb2380688b2279b75557d443d2e238eb32a1056",
"triggerType" : "PUSH"
}, {
"hash" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714",
"triggerID" : "86bcb912ae11da007eb9228660a06a3f22965e27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cbb2380688b2279b75557d443d2e238eb32a1056 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6649)
* 86bcb912ae11da007eb9228660a06a3f22965e27 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6714)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org