You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2025/08/02 01:01:04 UTC
[PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
nsivabalan opened a new pull request, #13670:
URL: https://github.com/apache/hudi/pull/13670
### Change Logs
- Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration
- Added FileGroupRecordBufferLoader.createDefaultWithRecords() which takes in a map of records to assist with this case.
- DefaultFileGroupRecordBufferLoader is enhances to work either with log files or map of records.
### Impact
Assist with COW merge handle migration
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148768008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e47e1762e86960423e95b5f010001050d803477 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487)
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250301160
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.PartialUpdateMode;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.table.read.UpdateProcessor;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> implements FileGroupRecordBufferLoader<T> {
+
+ private final Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords;
+
+ private RecordsBasedFileGroupRecordBufferLoader(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
+ super();
+ this.toBeMergedRecords = toBeMergedRecords;
+ }
+
+ static <T> RecordsBasedFileGroupRecordBufferLoader<T> getInstance(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
+ return new RecordsBasedFileGroupRecordBufferLoader<>(toBeMergedRecords);
+ }
+
+ @Override
+ public Option<Pair<HoodieFileGroupRecordBuffer<T>, List<String>>> getRecordBuffer(HoodieReaderContext<T> readerContext,
+ HoodieStorage storage,
Review Comment:
I dont' see any issues in my intellij. can you check the latest patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250579608
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -81,4 +89,13 @@ public long getLength() {
public boolean isParquetBaseFile() {
return baseFileOption.map(baseFile -> HoodieFileFormat.fromFileExtension(baseFile.getStoragePath().getFileExtension()) == HoodieFileFormat.PARQUET).orElse(false);
}
+
+ public boolean hasNoRecordsToMerge() {
+ return this.logFiles.isEmpty() && recordIterator.isEmpty();
+ }
+
+ public Option<Iterator<Pair<Serializable, BufferedRecord>>> getRecordIterator() {
+ ValidationUtils.checkArgument(recordIterator.isPresent(), "Record iterator is not initialized");
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250268892
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -128,14 +128,14 @@ private HoodieFileGroupReader(HoodieReaderContext<T> readerContext, HoodieStorag
*/
private void initRecordIterators() throws IOException {
ClosableIterator<T> iter = makeBaseFileIterator();
- if (inputSplit.getLogFiles().isEmpty()) {
+ Option<Pair<HoodieFileGroupRecordBuffer<T>, List<String>>> initializationResultOpt = recordBufferLoader.getRecordBuffer(
+ readerContext, storage, inputSplit, orderingFieldNames, metaClient, props, readerParameters, readStats, fileGroupUpdateCallback);
Review Comment:
I agree w/ you take here. we can do it as a follow up patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148969938
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498)
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149174532
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 2c95430bacfaf2805a59c420592cf12a8893b5c4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501)
* 83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250414587
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.PartialUpdateMode;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.table.read.UpdateProcessor;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> implements FileGroupRecordBufferLoader<T> {
+
+ private final Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords;
+
+ private RecordsBasedFileGroupRecordBufferLoader(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
+ super();
+ this.toBeMergedRecords = toBeMergedRecords;
+ }
+
+ static <T> RecordsBasedFileGroupRecordBufferLoader<T> getInstance(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
+ return new RecordsBasedFileGroupRecordBufferLoader<>(toBeMergedRecords);
+ }
Review Comment:
all instance creation are currently here and hence followed suit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250441496
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +41,30 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Pair<Serializable, BufferedRecord>>> recordIterator;
Review Comment:
Fixed it to be `Iterator<BufferedRecord>` which is more efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149633271
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
}, {
"hash" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507",
"triggerID" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"triggerType" : "PUSH"
}, {
"hash" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7510",
"triggerID" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"triggerType" : "PUSH"
}, {
"hash" : "f87ee945fcedab791ee4864fb321bebff880d7e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f87ee945fcedab791ee4864fb321bebff880d7e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
* 05421e78b81645df80c94b96a5957a56b1fb4dae Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507)
* 8a366f32a9f3fb18a68691c540224aa2f5784621 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7510)
* f87ee945fcedab791ee4864fb321bebff880d7e1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148837725
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250387786
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +41,30 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Pair<Serializable, BufferedRecord>>> recordIterator;
Review Comment:
Cur code seems more clean to me. Prefer to leave it as is.
The constructor seems tight where only one of log files or record iterator can be instantiated. internally we can keep them separate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149001909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498)
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2251154357
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/buffer/BaseTestFileGroupRecordBuffer.java:
##########
@@ -54,21 +51,21 @@ protected static GenericRecord createTestRecord(String recordKey, int counter, l
return record;
}
- protected static Map<Serializable, BufferedRecord> convertToBufferedRecordsMap(List<IndexedRecord> indexedRecords,
+ protected static List<BufferedRecord> convertToBufferedRecordsMap(List<IndexedRecord> indexedRecords,
Review Comment:
convertToBufferedRecordsMap -> convertToBufferedRecordList? Also fix the intentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3151968161
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
}, {
"hash" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507",
"triggerID" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"triggerType" : "PUSH"
}, {
"hash" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7510",
"triggerID" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"triggerType" : "PUSH"
}, {
"hash" : "f87ee945fcedab791ee4864fb321bebff880d7e1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7513",
"triggerID" : "f87ee945fcedab791ee4864fb321bebff880d7e1",
"triggerType" : "PUSH"
}, {
"hash" : "a2d81395369cd2ec368413e34bcfbb5df656bc0a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a2d81395369cd2ec368413e34bcfbb5df656bc0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
* f87ee945fcedab791ee4864fb321bebff880d7e1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7513)
* a2d81395369cd2ec368413e34bcfbb5df656bc0a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2249057676
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/DefaultFileGroupRecordBufferLoader.java:
##########
@@ -42,11 +48,29 @@
class DefaultFileGroupRecordBufferLoader<T> extends LogScanningRecordBufferLoader implements FileGroupRecordBufferLoader<T> {
Review Comment:
Instead of making this class comply with the new use case, can you make a new loader that works with an incoming stream of records?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -70,6 +70,7 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private final HoodieReaderContext<T> readerContext;
private final HoodieTableMetaClient metaClient;
private final InputSplit inputSplit;
+ private final boolean couldHaveLogFiles;
Review Comment:
Can we instead move this into the InputSplit logic? For example, have a parameter like "requires base file read" instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148722807
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa86ba222bebce5aa11a508664acb3943f84710f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485)
* 2e47e1762e86960423e95b5f010001050d803477 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250172204
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> extends DefaultFileGroupRecordBufferLoader<T> {
Review Comment:
This does not need to extend `DefaultFileGroupRecordBuffer`, it can simply implement the interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148875788
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489)
* 86072f4c0e6e2fc2667123886be80b32b9449e75 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495)
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250257233
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBufferLoader.java:
##########
@@ -52,6 +56,10 @@ static <T> FileGroupRecordBufferLoader<T> createDefault() {
return DefaultFileGroupRecordBufferLoader.getInstance();
}
+ static <T> FileGroupRecordBufferLoader<T> createRecordsBasedLoader(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
Review Comment:
```suggestion
static <T> FileGroupRecordBufferLoader<T> createRecordsBasedLoader(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> recordIterator) {
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBufferLoader.java:
##########
@@ -52,6 +56,10 @@ static <T> FileGroupRecordBufferLoader<T> createDefault() {
return DefaultFileGroupRecordBufferLoader.getInstance();
}
+ static <T> FileGroupRecordBufferLoader<T> createRecordsBasedLoader(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
Review Comment:
Rename the method based on the class renaming too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250292915
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +39,25 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Map.Entry<Serializable, BufferedRecord>>> inputs;
private final String partitionPath;
// Byte offset to start reading from the base file
private final long start;
// Length of bytes to read from the base file
private final long length;
- InputSplit(Option<HoodieBaseFile> baseFileOption, Stream<HoodieLogFile> logFiles, String partitionPath, long start, long length) {
+ InputSplit(Option<HoodieBaseFile> baseFileOption, Stream<HoodieLogFile> logFiles, String partitionPath, long start, long length,
Review Comment:
I am taking in Either as an arg.
`Either<Stream<HoodieLogFile>,Iterator<Pair<Serializable, BufferedRecord>>> logFilesOrRecordIterator`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149005875
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498)
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 2c95430bacfaf2805a59c420592cf12a8893b5c4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149299058
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
}, {
"hash" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507",
"triggerID" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504)
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
* 05421e78b81645df80c94b96a5957a56b1fb4dae Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148873769
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489)
* 86072f4c0e6e2fc2667123886be80b32b9449e75 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250287202
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/StreamingFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.PartialUpdateMode;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.table.read.UpdateProcessor;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in an iterator records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ *
+ * @param <T> Engine native presentation of the record.
+ */
+public class StreamingFileGroupRecordBufferLoader<T> implements FileGroupRecordBufferLoader<T> {
+
+ private final Iterator<Pair<Serializable, BufferedRecord<T>>> recordIterator;
+
+ private StreamingFileGroupRecordBufferLoader(Iterator<Pair<Serializable, BufferedRecord<T>>> recordIterator) {
+ super();
+ this.recordIterator = recordIterator;
+ }
+
+ static <T> StreamingFileGroupRecordBufferLoader<T> getInstance(Iterator<Pair<Serializable, BufferedRecord<T>>> recordIterator) {
+ return new StreamingFileGroupRecordBufferLoader<>(recordIterator);
+ }
+
+ @Override
+ public Option<Pair<HoodieFileGroupRecordBuffer<T>, List<String>>> getRecordBuffer(HoodieReaderContext<T> readerContext,
+ HoodieStorage storage,
+ InputSplit inputSplit,
+ List<String> orderingFieldNames,
+ HoodieTableMetaClient hoodieTableMetaClient,
+ TypedProperties props,
+ ReaderParameters readerParameters,
+ HoodieReadStats readStats,
+ Option<BaseFileUpdateCallback<T>> fileGroupUpdateCallback) {
+ PartialUpdateMode partialUpdateMode = hoodieTableMetaClient.getTableConfig().getPartialUpdateMode();
+ UpdateProcessor<T> updateProcessor = UpdateProcessor.create(readStats, readerContext, readerParameters.emitDeletes(), fileGroupUpdateCallback);
+ FileGroupRecordBuffer<T> recordBuffer;
+ if (readerParameters.sortOutputs()) {
+ recordBuffer = new SortedKeyBasedFileGroupRecordBuffer<>(
+ readerContext, hoodieTableMetaClient, readerContext.getMergeMode(), partialUpdateMode, props, orderingFieldNames, updateProcessor);
+ } else {
+ recordBuffer = new KeyBasedFileGroupRecordBuffer<>(
+ readerContext, hoodieTableMetaClient, readerContext.getMergeMode(), partialUpdateMode, props, orderingFieldNames, updateProcessor);
+ }
+ while (recordIterator.hasNext()) {
+ Map.Entry<Serializable, BufferedRecord<T>> entry = recordIterator.next();
Review Comment:
yes, fixing it in next commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148997916
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498)
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148989425
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498)
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149016322
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498)
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 2c95430bacfaf2805a59c420592cf12a8893b5c4 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250381531
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -81,4 +88,12 @@ public long getLength() {
public boolean isParquetBaseFile() {
return baseFileOption.map(baseFile -> HoodieFileFormat.fromFileExtension(baseFile.getStoragePath().getFileExtension()) == HoodieFileFormat.PARQUET).orElse(false);
}
+
+ public boolean hasNoRecordsToMerge() {
+ return this.logFiles.isEmpty() && recordIterator.isEmpty();
+ }
+
+ public Option<Iterator<Pair<Serializable, BufferedRecord>>> getRecordIterator() {
+ return this.recordIterator;
+ }
Review Comment:
In this way we can catch if `getLogFiles()` and `getRecordIterator()` are called in the wrong place.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -81,4 +88,12 @@ public long getLength() {
public boolean isParquetBaseFile() {
return baseFileOption.map(baseFile -> HoodieFileFormat.fromFileExtension(baseFile.getStoragePath().getFileExtension()) == HoodieFileFormat.PARQUET).orElse(false);
}
+
+ public boolean hasNoRecordsToMerge() {
+ return this.logFiles.isEmpty() && recordIterator.isEmpty();
Review Comment:
```suggestion
return this.recordsToMerge.isLeft() && this.recordsToMerge.asLeft().isEmpty();
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +41,30 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Pair<Serializable, BufferedRecord>>> recordIterator;
Review Comment:
Merge `logFiles` and `recordIterator` into `private final Either<List<HoodieLogFile>, Iterator<Pair<Serializable, BufferedRecord>>> recordsToMerge`.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -81,4 +88,12 @@ public long getLength() {
public boolean isParquetBaseFile() {
return baseFileOption.map(baseFile -> HoodieFileFormat.fromFileExtension(baseFile.getStoragePath().getFileExtension()) == HoodieFileFormat.PARQUET).orElse(false);
}
+
+ public boolean hasNoRecordsToMerge() {
+ return this.logFiles.isEmpty() && recordIterator.isEmpty();
+ }
+
+ public Option<Iterator<Pair<Serializable, BufferedRecord>>> getRecordIterator() {
+ return this.recordIterator;
+ }
Review Comment:
```suggestion
public Iterator<Pair<Serializable, BufferedRecord>> getRecordIterator() {
ValidationUtils.checkArgument(recordsToMerge.isRight(), "...");
return this.recordsToMerge.asRight();
}
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -81,4 +88,12 @@ public long getLength() {
public boolean isParquetBaseFile() {
return baseFileOption.map(baseFile -> HoodieFileFormat.fromFileExtension(baseFile.getStoragePath().getFileExtension()) == HoodieFileFormat.PARQUET).orElse(false);
}
+
+ public boolean hasNoRecordsToMerge() {
+ return this.logFiles.isEmpty() && recordIterator.isEmpty();
+ }
+
+ public Option<Iterator<Pair<Serializable, BufferedRecord>>> getRecordIterator() {
+ return this.recordIterator;
+ }
Review Comment:
Do the same for `getLogFiles` to validate and return `this.recordsToMerge.asLeft()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250402249
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +39,25 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Map.Entry<Serializable, BufferedRecord>>> inputs;
private final String partitionPath;
// Byte offset to start reading from the base file
private final long start;
// Length of bytes to read from the base file
private final long length;
- InputSplit(Option<HoodieBaseFile> baseFileOption, Stream<HoodieLogFile> logFiles, String partitionPath, long start, long length) {
+ InputSplit(Option<HoodieBaseFile> baseFileOption, Stream<HoodieLogFile> logFiles, String partitionPath, long start, long length,
Review Comment:
Usually the caller should not use `Either` when constructing the instance to reduce complexity, and rely on builder to pass in necessary components (see the builder example here: https://github.com/apache/hudi/blob/f51c36fd64398718c039b26c15767e793b82904e/hudi-common/src/main/java/org/apache/hudi/io/storage/HFileReaderFactory.java#L89). Since there is no builder here, it's ok for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149180981
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504)
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250578252
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/buffer/TestKeyBasedFileGroupRecordBuffer.java:
##########
@@ -146,11 +151,54 @@ void readWithEventTimeOrderingAndDeleteBlock() throws IOException {
assertEquals(2, readStats.getNumUpdates());
}
+ @Test
+ void readWithEventTimeOrderingWithRecords() throws IOException {
Review Comment:
pushed an update.
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/buffer/TestKeyBasedFileGroupRecordBuffer.java:
##########
@@ -146,11 +151,54 @@ void readWithEventTimeOrderingAndDeleteBlock() throws IOException {
assertEquals(2, readStats.getNumUpdates());
}
+ @Test
+ void readWithEventTimeOrderingWithRecords() throws IOException {
+ HoodieReadStats readStats = new HoodieReadStats();
+ TypedProperties properties = new TypedProperties();
+ properties.setProperty(HoodieTableConfig.PRECOMBINE_FIELDS.key(), "ts");
+ properties.setProperty(DELETE_KEY, "counter");
+ properties.setProperty(DELETE_MARKER, "3");
+ HoodieTableConfig tableConfig = mock(HoodieTableConfig.class);
+ when(tableConfig.getRecordKeyFields()).thenReturn(Option.of(new String[] {"record_key"}));
+ StorageConfiguration<?> storageConfiguration = mock(StorageConfiguration.class);
+ HoodieReaderContext<IndexedRecord> readerContext = new HoodieAvroReaderContext(storageConfiguration, tableConfig, Option.empty(), Option.empty());
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(false);
+ FileGroupReaderSchemaHandler schemaHandler = new FileGroupReaderSchemaHandler(readerContext, SCHEMA, SCHEMA, Option.empty(), tableConfig,
+ properties);
+ readerContext.setSchemaHandler(schemaHandler);
+ Map<Serializable, BufferedRecord> inputRecords = convertToBufferedRecordsMap(Arrays.asList(testRecord1UpdateWithSameTime, testRecord2Update, testRecord3Update,
+ testRecord4EarlierUpdate), readerContext, properties, new String[]{"ts"});
+ inputRecords.putAll(convertToBufferedRecordsMapForDeletes(Arrays.asList(testRecord5DeleteByCustomMarker, testRecord6DeleteByCustomMarker), false));
+ KeyBasedFileGroupRecordBuffer<IndexedRecord> fileGroupRecordBuffer = buildKeyBasedFileGroupRecordBuffer(readerContext, tableConfig, readStats, null,
+ RecordMergeMode.EVENT_TIME_ORDERING, Collections.singletonList("ts"), properties);
+
+ fileGroupRecordBuffer.setBaseFileIterator(ClosableIterator.wrap(Arrays.asList(testRecord1, testRecord2, testRecord3, testRecord4,
+ testRecord5, testRecord6).iterator()));
+
+ inputRecords.entrySet().forEach(kv -> {
+ try {
+ fileGroupRecordBuffer.processNextDataRecord(kv.getValue(), kv.getKey());
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to process next data ", e);
+ }
+ });
+
+ List<IndexedRecord> actualRecords = getActualRecords(fileGroupRecordBuffer);
+ // update for 4 is ignored due to lower ordering value.
+ // record5 is deleted.
+ // delete for 6 is ignored due to lower ordering value.
+ assertEquals(Arrays.asList(testRecord1UpdateWithSameTime, testRecord2Update, testRecord3Update, testRecord4, testRecord6), actualRecords);
+ assertEquals(0, readStats.getNumInserts());
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250138250
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -70,6 +70,7 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private final HoodieReaderContext<T> readerContext;
private final HoodieTableMetaClient metaClient;
private final InputSplit inputSplit;
+ private final boolean couldHaveLogFiles;
Review Comment:
I was looking to move this inside `InputSplit`.
but looking at what it tracks and what it is used for, feel 'InputSplit` may not be the right place to fit this in.
All it contains is, a base file, N log files, partition path and start and end pos.
Whether to load records from buffer or not, why would `InputSplit` track that.
Let me know if you guys have a better option on this. If not, I will leave this as is. ie. exposing `hasLogfiles()` in `FileGroupRecordBufferLoader` interface only which does not feel out of place to me.
maybe we can rename this variable in HoodieFileGroupReader from `couldHaveLogFiles` to 'useRecordBufferToInitRecordIterator`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148704338
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 993c4eb84ae768a8d590af6c0f98cfbaafa02f24 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460)
* fa86ba222bebce5aa11a508664acb3943f84710f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250160569
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> extends DefaultFileGroupRecordBufferLoader<T> {
+
+ private final Map<Serializable, BufferedRecord<T>> toBeMergedRecords;
+
+ private RecordsBasedFileGroupRecordBufferLoader(Map<Serializable, BufferedRecord<T>> toBeMergedRecords) {
+ super();
+ this.toBeMergedRecords = toBeMergedRecords;
Review Comment:
I thought about this before raising the patch on friday.
we have input which stems from HoodieWriteHandle.
we have the sink externalSpillableMap here which is within the record buffer.
We won't be able to fix the sink here, since we plan to leverage `processNextDataBlock`.
so only option is to fix the input.
As of now, its within HoodieWriteHandle (base class) which receives an iterator of records and fills in the `keyToNewRecords` https://github.com/apache/hudi/blob/f51c36fd64398718c039b26c15767e793b82904e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieWriteMergeHandle.java#L214
But it should be doable.
I can take up this fix in my next patch were I migrate the COW merge handle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250161213
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -70,6 +70,7 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private final HoodieReaderContext<T> readerContext;
private final HoodieTableMetaClient metaClient;
private final InputSplit inputSplit;
+ private final boolean couldHaveLogFiles;
Review Comment:
pushed an update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250297460
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/DefaultFileGroupRecordBufferLoader.java:
##########
@@ -46,7 +47,7 @@ static <T> DefaultFileGroupRecordBufferLoader<T> getInstance() {
return INSTANCE;
}
- private DefaultFileGroupRecordBufferLoader() {
Review Comment:
unnecessary change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148986210
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498)
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250388231
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -81,4 +88,12 @@ public long getLength() {
public boolean isParquetBaseFile() {
return baseFileOption.map(baseFile -> HoodieFileFormat.fromFileExtension(baseFile.getStoragePath().getFileExtension()) == HoodieFileFormat.PARQUET).orElse(false);
}
+
+ public boolean hasNoRecordsToMerge() {
+ return this.logFiles.isEmpty() && recordIterator.isEmpty();
Review Comment:
same comment as above.
this seems more maintainable to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250397229
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.PartialUpdateMode;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.table.read.UpdateProcessor;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> implements FileGroupRecordBufferLoader<T> {
+
+ private final Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords;
+
+ private RecordsBasedFileGroupRecordBufferLoader(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
+ super();
+ this.toBeMergedRecords = toBeMergedRecords;
+ }
+
+ static <T> RecordsBasedFileGroupRecordBufferLoader<T> getInstance(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
+ return new RecordsBasedFileGroupRecordBufferLoader<>(toBeMergedRecords);
+ }
+
+ @Override
+ public Option<Pair<HoodieFileGroupRecordBuffer<T>, List<String>>> getRecordBuffer(HoodieReaderContext<T> readerContext,
+ HoodieStorage storage,
Review Comment:
It's fixed now. The indentation was off as you can see the lines above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250405583
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -81,4 +89,13 @@ public long getLength() {
public boolean isParquetBaseFile() {
return baseFileOption.map(baseFile -> HoodieFileFormat.fromFileExtension(baseFile.getStoragePath().getFileExtension()) == HoodieFileFormat.PARQUET).orElse(false);
}
+
+ public boolean hasNoRecordsToMerge() {
+ return this.logFiles.isEmpty() && recordIterator.isEmpty();
+ }
+
+ public Option<Iterator<Pair<Serializable, BufferedRecord>>> getRecordIterator() {
+ ValidationUtils.checkArgument(recordIterator.isPresent(), "Record iterator is not initialized");
Review Comment:
Let's add the check in `getLogFiles()` that record iterator should not be set for using log files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2251890881
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -36,33 +39,40 @@
*/
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
+ private final Either<Stream<HoodieLogFile>, Iterator<BufferedRecord>> recordsToMerge;
Review Comment:
Revert this to `Option<Iterator<BufferedRecord>>` if the decision is to use keep both the record iterator and a list of log files as members?
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/buffer/TestKeyBasedFileGroupRecordBuffer.java:
##########
@@ -146,11 +153,50 @@ void readWithEventTimeOrderingAndDeleteBlock() throws IOException {
assertEquals(2, readStats.getNumUpdates());
}
+ @Test
+ void readWithEventTimeOrderingWithRecords() throws IOException {
Review Comment:
Same for these tests too, better to have a separate test class for the streaming record buffer loader?
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/buffer/TestSortedKeyBasedFileGroupRecordBuffer.java:
##########
@@ -58,31 +70,92 @@ class TestSortedKeyBasedFileGroupRecordBuffer {
private final TestRecord testRecord6 = new TestRecord("6", 0);
private final TestRecord testRecord6Update = new TestRecord("6", 1);
+ private final IndexedRecord testIndexedRecord1 = createTestRecord("1", 1, 1L);
+ private final IndexedRecord testIndexedRecord2 = createTestRecord("2", 1, 1L);
+ private final IndexedRecord testIndexedRecord2Update = createTestRecord("2", 1, 2L);
+ private final IndexedRecord testIndexedRecord3 = createTestRecord("3", 1, 1L);
+ private final IndexedRecord testIndexedRecord4 = createTestRecord("4", 2, 2L);
+ private final IndexedRecord testIndexedRecord4LowerOrdering = createTestRecord("4", 2, 1L);
+ private final IndexedRecord testIndexedRecord5 = createTestRecord("5", 1, 1L);
+ private final IndexedRecord testRecord5DeleteByCustomMarker = createTestRecord("5", 3, 2L);
+ private final IndexedRecord testIndexedRecord6 = createTestRecord("6", 1, 5L);
+ private final IndexedRecord testIndexedRecord6Update = createTestRecord("6", 2, 10L);
+
@Test
void readBaseFileAndLogFile() throws IOException {
HoodieReadStats readStats = new HoodieReadStats();
HoodieReaderContext<TestRecord> mockReaderContext = mock(HoodieReaderContext.class, RETURNS_DEEP_STUBS);
+
SortedKeyBasedFileGroupRecordBuffer<TestRecord> fileGroupRecordBuffer = buildSortedKeyBasedFileGroupRecordBuffer(mockReaderContext, readStats);
fileGroupRecordBuffer.setBaseFileIterator(ClosableIterator.wrap(Arrays.asList(testRecord2, testRecord3, testRecord5).iterator()));
HoodieDataBlock dataBlock = mock(HoodieDataBlock.class);
when(dataBlock.getSchema()).thenReturn(HoodieTestDataGenerator.AVRO_SCHEMA);
- when(dataBlock.getEngineRecordIterator(mockReaderContext)).thenReturn(ClosableIterator.wrap(Arrays.asList(testRecord6, testRecord4, testRecord1, testRecord6Update, testRecord2Update).iterator()));
+ when(dataBlock.getEngineRecordIterator(mockReaderContext)).thenReturn(
+ ClosableIterator.wrap(Arrays.asList(testRecord6, testRecord4, testRecord1, testRecord6Update, testRecord2Update).iterator()));
HoodieDeleteBlock deleteBlock = mock(HoodieDeleteBlock.class);
- when(deleteBlock.getRecordsToDelete()).thenReturn(new DeleteRecord[]{DeleteRecord.create("3", "")});
+ when(deleteBlock.getRecordsToDelete()).thenReturn(new DeleteRecord[] {DeleteRecord.create("3", "")});
fileGroupRecordBuffer.processDataBlock(dataBlock, Option.empty());
fileGroupRecordBuffer.processDeleteBlock(deleteBlock);
-
List<TestRecord> actualRecords = getActualRecords(fileGroupRecordBuffer);
assertEquals(Arrays.asList(testRecord1, testRecord2Update, testRecord4, testRecord5, testRecord6Update), actualRecords);
assertEquals(3, readStats.getNumInserts());
assertEquals(1, readStats.getNumUpdates());
assertEquals(1, readStats.getNumDeletes());
}
+ @Test
+ void readWithEventTimeOrderingWithRecords() throws IOException {
Review Comment:
```suggestion
void readWithStreamingRecordBufferLoaderAndEventTimeOrdering() throws IOException {
```
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/buffer/TestSortedKeyBasedFileGroupRecordBuffer.java:
##########
@@ -58,31 +70,92 @@ class TestSortedKeyBasedFileGroupRecordBuffer {
private final TestRecord testRecord6 = new TestRecord("6", 0);
private final TestRecord testRecord6Update = new TestRecord("6", 1);
+ private final IndexedRecord testIndexedRecord1 = createTestRecord("1", 1, 1L);
+ private final IndexedRecord testIndexedRecord2 = createTestRecord("2", 1, 1L);
+ private final IndexedRecord testIndexedRecord2Update = createTestRecord("2", 1, 2L);
+ private final IndexedRecord testIndexedRecord3 = createTestRecord("3", 1, 1L);
+ private final IndexedRecord testIndexedRecord4 = createTestRecord("4", 2, 2L);
+ private final IndexedRecord testIndexedRecord4LowerOrdering = createTestRecord("4", 2, 1L);
+ private final IndexedRecord testIndexedRecord5 = createTestRecord("5", 1, 1L);
+ private final IndexedRecord testRecord5DeleteByCustomMarker = createTestRecord("5", 3, 2L);
+ private final IndexedRecord testIndexedRecord6 = createTestRecord("6", 1, 5L);
+ private final IndexedRecord testIndexedRecord6Update = createTestRecord("6", 2, 10L);
+
@Test
void readBaseFileAndLogFile() throws IOException {
HoodieReadStats readStats = new HoodieReadStats();
HoodieReaderContext<TestRecord> mockReaderContext = mock(HoodieReaderContext.class, RETURNS_DEEP_STUBS);
+
SortedKeyBasedFileGroupRecordBuffer<TestRecord> fileGroupRecordBuffer = buildSortedKeyBasedFileGroupRecordBuffer(mockReaderContext, readStats);
fileGroupRecordBuffer.setBaseFileIterator(ClosableIterator.wrap(Arrays.asList(testRecord2, testRecord3, testRecord5).iterator()));
HoodieDataBlock dataBlock = mock(HoodieDataBlock.class);
when(dataBlock.getSchema()).thenReturn(HoodieTestDataGenerator.AVRO_SCHEMA);
- when(dataBlock.getEngineRecordIterator(mockReaderContext)).thenReturn(ClosableIterator.wrap(Arrays.asList(testRecord6, testRecord4, testRecord1, testRecord6Update, testRecord2Update).iterator()));
+ when(dataBlock.getEngineRecordIterator(mockReaderContext)).thenReturn(
+ ClosableIterator.wrap(Arrays.asList(testRecord6, testRecord4, testRecord1, testRecord6Update, testRecord2Update).iterator()));
HoodieDeleteBlock deleteBlock = mock(HoodieDeleteBlock.class);
- when(deleteBlock.getRecordsToDelete()).thenReturn(new DeleteRecord[]{DeleteRecord.create("3", "")});
+ when(deleteBlock.getRecordsToDelete()).thenReturn(new DeleteRecord[] {DeleteRecord.create("3", "")});
fileGroupRecordBuffer.processDataBlock(dataBlock, Option.empty());
fileGroupRecordBuffer.processDeleteBlock(deleteBlock);
-
List<TestRecord> actualRecords = getActualRecords(fileGroupRecordBuffer);
assertEquals(Arrays.asList(testRecord1, testRecord2Update, testRecord4, testRecord5, testRecord6Update), actualRecords);
assertEquals(3, readStats.getNumInserts());
assertEquals(1, readStats.getNumUpdates());
assertEquals(1, readStats.getNumDeletes());
}
+ @Test
+ void readWithEventTimeOrderingWithRecords() throws IOException {
Review Comment:
Should this test be moved to a new class `TestStreamingFileGroupRecordBufferLoader`, then the method can be name simply as `readWithEventTimeOrdering`?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -36,33 +39,40 @@
*/
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
+ private final Either<Stream<HoodieLogFile>, Iterator<BufferedRecord>> recordsToMerge;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<BufferedRecord>> recordIterator;
private final String partitionPath;
// Byte offset to start reading from the base file
private final long start;
// Length of bytes to read from the base file
private final long length;
- InputSplit(Option<HoodieBaseFile> baseFileOption, Stream<HoodieLogFile> logFiles, String partitionPath, long start, long length) {
+ InputSplit(Option<HoodieBaseFile> baseFileOption,
+ Either<Stream<HoodieLogFile>, Iterator<BufferedRecord>> recordsToMerge,
+ String partitionPath, long start, long length) {
this.baseFileOption = baseFileOption;
- this.logFiles = logFiles.sorted(HoodieLogFile.getLogFileComparator())
- .filter(logFile -> !logFile.getFileName().endsWith(HoodieCDCUtils.CDC_LOGFILE_SUFFIX))
- .collect(Collectors.toList());
+ this.recordsToMerge = recordsToMerge;
+ if (recordsToMerge.isLeft()) {
+ this.logFiles = recordsToMerge.asLeft().sorted(HoodieLogFile.getLogFileComparator())
+ .filter(logFile -> !logFile.getFileName().endsWith(HoodieCDCUtils.CDC_LOGFILE_SUFFIX))
+ .collect(Collectors.toList());
+ this.recordIterator = Option.empty();
+ } else {
+ this.logFiles = Collections.emptyList();
+ this.recordIterator = Option.of(recordsToMerge.asRight());
+ }
this.partitionPath = partitionPath;
this.start = start;
this.length = length;
}
- static InputSplit fromFileSlice(FileSlice fileSlice, long start, long length) {
- return new InputSplit(fileSlice.getBaseFile(), fileSlice.getLogFiles(), fileSlice.getPartitionPath(),
- start, length);
- }
-
public Option<HoodieBaseFile> getBaseFileOption() {
return baseFileOption;
}
public List<HoodieLogFile> getLogFiles() {
+ ValidationUtils.checkArgument(recordIterator.isEmpty(),"Log files are not initialized");
Review Comment:
```suggestion
ValidationUtils.checkArgument(recordIterator.isEmpty(), "Log files are not initialized");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250272824
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.PartialUpdateMode;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.table.read.UpdateProcessor;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> implements FileGroupRecordBufferLoader<T> {
+
+ private final Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords;
+
+ private RecordsBasedFileGroupRecordBufferLoader(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
+ super();
+ this.toBeMergedRecords = toBeMergedRecords;
+ }
+
+ static <T> RecordsBasedFileGroupRecordBufferLoader<T> getInstance(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
+ return new RecordsBasedFileGroupRecordBufferLoader<>(toBeMergedRecords);
+ }
+
+ @Override
+ public Option<Pair<HoodieFileGroupRecordBuffer<T>, List<String>>> getRecordBuffer(HoodieReaderContext<T> readerContext,
+ HoodieStorage storage,
Review Comment:
nit: check indentation
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.PartialUpdateMode;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.table.read.UpdateProcessor;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> implements FileGroupRecordBufferLoader<T> {
+
+ private final Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords;
+
+ private RecordsBasedFileGroupRecordBufferLoader(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
+ super();
+ this.toBeMergedRecords = toBeMergedRecords;
+ }
+
+ static <T> RecordsBasedFileGroupRecordBufferLoader<T> getInstance(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
+ return new RecordsBasedFileGroupRecordBufferLoader<>(toBeMergedRecords);
+ }
Review Comment:
nit: we can remove this as it simple; the caller can directly use `new RecordsBasedFileGroupRecordBufferLoader<>(toBeMergedRecords)`
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.PartialUpdateMode;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.table.read.UpdateProcessor;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> implements FileGroupRecordBufferLoader<T> {
+
+ private final Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords;
+
+ private RecordsBasedFileGroupRecordBufferLoader(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> toBeMergedRecords) {
Review Comment:
```suggestion
private RecordsBasedFileGroupRecordBufferLoader(Iterator<Map.Entry<Serializable, BufferedRecord<T>>> recordIterator) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250269572
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> extends DefaultFileGroupRecordBufferLoader<T> {
+
+ private final Map<Serializable, BufferedRecord<T>> toBeMergedRecords;
+
+ private RecordsBasedFileGroupRecordBufferLoader(Map<Serializable, BufferedRecord<T>> toBeMergedRecords) {
+ super();
+ this.toBeMergedRecords = toBeMergedRecords;
Review Comment:
+1 on using `Iterator<Pair<String, BufferedRecord<T>>`. `keyToNewRecords` is more of a implementation detail in the current way of how the merge handle is implemented. I think the reason is that historically the merge handle takes care of both COW merge and MOR compaction where the MOR compaction uses a separate constructor which takes `keyToNewRecords` directly.
Now that MOR compaction completely relies on the file group reader-based logic in a separate merge handle class `FileGroupReaderBasedMergeHandle`, the "write" merge handle `HoodieWriteMergeHandle` should only take an iterator of records as the input. And this record buffer loader can directly take the iterator of records and does the merging inside the loader (i.e., such a logic below can be migrated from the merge handle to the buffer loader class), so that the merge handle can handle write logic only, which look better in terms of single responsibility of classes.
```
protected void populateIncomingRecordsMap(Iterator<HoodieRecord<T>> newRecordsItr) {
initIncomingRecordsMap();
while (newRecordsItr.hasNext()) {
HoodieRecord<T> record = newRecordsItr.next();
// update the new location of the record, so we know where to find it next
if (needsUpdateLocation()) {
record.unseal();
record.setNewLocation(newRecordLocation);
record.seal();
}
// NOTE: Once Records are added to map (spillable-map), DO NOT change it as they won't persist
keyToNewRecords.put(record.getRecordKey(), record);
}
if (keyToNewRecords instanceof ExternalSpillableMap) {
ExternalSpillableMap<String, HoodieRecord<T>> spillableMap = (ExternalSpillableMap<String, HoodieRecord<T>>) keyToNewRecords;
LOG.info("Number of entries in MemoryBasedMap => {}, Total size in bytes of MemoryBasedMap => {}, "
+ "Number of entries in BitCaskDiskMap => {}, Size of file spilled to disk => {}",
spillableMap.getInMemoryMapNumEntries(), spillableMap.getCurrentInMemoryMapSize(), spillableMap.getDiskBasedMapNumEntries(), spillableMap.getSizeOfFileOnDiskInBytes());
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148938211
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* 86072f4c0e6e2fc2667123886be80b32b9449e75 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495)
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250178786
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> extends DefaultFileGroupRecordBufferLoader<T> {
+
+ private final Map<Serializable, BufferedRecord<T>> toBeMergedRecords;
+
+ private RecordsBasedFileGroupRecordBufferLoader(Map<Serializable, BufferedRecord<T>> toBeMergedRecords) {
+ super();
+ this.toBeMergedRecords = toBeMergedRecords;
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250176830
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> extends DefaultFileGroupRecordBufferLoader<T> {
Review Comment:
another option is to introduce a BaseFileGroupRecordBufferLoader w/ just this method and we can make both Default and Records based FG reader extend from this baseFileGroupRecordBufferLoader
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148745206
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e47e1762e86960423e95b5f010001050d803477 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487)
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149365266
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
}, {
"hash" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507",
"triggerID" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"triggerType" : "PUSH"
}, {
"hash" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504)
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
* 05421e78b81645df80c94b96a5957a56b1fb4dae Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507)
* 8a366f32a9f3fb18a68691c540224aa2f5784621 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148947255
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498)
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250293207
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +39,25 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Map.Entry<Serializable, BufferedRecord>>> inputs;
Review Comment:
The inputs and logFiles work for same purposes, the only difference is physical presentation(one from files and one from memory), so I moved it into the `InputSplit` to get rid of the confusing flag switch and clean the notions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148999804
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498)
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149177827
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 2c95430bacfaf2805a59c420592cf12a8893b5c4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501)
* 83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504)
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3146146962
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 993c4eb84ae768a8d590af6c0f98cfbaafa02f24 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2249096181
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -70,6 +70,7 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private final HoodieReaderContext<T> readerContext;
private final HoodieTableMetaClient metaClient;
private final InputSplit inputSplit;
+ private final boolean couldHaveLogFiles;
Review Comment:
+1, when a special loader has been set up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148705219
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 993c4eb84ae768a8d590af6c0f98cfbaafa02f24 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460)
* fa86ba222bebce5aa11a508664acb3943f84710f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148737076
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa86ba222bebce5aa11a508664acb3943f84710f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485)
* 2e47e1762e86960423e95b5f010001050d803477 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250252720
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.PartialUpdateMode;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.table.read.UpdateProcessor;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> implements FileGroupRecordBufferLoader<T> {
Review Comment:
Should we rename this to `StreamingFileGroupRecordBufferLoader` (to be used by merge handle taking records streamed from Spark iterator on the executor side)? `FileGroupRecordBufferLoader` suggests it loads records, so `Records` prefix is still broad in indicating the functionality of this class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250287115
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +39,25 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Map.Entry<Serializable, BufferedRecord>>> inputs;
Review Comment:
Do we need this in the `InputSplits` or can the iterator be passed as an arg to the `RecordsBasedFileGroupRecordBufferLoader`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250285790
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +39,25 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Map.Entry<Serializable, BufferedRecord>>> inputs;
Review Comment:
yeah, +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250298225
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/DefaultFileGroupRecordBufferLoader.java:
##########
@@ -59,25 +60,36 @@ public Pair<HoodieFileGroupRecordBuffer<T>, List<String>> getRecordBuffer(Hoodie
ReaderParameters readerParameters,
HoodieReadStats readStats,
Option<BaseFileUpdateCallback<T>> fileGroupUpdateCallback) {
+ FileGroupRecordBuffer<T> recordBuffer = getFileGroupRecordBuffer(readerContext, inputSplit, orderingFieldNames, hoodieTableMetaClient, props,
+ readerParameters, readStats, fileGroupUpdateCallback);
+ return Pair.of(recordBuffer, scanLogFiles(readerContext, storage, inputSplit, hoodieTableMetaClient, props,
+ readerParameters, readStats, recordBuffer));
+ }
+ protected FileGroupRecordBuffer<T> getFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
+ InputSplit inputSplit,
Review Comment:
nit: the indentation seems not right.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250293207
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +39,25 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Map.Entry<Serializable, BufferedRecord>>> inputs;
Review Comment:
The inputs and logFiles works for same purposes, so I moved it into the `InputSplit` to get rid of the confusing flag switch and clean the notions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250357469
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -128,14 +128,14 @@ private HoodieFileGroupReader(HoodieReaderContext<T> readerContext, HoodieStorag
*/
private void initRecordIterators() throws IOException {
ClosableIterator<T> iter = makeBaseFileIterator();
- if (inputSplit.getLogFiles().isEmpty()) {
+ Option<Pair<HoodieFileGroupRecordBuffer<T>, List<String>>> initializationResultOpt = recordBufferLoader.getRecordBuffer(
+ readerContext, storage, inputSplit, orderingFieldNames, metaClient, props, readerParameters, readStats, fileGroupUpdateCallback);
Review Comment:
Fixed now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250389302
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/StreamingFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.PartialUpdateMode;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.table.read.UpdateProcessor;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.exception.HoodieValidationException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class StreamingFileGroupRecordBufferLoader<T> implements FileGroupRecordBufferLoader<T> {
+ private static final StreamingFileGroupRecordBufferLoader INSTANCE = new StreamingFileGroupRecordBufferLoader<>();
+
+ static <T> StreamingFileGroupRecordBufferLoader<T> getInstance() {
+ return INSTANCE;
+ }
+
+ @Override
+ public Pair<HoodieFileGroupRecordBuffer<T>, List<String>> getRecordBuffer(HoodieReaderContext<T> readerContext, HoodieStorage storage, InputSplit inputSplit,
+ List<String> orderingFieldNames, HoodieTableMetaClient hoodieTableMetaClient,
+ TypedProperties props, ReaderParameters readerParameters, HoodieReadStats readStats,
+ Option<BaseFileUpdateCallback<T>> fileGroupUpdateCallback) {
+ PartialUpdateMode partialUpdateMode = hoodieTableMetaClient.getTableConfig().getPartialUpdateMode();
+ UpdateProcessor<T> updateProcessor = UpdateProcessor.create(readStats, readerContext, readerParameters.emitDeletes(), fileGroupUpdateCallback);
+ FileGroupRecordBuffer<T> recordBuffer;
+ if (readerParameters.sortOutputs()) {
+ recordBuffer = new SortedKeyBasedFileGroupRecordBuffer<>(
+ readerContext, hoodieTableMetaClient, readerContext.getMergeMode(), partialUpdateMode, props, orderingFieldNames, updateProcessor);
+ } else {
+ recordBuffer = new KeyBasedFileGroupRecordBuffer<>(
+ readerContext, hoodieTableMetaClient, readerContext.getMergeMode(), partialUpdateMode, props, orderingFieldNames, updateProcessor);
+ }
+
+ Iterator<Pair<Serializable, BufferedRecord>> recordIterator = inputSplit.getRecordIterator()
+ .orElseThrow(() -> new HoodieValidationException("The record iterator has not been setup"));
+
+ while (recordIterator.hasNext()) {
+ Pair<Serializable, BufferedRecord> entry = recordIterator.next();
+ try {
+ recordBuffer.processNextDataRecord(entry.getValue(), entry.getKey());
Review Comment:
To revisit later: for upserts which does dedup and precombine first, the record iterator should have unique record keys, so there is no need to do merging. See if there is any room for optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149639513
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
}, {
"hash" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507",
"triggerID" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"triggerType" : "PUSH"
}, {
"hash" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7510",
"triggerID" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"triggerType" : "PUSH"
}, {
"hash" : "f87ee945fcedab791ee4864fb321bebff880d7e1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7513",
"triggerID" : "f87ee945fcedab791ee4864fb321bebff880d7e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
* 8a366f32a9f3fb18a68691c540224aa2f5784621 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7510)
* f87ee945fcedab791ee4864fb321bebff880d7e1 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7513)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3151963937
@yihua : addressed all feedback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #13670:
URL: https://github.com/apache/hudi/pull/13670
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2249091992
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/DefaultFileGroupRecordBufferLoader.java:
##########
@@ -42,11 +48,29 @@
class DefaultFileGroupRecordBufferLoader<T> extends LogScanningRecordBufferLoader implements FileGroupRecordBufferLoader<T> {
Review Comment:
+1, `FileGroupRecordBufferLoader` itself is extensible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148744286
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa86ba222bebce5aa11a508664acb3943f84710f Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485)
* 2e47e1762e86960423e95b5f010001050d803477 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487)
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148746110
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e47e1762e86960423e95b5f010001050d803477 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487)
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250176947
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> extends DefaultFileGroupRecordBufferLoader<T> {
+
+ private final Map<Serializable, BufferedRecord<T>> toBeMergedRecords;
+
+ private RecordsBasedFileGroupRecordBufferLoader(Map<Serializable, BufferedRecord<T>> toBeMergedRecords) {
+ super();
+ this.toBeMergedRecords = toBeMergedRecords;
Review Comment:
sg
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148871781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489)
* 86072f4c0e6e2fc2667123886be80b32b9449e75 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148942441
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* 86072f4c0e6e2fc2667123886be80b32b9449e75 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495)
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148972281
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498)
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250293207
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +39,25 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Map.Entry<Serializable, BufferedRecord>>> inputs;
Review Comment:
The inputs and logFiles work for same purposes, so I moved it into the `InputSplit` to get rid of the confusing flag switch and clean the notions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149029344
@nsivabalan
Another thing that can improve here is: if we change the inputs type to `Iterator<HoodieRecord>` instead of `List<Pair<String, BufferedRecord>>` , we can wrap the inputs deduplication into this new record buffer, which is super convenient for all the engines (engines do not need to cate about this for COW write path)
We can take it in a separate PR though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149454648
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
}, {
"hash" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507",
"triggerID" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"triggerType" : "PUSH"
}, {
"hash" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7510",
"triggerID" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
* 05421e78b81645df80c94b96a5957a56b1fb4dae Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507)
* 8a366f32a9f3fb18a68691c540224aa2f5784621 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7510)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148721817
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 993c4eb84ae768a8d590af6c0f98cfbaafa02f24 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460)
* fa86ba222bebce5aa11a508664acb3943f84710f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485)
* 2e47e1762e86960423e95b5f010001050d803477 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2252347139
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBufferLoader.java:
##########
@@ -52,6 +52,10 @@ static <T> FileGroupRecordBufferLoader<T> createDefault() {
return DefaultFileGroupRecordBufferLoader.getInstance();
}
+ static <T> FileGroupRecordBufferLoader<T> createStreamingRecordsBufferLoader() {
Review Comment:
nit:
```suggestion
static <T> FileGroupRecordBufferLoader<T> createStreaming() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148770121
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e47e1762e86960423e95b5f010001050d803477 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487)
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149118301
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 2c95430bacfaf2805a59c420592cf12a8893b5c4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501)
* 83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250403405
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -94,6 +95,7 @@ public boolean hasNoRecordsToMerge() {
}
public Option<Iterator<Pair<Serializable, BufferedRecord>>> getRecordIterator() {
+ ValidationUtils.checkArgument(recordIterator.isPresent(), "Record iterator is not initialized");
Review Comment:
This is fixed after my comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3146077762
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 993c4eb84ae768a8d590af6c0f98cfbaafa02f24 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2252561447
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/buffer/TestSortedKeyBasedFileGroupRecordBuffer.java:
##########
@@ -58,31 +70,92 @@ class TestSortedKeyBasedFileGroupRecordBuffer {
private final TestRecord testRecord6 = new TestRecord("6", 0);
private final TestRecord testRecord6Update = new TestRecord("6", 1);
+ private final IndexedRecord testIndexedRecord1 = createTestRecord("1", 1, 1L);
+ private final IndexedRecord testIndexedRecord2 = createTestRecord("2", 1, 1L);
+ private final IndexedRecord testIndexedRecord2Update = createTestRecord("2", 1, 2L);
+ private final IndexedRecord testIndexedRecord3 = createTestRecord("3", 1, 1L);
+ private final IndexedRecord testIndexedRecord4 = createTestRecord("4", 2, 2L);
+ private final IndexedRecord testIndexedRecord4LowerOrdering = createTestRecord("4", 2, 1L);
+ private final IndexedRecord testIndexedRecord5 = createTestRecord("5", 1, 1L);
+ private final IndexedRecord testRecord5DeleteByCustomMarker = createTestRecord("5", 3, 2L);
+ private final IndexedRecord testIndexedRecord6 = createTestRecord("6", 1, 5L);
+ private final IndexedRecord testIndexedRecord6Update = createTestRecord("6", 2, 10L);
+
@Test
void readBaseFileAndLogFile() throws IOException {
HoodieReadStats readStats = new HoodieReadStats();
HoodieReaderContext<TestRecord> mockReaderContext = mock(HoodieReaderContext.class, RETURNS_DEEP_STUBS);
+
SortedKeyBasedFileGroupRecordBuffer<TestRecord> fileGroupRecordBuffer = buildSortedKeyBasedFileGroupRecordBuffer(mockReaderContext, readStats);
fileGroupRecordBuffer.setBaseFileIterator(ClosableIterator.wrap(Arrays.asList(testRecord2, testRecord3, testRecord5).iterator()));
HoodieDataBlock dataBlock = mock(HoodieDataBlock.class);
when(dataBlock.getSchema()).thenReturn(HoodieTestDataGenerator.AVRO_SCHEMA);
- when(dataBlock.getEngineRecordIterator(mockReaderContext)).thenReturn(ClosableIterator.wrap(Arrays.asList(testRecord6, testRecord4, testRecord1, testRecord6Update, testRecord2Update).iterator()));
+ when(dataBlock.getEngineRecordIterator(mockReaderContext)).thenReturn(
+ ClosableIterator.wrap(Arrays.asList(testRecord6, testRecord4, testRecord1, testRecord6Update, testRecord2Update).iterator()));
HoodieDeleteBlock deleteBlock = mock(HoodieDeleteBlock.class);
- when(deleteBlock.getRecordsToDelete()).thenReturn(new DeleteRecord[]{DeleteRecord.create("3", "")});
+ when(deleteBlock.getRecordsToDelete()).thenReturn(new DeleteRecord[] {DeleteRecord.create("3", "")});
fileGroupRecordBuffer.processDataBlock(dataBlock, Option.empty());
fileGroupRecordBuffer.processDeleteBlock(deleteBlock);
-
- List<TestRecord> actualRecords = getActualRecords(fileGroupRecordBuffer);
+ List<TestRecord> actualRecords = getActualRecordsForSortedKeyBased(fileGroupRecordBuffer);
assertEquals(Arrays.asList(testRecord1, testRecord2Update, testRecord4, testRecord5, testRecord6Update), actualRecords);
assertEquals(3, readStats.getNumInserts());
assertEquals(1, readStats.getNumUpdates());
assertEquals(1, readStats.getNumDeletes());
}
+ @Test
+ void readWithStreamingRecordBufferLoaderAndEventTimeOrdering() throws IOException {
Review Comment:
nit: move this too
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/buffer/TestKeyBasedFileGroupRecordBuffer.java:
##########
@@ -176,12 +205,49 @@ void readWithCommitTimeOrdering() throws IOException {
assertEquals(2, readStats.getNumUpdates());
}
+ @Test
+ void readWithCommitTimeOrderingWithRecords() throws IOException {
Review Comment:
Could this test and other new tests for streaming loader be removed since it's added in the new test class for streaming record buffer loader?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250143977
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> extends DefaultFileGroupRecordBufferLoader<T> {
+
+ private final Map<Serializable, BufferedRecord<T>> toBeMergedRecords;
+
+ private RecordsBasedFileGroupRecordBufferLoader(Map<Serializable, BufferedRecord<T>> toBeMergedRecords) {
+ super();
+ this.toBeMergedRecords = toBeMergedRecords;
Review Comment:
Can this take in an `Iterator<Pair<String, BufferedRecord<T>>`? Then we can eventually avoid making a map before the merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250143483
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -70,6 +70,7 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private final HoodieReaderContext<T> readerContext;
private final HoodieTableMetaClient metaClient;
private final InputSplit inputSplit;
+ private final boolean couldHaveLogFiles;
Review Comment:
Another option is to have this associated with the output of `getRecordBuffer`. For example, we return `Option<Pair<>>` instead of `Pair<>`. If the option is empty, we only need to read the base file. If it is present, it is assumed we need to merge this data with the base file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250174398
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> extends DefaultFileGroupRecordBufferLoader<T> {
+
+ private final Map<Serializable, BufferedRecord<T>> toBeMergedRecords;
+
+ private RecordsBasedFileGroupRecordBufferLoader(Map<Serializable, BufferedRecord<T>> toBeMergedRecords) {
+ super();
+ this.toBeMergedRecords = toBeMergedRecords;
Review Comment:
Right now this is only used in the tests right? It seems like it makes more sense to start with this as an iterator so you don't have to refactor all the tests later as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250161057
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -70,6 +70,7 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private final HoodieReaderContext<T> readerContext;
private final HoodieTableMetaClient metaClient;
private final InputSplit inputSplit;
+ private final boolean couldHaveLogFiles;
Review Comment:
giving this one a try now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250239631
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -128,14 +128,14 @@ private HoodieFileGroupReader(HoodieReaderContext<T> readerContext, HoodieStorag
*/
private void initRecordIterators() throws IOException {
ClosableIterator<T> iter = makeBaseFileIterator();
- if (inputSplit.getLogFiles().isEmpty()) {
+ Option<Pair<HoodieFileGroupRecordBuffer<T>, List<String>>> initializationResultOpt = recordBufferLoader.getRecordBuffer(
+ readerContext, storage, inputSplit, orderingFieldNames, metaClient, props, readerParameters, readStats, fileGroupUpdateCallback);
Review Comment:
I assume the intention is that the new "updates" can either come from the log files (from storage for read, compaction, etc.) or record iterator (from ingestion input for the merge handle). Is there a plan to provide a new abstraction to represent this instead of `InputSplit` which currently represents base and log files only?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -128,14 +128,14 @@ private HoodieFileGroupReader(HoodieReaderContext<T> readerContext, HoodieStorag
*/
private void initRecordIterators() throws IOException {
ClosableIterator<T> iter = makeBaseFileIterator();
- if (inputSplit.getLogFiles().isEmpty()) {
+ Option<Pair<HoodieFileGroupRecordBuffer<T>, List<String>>> initializationResultOpt = recordBufferLoader.getRecordBuffer(
+ readerContext, storage, inputSplit, orderingFieldNames, metaClient, props, readerParameters, readStats, fileGroupUpdateCallback);
Review Comment:
This can be done separate, to make the code easier to follow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250276891
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/StreamingFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,92 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.PartialUpdateMode;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.table.read.UpdateProcessor;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in an iterator records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ *
+ * @param <T> Engine native presentation of the record.
+ */
+public class StreamingFileGroupRecordBufferLoader<T> implements FileGroupRecordBufferLoader<T> {
+
+ private final Iterator<Pair<Serializable, BufferedRecord<T>>> recordIterator;
+
+ private StreamingFileGroupRecordBufferLoader(Iterator<Pair<Serializable, BufferedRecord<T>>> recordIterator) {
+ super();
+ this.recordIterator = recordIterator;
+ }
+
+ static <T> StreamingFileGroupRecordBufferLoader<T> getInstance(Iterator<Pair<Serializable, BufferedRecord<T>>> recordIterator) {
+ return new StreamingFileGroupRecordBufferLoader<>(recordIterator);
+ }
+
+ @Override
+ public Option<Pair<HoodieFileGroupRecordBuffer<T>, List<String>>> getRecordBuffer(HoodieReaderContext<T> readerContext,
+ HoodieStorage storage,
+ InputSplit inputSplit,
+ List<String> orderingFieldNames,
+ HoodieTableMetaClient hoodieTableMetaClient,
+ TypedProperties props,
+ ReaderParameters readerParameters,
+ HoodieReadStats readStats,
+ Option<BaseFileUpdateCallback<T>> fileGroupUpdateCallback) {
+ PartialUpdateMode partialUpdateMode = hoodieTableMetaClient.getTableConfig().getPartialUpdateMode();
+ UpdateProcessor<T> updateProcessor = UpdateProcessor.create(readStats, readerContext, readerParameters.emitDeletes(), fileGroupUpdateCallback);
+ FileGroupRecordBuffer<T> recordBuffer;
+ if (readerParameters.sortOutputs()) {
+ recordBuffer = new SortedKeyBasedFileGroupRecordBuffer<>(
+ readerContext, hoodieTableMetaClient, readerContext.getMergeMode(), partialUpdateMode, props, orderingFieldNames, updateProcessor);
+ } else {
+ recordBuffer = new KeyBasedFileGroupRecordBuffer<>(
+ readerContext, hoodieTableMetaClient, readerContext.getMergeMode(), partialUpdateMode, props, orderingFieldNames, updateProcessor);
+ }
+ while (recordIterator.hasNext()) {
+ Map.Entry<Serializable, BufferedRecord<T>> entry = recordIterator.next();
Review Comment:
Should this be `Pair` instead of `Map.Entry`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250283257
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +39,25 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Map.Entry<Serializable, BufferedRecord>>> inputs;
Review Comment:
Use `Either` here so that the `InputSplit` can take either `logFiles` or `recordIterator`, but not both.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +39,25 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Map.Entry<Serializable, BufferedRecord>>> inputs;
Review Comment:
```suggestion
private final Option<Iterator<Map.Entry<Serializable, BufferedRecord>>> recordIterator;
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +39,25 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Map.Entry<Serializable, BufferedRecord>>> inputs;
private final String partitionPath;
// Byte offset to start reading from the base file
private final long start;
// Length of bytes to read from the base file
private final long length;
- InputSplit(Option<HoodieBaseFile> baseFileOption, Stream<HoodieLogFile> logFiles, String partitionPath, long start, long length) {
+ InputSplit(Option<HoodieBaseFile> baseFileOption, Stream<HoodieLogFile> logFiles, String partitionPath, long start, long length,
Review Comment:
Provide two constructors: one taking `baseFileOption` and `logFiles`, another taking `baseFileOption` and `recordIterator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250293207
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +39,25 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Map.Entry<Serializable, BufferedRecord>>> inputs;
Review Comment:
The inputs and logFiles work for same purposes, the only difference is phisical presentation(one from files and one from memory), so I moved it into the `InputSplit` to get rid of the confusing flag switch and clean the notions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250389169
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +41,30 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Pair<Serializable, BufferedRecord>>> recordIterator;
Review Comment:
Can it be just `Iterator<BufferedRecord>` ? The pair value is the record key which is already in the `BufferedRecord`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250395819
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -37,27 +41,30 @@
public class InputSplit {
private final Option<HoodieBaseFile> baseFileOption;
private final List<HoodieLogFile> logFiles;
+ private final Option<Iterator<Pair<Serializable, BufferedRecord>>> recordIterator;
Review Comment:
It's about safety. After the input split is constructed, caller can still call `getLogFiles()` and `getRecordIterator()` regardless of what's passed in.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250418873
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/buffer/TestKeyBasedFileGroupRecordBuffer.java:
##########
@@ -146,11 +151,54 @@ void readWithEventTimeOrderingAndDeleteBlock() throws IOException {
assertEquals(2, readStats.getNumUpdates());
}
+ @Test
+ void readWithEventTimeOrderingWithRecords() throws IOException {
Review Comment:
Could these new tests be revised to directly construct `StreamingFileGroupRecordBufferLoader` and call `StreamingFileGroupRecordBufferLoader#getRecordBuffer` for testing instead of mimicking the behavior of calling `HoodieFileGroupRecordBuffer#processNextDataRecord`?
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/buffer/TestKeyBasedFileGroupRecordBuffer.java:
##########
@@ -146,11 +151,54 @@ void readWithEventTimeOrderingAndDeleteBlock() throws IOException {
assertEquals(2, readStats.getNumUpdates());
}
+ @Test
+ void readWithEventTimeOrderingWithRecords() throws IOException {
+ HoodieReadStats readStats = new HoodieReadStats();
+ TypedProperties properties = new TypedProperties();
+ properties.setProperty(HoodieTableConfig.PRECOMBINE_FIELDS.key(), "ts");
+ properties.setProperty(DELETE_KEY, "counter");
+ properties.setProperty(DELETE_MARKER, "3");
+ HoodieTableConfig tableConfig = mock(HoodieTableConfig.class);
+ when(tableConfig.getRecordKeyFields()).thenReturn(Option.of(new String[] {"record_key"}));
+ StorageConfiguration<?> storageConfiguration = mock(StorageConfiguration.class);
+ HoodieReaderContext<IndexedRecord> readerContext = new HoodieAvroReaderContext(storageConfiguration, tableConfig, Option.empty(), Option.empty());
+ readerContext.setHasLogFiles(false);
+ readerContext.setHasBootstrapBaseFile(false);
+ FileGroupReaderSchemaHandler schemaHandler = new FileGroupReaderSchemaHandler(readerContext, SCHEMA, SCHEMA, Option.empty(), tableConfig,
+ properties);
+ readerContext.setSchemaHandler(schemaHandler);
+ Map<Serializable, BufferedRecord> inputRecords = convertToBufferedRecordsMap(Arrays.asList(testRecord1UpdateWithSameTime, testRecord2Update, testRecord3Update,
+ testRecord4EarlierUpdate), readerContext, properties, new String[]{"ts"});
+ inputRecords.putAll(convertToBufferedRecordsMapForDeletes(Arrays.asList(testRecord5DeleteByCustomMarker, testRecord6DeleteByCustomMarker), false));
+ KeyBasedFileGroupRecordBuffer<IndexedRecord> fileGroupRecordBuffer = buildKeyBasedFileGroupRecordBuffer(readerContext, tableConfig, readStats, null,
+ RecordMergeMode.EVENT_TIME_ORDERING, Collections.singletonList("ts"), properties);
+
+ fileGroupRecordBuffer.setBaseFileIterator(ClosableIterator.wrap(Arrays.asList(testRecord1, testRecord2, testRecord3, testRecord4,
+ testRecord5, testRecord6).iterator()));
+
+ inputRecords.entrySet().forEach(kv -> {
+ try {
+ fileGroupRecordBuffer.processNextDataRecord(kv.getValue(), kv.getKey());
+ } catch (IOException e) {
+ throw new HoodieIOException("Failed to process next data ", e);
+ }
+ });
+
+ List<IndexedRecord> actualRecords = getActualRecords(fileGroupRecordBuffer);
+ // update for 4 is ignored due to lower ordering value.
+ // record5 is deleted.
+ // delete for 6 is ignored due to lower ordering value.
+ assertEquals(Arrays.asList(testRecord1UpdateWithSameTime, testRecord2Update, testRecord3Update, testRecord4, testRecord6), actualRecords);
+ assertEquals(0, readStats.getNumInserts());
Review Comment:
Add a case where `readStats.getNumInserts()` is positive in `TestKeyBasedFileGroupRecordBuffer`, e.g., by changing this test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3152409371
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
}, {
"hash" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507",
"triggerID" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"triggerType" : "PUSH"
}, {
"hash" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7510",
"triggerID" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"triggerType" : "PUSH"
}, {
"hash" : "f87ee945fcedab791ee4864fb321bebff880d7e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7513",
"triggerID" : "f87ee945fcedab791ee4864fb321bebff880d7e1",
"triggerType" : "PUSH"
}, {
"hash" : "a2d81395369cd2ec368413e34bcfbb5df656bc0a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7523",
"triggerID" : "a2d81395369cd2ec368413e34bcfbb5df656bc0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
* a2d81395369cd2ec368413e34bcfbb5df656bc0a Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7523)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3152424834
thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3152116828
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
}, {
"hash" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507",
"triggerID" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"triggerType" : "PUSH"
}, {
"hash" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7510",
"triggerID" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"triggerType" : "PUSH"
}, {
"hash" : "f87ee945fcedab791ee4864fb321bebff880d7e1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7513",
"triggerID" : "f87ee945fcedab791ee4864fb321bebff880d7e1",
"triggerType" : "PUSH"
}, {
"hash" : "a2d81395369cd2ec368413e34bcfbb5df656bc0a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7523",
"triggerID" : "a2d81395369cd2ec368413e34bcfbb5df656bc0a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
* f87ee945fcedab791ee4864fb321bebff880d7e1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7513)
* a2d81395369cd2ec368413e34bcfbb5df656bc0a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7523)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250176317
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> extends DefaultFileGroupRecordBufferLoader<T> {
Review Comment:
I did it to reuse the instantiation of the record buffer
```
protected Option<FileGroupRecordBuffer<T>> getFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
InputSplit inputSplit,
List<String> orderingFieldNames,
HoodieTableMetaClient hoodieTableMetaClient,
TypedProperties props,
ReaderParameters readerParameters,
HoodieReadStats readStats,
Option<BaseFileUpdateCallback<T>> fileGroupUpdateCallback,
boolean ignoreNoLogFiles) {
if (!ignoreNoLogFiles && inputSplit.getLogFiles().isEmpty()) {
return Option.empty();
}
boolean isSkipMerge = ConfigUtils.getStringWithAltKeys(props, HoodieReaderConfig.MERGE_TYPE, true).equalsIgnoreCase(HoodieReaderConfig.REALTIME_SKIP_MERGE);
PartialUpdateMode partialUpdateMode = hoodieTableMetaClient.getTableConfig().getPartialUpdateMode();
UpdateProcessor<T> updateProcessor = UpdateProcessor.create(readStats, readerContext, readerParameters.emitDeletes(), fileGroupUpdateCallback);
FileGroupRecordBuffer<T> recordBuffer;
if (isSkipMerge) {
recordBuffer = new UnmergedFileGroupRecordBuffer<>(
readerContext, hoodieTableMetaClient, readerContext.getMergeMode(), partialUpdateMode, props, readStats);
} else if (readerParameters.sortOutputs()) {
recordBuffer = new SortedKeyBasedFileGroupRecordBuffer<>(
readerContext, hoodieTableMetaClient, readerContext.getMergeMode(), partialUpdateMode, props, orderingFieldNames, updateProcessor);
} else if (readerParameters.useRecordPosition() && inputSplit.getBaseFileOption().isPresent()) {
recordBuffer = new PositionBasedFileGroupRecordBuffer<>(
readerContext, hoodieTableMetaClient, readerContext.getMergeMode(), partialUpdateMode, inputSplit.getBaseFileOption().get().getCommitTime(), props,
orderingFieldNames, updateProcessor);
} else {
recordBuffer = new KeyBasedFileGroupRecordBuffer<>(
readerContext, hoodieTableMetaClient, readerContext.getMergeMode(), partialUpdateMode, props, orderingFieldNames, updateProcessor);
}
return Option.of(recordBuffer);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148877903
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* 86072f4c0e6e2fc2667123886be80b32b9449e75 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495)
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3148944701
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* 86072f4c0e6e2fc2667123886be80b32b9449e75 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495)
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 85965837c10df0abdb7cbd1765efc521173e9409 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498)
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149213647
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
}, {
"hash" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504)
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
* 05421e78b81645df80c94b96a5957a56b1fb4dae UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149370039
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
}, {
"hash" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507",
"triggerID" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"triggerType" : "PUSH"
}, {
"hash" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
* 05421e78b81645df80c94b96a5957a56b1fb4dae Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507)
* 8a366f32a9f3fb18a68691c540224aa2f5784621 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250177504
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/RecordsBasedFileGroupRecordBufferLoader.java:
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.common.table.read.buffer;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.read.BaseFileUpdateCallback;
+import org.apache.hudi.common.table.read.BufferedRecord;
+import org.apache.hudi.common.table.read.HoodieReadStats;
+import org.apache.hudi.common.table.read.InputSplit;
+import org.apache.hudi.common.table.read.ReaderParameters;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.storage.HoodieStorage;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Records based {@link FileGroupRecordBuffer} which takes in a map of records to be merged with a base file of interest.
+ * This will be used in write paths for COW merge cases.
+ * @param <T> Engine native presentation of the record.
+ */
+public class RecordsBasedFileGroupRecordBufferLoader<T> extends DefaultFileGroupRecordBufferLoader<T> {
Review Comment:
Do we want to support skip merge in this case though? We may only want the merging ones moved to that common base
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250246330
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/DefaultFileGroupRecordBufferLoader.java:
##########
@@ -59,7 +60,23 @@ public Pair<HoodieFileGroupRecordBuffer<T>, List<String>> getRecordBuffer(Hoodie
ReaderParameters readerParameters,
HoodieReadStats readStats,
Option<BaseFileUpdateCallback<T>> fileGroupUpdateCallback) {
+ Option<FileGroupRecordBuffer<T>> recordBufferOpt = getFileGroupRecordBuffer(readerContext, inputSplit, orderingFieldNames, hoodieTableMetaClient, props,
+ readerParameters, readStats, fileGroupUpdateCallback);
+ return recordBufferOpt.map(recordBuffer -> Pair.of(recordBufferOpt.get(), scanLogFiles(readerContext, storage, inputSplit, hoodieTableMetaClient, props,
+ readerParameters, readStats, recordBufferOpt.get())));
+ }
+ private Option<FileGroupRecordBuffer<T>> getFileGroupRecordBuffer(HoodieReaderContext<T> readerContext,
+ InputSplit inputSplit,
+ List<String> orderingFieldNames,
+ HoodieTableMetaClient hoodieTableMetaClient,
+ TypedProperties props,
+ ReaderParameters readerParameters,
+ HoodieReadStats readStats,
+ Option<BaseFileUpdateCallback<T>> fileGroupUpdateCallback) {
+ if (inputSplit.getLogFiles().isEmpty()) {
+ return Option.empty();
+ }
Review Comment:
nit: no need to separate the method out since `DefaultFileGroupRecordBufferLoader` is not extended and there is no other usage of the method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250423524
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/buffer/TestKeyBasedFileGroupRecordBuffer.java:
##########
@@ -146,11 +151,54 @@ void readWithEventTimeOrderingAndDeleteBlock() throws IOException {
assertEquals(2, readStats.getNumUpdates());
}
+ @Test
+ void readWithEventTimeOrderingWithRecords() throws IOException {
Review Comment:
The reason I asked is because the following lines in `StreamingFileGroupRecordBufferLoader#getRecordBuffer` are not covered by the tests
```
while (recordIterator.hasNext()) {
Pair<Serializable, BufferedRecord> entry = recordIterator.next();
try {
recordBuffer.processNextDataRecord(entry.getValue(), entry.getKey());
} catch (IOException e) {
throw new HoodieIOException("Failed to process next toBeMergedRecord ", e);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149862919
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
}, {
"hash" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7504",
"triggerID" : "83a0a1e8c3eebd0ef1dbc5d64f04c02bf1058843",
"triggerType" : "PUSH"
}, {
"hash" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ab0149ebb689d88f8ec570926c866b9057e8a9e",
"triggerType" : "PUSH"
}, {
"hash" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7507",
"triggerID" : "05421e78b81645df80c94b96a5957a56b1fb4dae",
"triggerType" : "PUSH"
}, {
"hash" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7510",
"triggerID" : "8a366f32a9f3fb18a68691c540224aa2f5784621",
"triggerType" : "PUSH"
}, {
"hash" : "f87ee945fcedab791ee4864fb321bebff880d7e1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7513",
"triggerID" : "f87ee945fcedab791ee4864fb321bebff880d7e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 4ab0149ebb689d88f8ec570926c866b9057e8a9e UNKNOWN
* f87ee945fcedab791ee4864fb321bebff880d7e1 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7513)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2249091992
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/DefaultFileGroupRecordBufferLoader.java:
##########
@@ -42,11 +48,29 @@
class DefaultFileGroupRecordBufferLoader<T> extends LogScanningRecordBufferLoader implements FileGroupRecordBufferLoader<T> {
Review Comment:
+1, `FileGroupRecordBufferLoader` itself designed to be extensible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3146078912
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 993c4eb84ae768a8d590af6c0f98cfbaafa02f24 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13670:
URL: https://github.com/apache/hudi/pull/13670#issuecomment-3149115725
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7460",
"triggerID" : "993c4eb84ae768a8d590af6c0f98cfbaafa02f24",
"triggerType" : "PUSH"
}, {
"hash" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7485",
"triggerID" : "fa86ba222bebce5aa11a508664acb3943f84710f",
"triggerType" : "PUSH"
}, {
"hash" : "2e47e1762e86960423e95b5f010001050d803477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7487",
"triggerID" : "2e47e1762e86960423e95b5f010001050d803477",
"triggerType" : "PUSH"
}, {
"hash" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b8bad6098ee49a1ac04cd682f90ce27f9912c1af",
"triggerType" : "PUSH"
}, {
"hash" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "88cfd5b75239e232f9602d47598309aae8183fb8",
"triggerType" : "PUSH"
}, {
"hash" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7489",
"triggerID" : "d5ed3248f0953bf546ef7316cdd60bdf2fbd6c07",
"triggerType" : "PUSH"
}, {
"hash" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7495",
"triggerID" : "86072f4c0e6e2fc2667123886be80b32b9449e75",
"triggerType" : "PUSH"
}, {
"hash" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7035a9392aa094d25f30412aeb62550cac92e5d",
"triggerType" : "PUSH"
}, {
"hash" : "85965837c10df0abdb7cbd1765efc521173e9409",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7498",
"triggerID" : "85965837c10df0abdb7cbd1765efc521173e9409",
"triggerType" : "PUSH"
}, {
"hash" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89ce93cd4e903076bf532d54491d6efa63abafc8",
"triggerType" : "PUSH"
}, {
"hash" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ecd9486ca9d0baab12831ef52b48932262af226a",
"triggerType" : "PUSH"
}, {
"hash" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2b2ddda890df152f7699e938c70c080acbdbb286",
"triggerType" : "PUSH"
}, {
"hash" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa",
"triggerType" : "PUSH"
}, {
"hash" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0be18e06f6d717090be3e338e4c0b03a487c5fcd",
"triggerType" : "PUSH"
}, {
"hash" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bc6688ca335512fbbf0af853cc1f55ef6f0aa736",
"triggerType" : "PUSH"
}, {
"hash" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1b8452a9614970ed91b76738f957b7a82ab3b2df",
"triggerType" : "PUSH"
}, {
"hash" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a100216f92b5860a5a402688b9b17a5a0ee59cbf",
"triggerType" : "PUSH"
}, {
"hash" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501",
"triggerID" : "2c95430bacfaf2805a59c420592cf12a8893b5c4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b8bad6098ee49a1ac04cd682f90ce27f9912c1af UNKNOWN
* 88cfd5b75239e232f9602d47598309aae8183fb8 UNKNOWN
* a7035a9392aa094d25f30412aeb62550cac92e5d UNKNOWN
* 89ce93cd4e903076bf532d54491d6efa63abafc8 UNKNOWN
* ecd9486ca9d0baab12831ef52b48932262af226a UNKNOWN
* 2b2ddda890df152f7699e938c70c080acbdbb286 UNKNOWN
* 12eb7f44e3d98a5f4bfb2d4f15e3ae034785a1fa UNKNOWN
* 0be18e06f6d717090be3e338e4c0b03a487c5fcd UNKNOWN
* bc6688ca335512fbbf0af853cc1f55ef6f0aa736 UNKNOWN
* 1b8452a9614970ed91b76738f957b7a82ab3b2df UNKNOWN
* a100216f92b5860a5a402688b9b17a5a0ee59cbf UNKNOWN
* 2c95430bacfaf2805a59c420592cf12a8893b5c4 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=7501)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9682] Add support to FileGroupRecordBuffer to assist w/ cow merge handle migration [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13670:
URL: https://github.com/apache/hudi/pull/13670#discussion_r2250400504
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/InputSplit.java:
##########
@@ -94,6 +95,7 @@ public boolean hasNoRecordsToMerge() {
}
public Option<Iterator<Pair<Serializable, BufferedRecord>>> getRecordIterator() {
+ ValidationUtils.checkArgument(recordIterator.isPresent(), "Record iterator is not initialized");
Review Comment:
we already have the check in `StreamingFileGroupRecordBufferLoader`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org