You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/05/17 03:44:06 UTC
[PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
the-other-tim-brown opened a new pull request, #13314:
URL: https://github.com/apache/hudi/pull/13314
### Change Logs
- Adds support for generating an iterator of HoodieRecord instead of simply the engine record in the FileGroupReader code. This will aide in migrating existing uses of LogScanner and other reader implementations that expect a HoodieRecord as the output.
### Impact
- Aides in the migration to FileGroupReader in other code paths
### Risk level (write none, low medium or high below)
Low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2889141936
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533",
"triggerID" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2895045030
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533",
"triggerID" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5534",
"triggerID" : "2889225965",
"triggerType" : "MANUAL"
}, {
"hash" : "12ebf1b41e44a6fd5821d630ef44e7c257604c9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5559",
"triggerID" : "12ebf1b41e44a6fd5821d630ef44e7c257604c9b",
"triggerType" : "PUSH"
}, {
"hash" : "365229106b9957952c12bd62fbd2ab47b45367e8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5587",
"triggerID" : "365229106b9957952c12bd62fbd2ab47b45367e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 365229106b9957952c12bd62fbd2ab47b45367e8 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5587)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2889107985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533",
"triggerID" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 08cf09a0df4ad51129ea2f09ee10cad229922dc9 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532)
* 0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094703531
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -317,6 +319,13 @@ private void validateOutputFromFileGroupReaderWithExistingRecords(StorageConfigu
// validate size is equivalent to ensure no duplicates are returned
assertEquals(expectedRecords.size(), actualRecordList.size());
assertEquals(new HashSet<>(expectedRecords), new HashSet<>(actualRecordList));
+ // validate records can be read from file group as HoodieRecords
+ actualRecordList = convertHoodieRecords(
+ readHoodieRecordsFromFileGroup(storageConf, tablePath, metaClient, fileSlices, avroSchema, recordMergeMode),
+ avroSchema, readerContext);
+ assertEquals(expectedRecords.size(), actualRecordList.size());
Review Comment:
not related to this patch. but I was reviewing this test class in general. looks like to compute final expected records (merging inserts + updates), we just choose the later if a record is part of update list (I am referring to method named "mergeRecordLists". So, this means, we can have tests only on commit time based merge mode. for others, the way we compute expected list of records may not work unless we fix this method impl.
not something we need to fix anything in this patch.but just wanted to call out that I noticed.
CC @yihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2889254521
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533",
"triggerID" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5534",
"triggerID" : "2889225965",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5534)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094164204
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -423,8 +410,8 @@ protected Pair<Boolean, T> merge(BufferedRecord<T> olderRecord, BufferedRecord<T
// TODO(HUDI-7843): decouple the merging logic from the merger
// and use the record merge mode to control how to merge partial updates
Option<Pair<HoodieRecord, Schema>> mergedRecord = recordMerger.get().partialMerge(
- readerContext.constructHoodieRecord(olderRecord), readerContext.getSchemaFromBufferRecord(olderRecord),
- readerContext.constructHoodieRecord(newerRecord), readerContext.getSchemaFromBufferRecord(newerRecord),
+ readerContext.constructHoodieRecord(olderRecord, partitionPath), readerContext.getSchemaFromBufferRecord(olderRecord),
+ readerContext.constructHoodieRecord(newerRecord, partitionPath), readerContext.getSchemaFromBufferRecord(newerRecord),
Review Comment:
Yes, I was wondering something similar. If we want to say that the readercontext is built per file group we can do this but if we want to share the reader context between file groups we'll need to keep it in its current form. I don't have a strong opinion either way so open to ideas here about how the class should be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2894686299
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533",
"triggerID" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5534",
"triggerID" : "2889225965",
"triggerType" : "MANUAL"
}, {
"hash" : "12ebf1b41e44a6fd5821d630ef44e7c257604c9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5559",
"triggerID" : "12ebf1b41e44a6fd5821d630ef44e7c257604c9b",
"triggerType" : "PUSH"
}, {
"hash" : "365229106b9957952c12bd62fbd2ab47b45367e8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "365229106b9957952c12bd62fbd2ab47b45367e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12ebf1b41e44a6fd5821d630ef44e7c257604c9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5559)
* 365229106b9957952c12bd62fbd2ab47b45367e8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2096575281
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -92,15 +92,16 @@ public Object getValue(InternalRow row, Schema schema, String fieldName) {
@Override
public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRow> bufferedRecord) {
+ HoodieKey hoodieKey = new HoodieKey(bufferedRecord.getRecordKey(), partitionPath);
if (bufferedRecord.isDelete()) {
return new HoodieEmptyRecord<>(
- new HoodieKey(bufferedRecord.getRecordKey(), null),
+ hoodieKey,
HoodieRecord.HoodieRecordType.SPARK);
}
Schema schema = getSchemaFromBufferRecord(bufferedRecord);
InternalRow row = bufferedRecord.getRecord();
- return new HoodieSparkRecord(row, HoodieInternalRowUtils.getCachedSchema(schema));
+ return new HoodieSparkRecord(hoodieKey, row, HoodieInternalRowUtils.getCachedSchema(schema), true);
Review Comment:
I synced with @yihua and he said it should be set to `false` so I have updated the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2889225965
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2093936067
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -91,16 +91,17 @@ public Object getValue(InternalRow row, Schema schema, String fieldName) {
}
@Override
- public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRow> bufferedRecord) {
+ public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRow> bufferedRecord, String partitionPath) {
+ HoodieKey hoodieKey = new HoodieKey(bufferedRecord.getRecordKey(), partitionPath);
if (bufferedRecord.isDelete()) {
return new HoodieEmptyRecord<>(
- new HoodieKey(bufferedRecord.getRecordKey(), null),
Review Comment:
is this a bug fix?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094253276
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -92,15 +92,16 @@ public Object getValue(InternalRow row, Schema schema, String fieldName) {
@Override
public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRow> bufferedRecord) {
+ HoodieKey hoodieKey = new HoodieKey(bufferedRecord.getRecordKey(), partitionPath);
if (bufferedRecord.isDelete()) {
return new HoodieEmptyRecord<>(
- new HoodieKey(bufferedRecord.getRecordKey(), null),
+ hoodieKey,
HoodieRecord.HoodieRecordType.SPARK);
}
Schema schema = getSchemaFromBufferRecord(bufferedRecord);
InternalRow row = bufferedRecord.getRecord();
- return new HoodieSparkRecord(row, HoodieInternalRowUtils.getCachedSchema(schema));
+ return new HoodieSparkRecord(hoodieKey, row, HoodieInternalRowUtils.getCachedSchema(schema), true);
Review Comment:
why set up `copy` as true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2889083863
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 08cf09a0df4ad51129ea2f09ee10cad229922dc9 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532)
* 0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094773427
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -92,15 +92,16 @@ public Object getValue(InternalRow row, Schema schema, String fieldName) {
@Override
public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRow> bufferedRecord) {
+ HoodieKey hoodieKey = new HoodieKey(bufferedRecord.getRecordKey(), partitionPath);
if (bufferedRecord.isDelete()) {
return new HoodieEmptyRecord<>(
- new HoodieKey(bufferedRecord.getRecordKey(), null),
+ hoodieKey,
HoodieRecord.HoodieRecordType.SPARK);
}
Schema schema = getSchemaFromBufferRecord(bufferedRecord);
InternalRow row = bufferedRecord.getRecord();
- return new HoodieSparkRecord(row, HoodieInternalRowUtils.getCachedSchema(schema));
+ return new HoodieSparkRecord(hoodieKey, row, HoodieInternalRowUtils.getCachedSchema(schema), true);
Review Comment:
maybe rename to `reusable`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2888489822
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19e0340fbb91c1b8a5d13443ddec6aaa4862f58b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522)
* 9f1feca4db82142d89faff20acae582c21702f48 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094352972
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -346,10 +360,20 @@ public void close() throws IOException {
}
}
- public HoodieFileGroupReaderIterator<T> getClosableIterator() {
+ public ClosableIterator<T> getClosableIterator() {
return new HoodieFileGroupReaderIterator<>(this);
}
+ /**
+ * @return An iterator over the records that wraps the engine-specific record in a HoodieRecord.
+ */
+ public ClosableIterator<HoodieRecord<T>> getClosableHoodieRecordIterator() {
Review Comment:
Is there a test for this iterator? I don't see any usages in this patch.
can we add some sanity tests.
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -92,15 +92,16 @@ public Object getValue(InternalRow row, Schema schema, String fieldName) {
@Override
public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRow> bufferedRecord) {
+ HoodieKey hoodieKey = new HoodieKey(bufferedRecord.getRecordKey(), partitionPath);
if (bufferedRecord.isDelete()) {
return new HoodieEmptyRecord<>(
- new HoodieKey(bufferedRecord.getRecordKey(), null),
+ hoodieKey,
HoodieRecord.HoodieRecordType.SPARK);
}
Schema schema = getSchemaFromBufferRecord(bufferedRecord);
InternalRow row = bufferedRecord.getRecord();
- return new HoodieSparkRecord(row, HoodieInternalRowUtils.getCachedSchema(schema));
+ return new HoodieSparkRecord(hoodieKey, row, HoodieInternalRowUtils.getCachedSchema(schema), true);
Review Comment:
the var name is slightly mis-leading.
its somewhat referring to "avoidAnyMoreCopies".
here we are setting it to true, so that anytime, if someone calls `copy()`, we return the same instance w/o making a copy.
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -129,6 +130,10 @@ public void setHasLogFiles(boolean hasLogFiles) {
this.hasLogFiles = hasLogFiles;
}
+ public void setPartitionPath(String partitionPath) {
Review Comment:
I am yet to review your patch on adding virtual keys support to File group reader.
but I assume all these will work w/ virtual keys enabled as well.
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestFileGroupRecordBuffer.java:
##########
@@ -84,8 +84,6 @@ class TestFileGroupRecordBuffer {
private final FileGroupReaderSchemaHandler schemaHandler =
mock(FileGroupReaderSchemaHandler.class);
private HoodieTableMetaClient hoodieTableMetaClient = mock(HoodieTableMetaClient.class);
- private Option<String> partitionNameOverrideOpt = Option.empty();
Review Comment:
same comment as above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094760584
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -317,6 +319,13 @@ private void validateOutputFromFileGroupReaderWithExistingRecords(StorageConfigu
// validate size is equivalent to ensure no duplicates are returned
assertEquals(expectedRecords.size(), actualRecordList.size());
assertEquals(new HashSet<>(expectedRecords), new HashSet<>(actualRecordList));
+ // validate records can be read from file group as HoodieRecords
+ actualRecordList = convertHoodieRecords(
+ readHoodieRecordsFromFileGroup(storageConf, tablePath, metaClient, fileSlices, avroSchema, recordMergeMode),
+ avroSchema, readerContext);
+ assertEquals(expectedRecords.size(), actualRecordList.size());
Review Comment:
We can update that in the future if required but we should really do unit testing for the merge order, see my other PR for this: https://github.com/apache/hudi/pull/13242
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2889082989
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39432de936daeeb67e3f027f8251cf6fb653b53e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527)
* 08cf09a0df4ad51129ea2f09ee10cad229922dc9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532)
* 0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2889226359
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533",
"triggerID" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5534",
"triggerID" : "2889225965",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533) Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5534)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2888088634
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19e0340fbb91c1b8a5d13443ddec6aaa4862f58b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094666673
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -317,6 +319,13 @@ private void validateOutputFromFileGroupReaderWithExistingRecords(StorageConfigu
// validate size is equivalent to ensure no duplicates are returned
assertEquals(expectedRecords.size(), actualRecordList.size());
assertEquals(new HashSet<>(expectedRecords), new HashSet<>(actualRecordList));
+ // validate records can be read from file group as HoodieRecords
+ actualRecordList = convertHoodieRecords(
Review Comment:
should we decouple the test then, the HoodieRecord iterator is a legacy API and should be deprecated already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2093937994
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -423,8 +410,8 @@ protected Pair<Boolean, T> merge(BufferedRecord<T> olderRecord, BufferedRecord<T
// TODO(HUDI-7843): decouple the merging logic from the merger
// and use the record merge mode to control how to merge partial updates
Option<Pair<HoodieRecord, Schema>> mergedRecord = recordMerger.get().partialMerge(
- readerContext.constructHoodieRecord(olderRecord), readerContext.getSchemaFromBufferRecord(olderRecord),
- readerContext.constructHoodieRecord(newerRecord), readerContext.getSchemaFromBufferRecord(newerRecord),
+ readerContext.constructHoodieRecord(olderRecord, partitionPath), readerContext.getSchemaFromBufferRecord(olderRecord),
+ readerContext.constructHoodieRecord(newerRecord, partitionPath), readerContext.getSchemaFromBufferRecord(newerRecord),
Review Comment:
is it possible we put the `partitionPath` into the `readerContext` so there is no need to change these apis.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094697790
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -92,15 +92,16 @@ public Object getValue(InternalRow row, Schema schema, String fieldName) {
@Override
public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRow> bufferedRecord) {
+ HoodieKey hoodieKey = new HoodieKey(bufferedRecord.getRecordKey(), partitionPath);
if (bufferedRecord.isDelete()) {
return new HoodieEmptyRecord<>(
- new HoodieKey(bufferedRecord.getRecordKey(), null),
+ hoodieKey,
HoodieRecord.HoodieRecordType.SPARK);
}
Schema schema = getSchemaFromBufferRecord(bufferedRecord);
InternalRow row = bufferedRecord.getRecord();
- return new HoodieSparkRecord(row, HoodieInternalRowUtils.getCachedSchema(schema));
+ return new HoodieSparkRecord(hoodieKey, row, HoodieInternalRowUtils.getCachedSchema(schema), true);
Review Comment:
@yihua knows the intricacies on this. can you chime in here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2894692709
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533",
"triggerID" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5534",
"triggerID" : "2889225965",
"triggerType" : "MANUAL"
}, {
"hash" : "12ebf1b41e44a6fd5821d630ef44e7c257604c9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5559",
"triggerID" : "12ebf1b41e44a6fd5821d630ef44e7c257604c9b",
"triggerType" : "PUSH"
}, {
"hash" : "365229106b9957952c12bd62fbd2ab47b45367e8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5587",
"triggerID" : "365229106b9957952c12bd62fbd2ab47b45367e8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12ebf1b41e44a6fd5821d630ef44e7c257604c9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5559)
* 365229106b9957952c12bd62fbd2ab47b45367e8 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5587)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2892432183
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533",
"triggerID" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5534",
"triggerID" : "2889225965",
"triggerType" : "MANUAL"
}, {
"hash" : "12ebf1b41e44a6fd5821d630ef44e7c257604c9b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "12ebf1b41e44a6fd5821d630ef44e7c257604c9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5534)
* 12ebf1b41e44a6fd5821d630ef44e7c257604c9b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2888490490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19e0340fbb91c1b8a5d13443ddec6aaa4862f58b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522)
* 9f1feca4db82142d89faff20acae582c21702f48 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2888544985
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39432de936daeeb67e3f027f8251cf6fb653b53e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2888033414
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19e0340fbb91c1b8a5d13443ddec6aaa4862f58b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094564885
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -92,15 +92,16 @@ public Object getValue(InternalRow row, Schema schema, String fieldName) {
@Override
public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRow> bufferedRecord) {
+ HoodieKey hoodieKey = new HoodieKey(bufferedRecord.getRecordKey(), partitionPath);
if (bufferedRecord.isDelete()) {
return new HoodieEmptyRecord<>(
- new HoodieKey(bufferedRecord.getRecordKey(), null),
+ hoodieKey,
HoodieRecord.HoodieRecordType.SPARK);
}
Schema schema = getSchemaFromBufferRecord(bufferedRecord);
InternalRow row = bufferedRecord.getRecord();
- return new HoodieSparkRecord(row, HoodieInternalRowUtils.getCachedSchema(schema));
+ return new HoodieSparkRecord(hoodieKey, row, HoodieInternalRowUtils.getCachedSchema(schema), true);
Review Comment:
@nsivabalan is there a way to know if we will need to copy based on the instance type of the spark row data?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2889081460
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39432de936daeeb67e3f027f8251cf6fb653b53e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527)
* 08cf09a0df4ad51129ea2f09ee10cad229922dc9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094568766
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -346,10 +360,20 @@ public void close() throws IOException {
}
}
- public HoodieFileGroupReaderIterator<T> getClosableIterator() {
+ public ClosableIterator<T> getClosableIterator() {
return new HoodieFileGroupReaderIterator<>(this);
}
+ /**
+ * @return An iterator over the records that wraps the engine-specific record in a HoodieRecord.
+ */
+ public ClosableIterator<HoodieRecord<T>> getClosableHoodieRecordIterator() {
Review Comment:
Added a test to the base class `TestHoodieFileGroupReaderBase` for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2888513521
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f1feca4db82142d89faff20acae582c21702f48 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526)
* 39432de936daeeb67e3f027f8251cf6fb653b53e Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2888032683
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19e0340fbb91c1b8a5d13443ddec6aaa4862f58b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2093963853
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -367,20 +395,16 @@ public boolean hasNext() {
}
@Override
- public T next() {
- return reader.next();
+ public U next() {
+ return recordSupplier.get();
Review Comment:
Just add another iterator, it does not deserve to add another per-row invocation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2093967509
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -367,20 +395,16 @@ public boolean hasNext() {
}
@Override
- public T next() {
- return reader.next();
+ public U next() {
+ return recordSupplier.get();
Review Comment:
Just wrap the existing iterator with a `MappingIterator` should works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2889080635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 39432de936daeeb67e3f027f8251cf6fb653b53e Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527)
* 08cf09a0df4ad51129ea2f09ee10cad229922dc9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2888493586
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 19e0340fbb91c1b8a5d13443ddec6aaa4862f58b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522)
* 9f1feca4db82142d89faff20acae582c21702f48 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526)
* 39432de936daeeb67e3f027f8251cf6fb653b53e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #13314:
URL: https://github.com/apache/hudi/pull/13314
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094759402
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -317,6 +319,13 @@ private void validateOutputFromFileGroupReaderWithExistingRecords(StorageConfigu
// validate size is equivalent to ensure no duplicates are returned
assertEquals(expectedRecords.size(), actualRecordList.size());
assertEquals(new HashSet<>(expectedRecords), new HashSet<>(actualRecordList));
+ // validate records can be read from file group as HoodieRecords
+ actualRecordList = convertHoodieRecords(
Review Comment:
It is not a legacy API yet unfortunately. Looking through all the code to migrate, it seems like we have a while before we can retire the HoodieRecord
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2892434813
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533",
"triggerID" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5534",
"triggerID" : "2889225965",
"triggerType" : "MANUAL"
}, {
"hash" : "12ebf1b41e44a6fd5821d630ef44e7c257604c9b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5559",
"triggerID" : "12ebf1b41e44a6fd5821d630ef44e7c257604c9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533) Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5534)
* 12ebf1b41e44a6fd5821d630ef44e7c257604c9b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5559)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2892542169
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5527",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
}, {
"hash" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5532",
"triggerID" : "08cf09a0df4ad51129ea2f09ee10cad229922dc9",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5533",
"triggerID" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"triggerType" : "PUSH"
}, {
"hash" : "0e5f7458c7a34cc0cc6f9580fc26fd93ede6905b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5534",
"triggerID" : "2889225965",
"triggerType" : "MANUAL"
}, {
"hash" : "12ebf1b41e44a6fd5821d630ef44e7c257604c9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5559",
"triggerID" : "12ebf1b41e44a6fd5821d630ef44e7c257604c9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 12ebf1b41e44a6fd5821d630ef44e7c257604c9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5559)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094165679
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/FileGroupRecordBuffer.java:
##########
@@ -423,8 +410,8 @@ protected Pair<Boolean, T> merge(BufferedRecord<T> olderRecord, BufferedRecord<T
// TODO(HUDI-7843): decouple the merging logic from the merger
// and use the record merge mode to control how to merge partial updates
Option<Pair<HoodieRecord, Schema>> mergedRecord = recordMerger.get().partialMerge(
- readerContext.constructHoodieRecord(olderRecord), readerContext.getSchemaFromBufferRecord(olderRecord),
- readerContext.constructHoodieRecord(newerRecord), readerContext.getSchemaFromBufferRecord(newerRecord),
+ readerContext.constructHoodieRecord(olderRecord, partitionPath), readerContext.getSchemaFromBufferRecord(olderRecord),
+ readerContext.constructHoodieRecord(newerRecord, partitionPath), readerContext.getSchemaFromBufferRecord(newerRecord),
Review Comment:
Actually, looking into the current class I see that there is `hasLogFiles` which implies the class is currently going to only work per file slice so I will move the partition path to keep the api the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094164241
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -91,16 +91,17 @@ public Object getValue(InternalRow row, Schema schema, String fieldName) {
}
@Override
- public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRow> bufferedRecord) {
+ public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRow> bufferedRecord, String partitionPath) {
+ HoodieKey hoodieKey = new HoodieKey(bufferedRecord.getRecordKey(), partitionPath);
if (bufferedRecord.isDelete()) {
return new HoodieEmptyRecord<>(
- new HoodieKey(bufferedRecord.getRecordKey(), null),
Review Comment:
Previously we did not have a need for the hoodie record outside of the merger fallback code so not a bug but now we have another use case with the new iterator so this partition field should be set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094164985
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -367,20 +395,16 @@ public boolean hasNext() {
}
@Override
- public T next() {
- return reader.next();
+ public U next() {
+ return recordSupplier.get();
Review Comment:
Makes sense, cleaning it up now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13314:
URL: https://github.com/apache/hudi/pull/13314#issuecomment-2888494330
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5522",
"triggerID" : "19e0340fbb91c1b8a5d13443ddec6aaa4862f58b",
"triggerType" : "PUSH"
}, {
"hash" : "9f1feca4db82142d89faff20acae582c21702f48",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526",
"triggerID" : "9f1feca4db82142d89faff20acae582c21702f48",
"triggerType" : "PUSH"
}, {
"hash" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "39432de936daeeb67e3f027f8251cf6fb653b53e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9f1feca4db82142d89faff20acae582c21702f48 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5526)
* 39432de936daeeb67e3f027f8251cf6fb653b53e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094565067
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestFileGroupRecordBuffer.java:
##########
@@ -84,8 +84,6 @@ class TestFileGroupRecordBuffer {
private final FileGroupReaderSchemaHandler schemaHandler =
mock(FileGroupReaderSchemaHandler.class);
private HoodieTableMetaClient hoodieTableMetaClient = mock(HoodieTableMetaClient.class);
- private Option<String> partitionNameOverrideOpt = Option.empty();
Review Comment:
This was an unused variable so I removed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9316] Add support for creating iterator of HoodieRecord from FGReader [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13314:
URL: https://github.com/apache/hudi/pull/13314#discussion_r2094565006
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -129,6 +130,10 @@ public void setHasLogFiles(boolean hasLogFiles) {
this.hasLogFiles = hasLogFiles;
}
+ public void setPartitionPath(String partitionPath) {
Review Comment:
Yes, this comes from the file slice so it is not related to the virtual keys
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org