You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "the-other-tim-brown (via GitHub)" <gi...@apache.org> on 2025/06/16 16:48:44 UTC
[PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
the-other-tim-brown opened a new pull request, #13445:
URL: https://github.com/apache/hudi/pull/13445
### Change Logs
- Removes usage of `HoodieMergedLogRecordScanner` in `HoodieTableMetadataUtil` and replace it with the `HoodieFileGroupReader`
- Fixes handling of deleted records when reading as `HoodieRecord`
### Impact
- Uses new standard way of reading
### Risk level (write none, low medium or high below)
Low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2155802377
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -389,7 +392,8 @@ public ClosableIterator<T> getClosableIterator() throws IOException {
*/
public ClosableIterator<HoodieRecord<T>> getClosableHoodieRecordIterator() throws IOException {
return new CloseableMappingIterator<>(getClosableIterator(), nextRecord -> {
- BufferedRecord<T> bufferedRecord = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
+ boolean isDelete = emitDelete && readerContext.isDeleteOperation(nextRecord);
Review Comment:
delete is a `DELETE`, you can not emit a delete when the `emitDelete` is false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151122789
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -378,6 +378,16 @@ public Comparable getOrderingValue(T record,
*/
public abstract HoodieRecord<T> constructHoodieRecord(BufferedRecord<T> bufferedRecord);
+ /**
+ * Constructs a new {@link HoodieRecord} based on the given record and schema.
+ *
+ * @param record The engine-specific row.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldName The name of the ordering field, if any.
+ * @return A new instance of {@link HoodieRecord}.
+ */
+ public abstract HoodieRecord<T> constructHoodieRecord(T record, Schema schema, Option<String> orderingFieldName);
Review Comment:
Without this we will need a way to determine if the record is a delete record even when the data may not be set. If there is a way to do this across all engine types (not just Flink) then we can use that. Without this you will get runtime errors when trying to read as a HoodieRecord.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985142953
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* bb842430968a51bb4bc6532cea2332fae2cd6544 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324)
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* 19a487daecfc29d93bbbf3629872bf811f6a1666 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2155157571
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordSerializer.java:
##########
@@ -58,9 +59,9 @@ public byte[] serialize(BufferedRecord<T> record) throws IOException {
output.writeBoolean(record.isDelete());
kryo.writeClassAndObject(output, record.getOrderingValue());
- byte[] avroBytes = recordSerializer.serialize(record.getRecord());
- output.writeInt(avroBytes.length);
- output.writeBytes(avroBytes);
+ byte[] recordBytes = record.getRecord() == null ? new byte[0] : recordSerializer.serialize(record.getRecord());
Review Comment:
Previously, this would fail for deletes where the record is null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985904643
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
}, {
"hash" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"triggerType" : "PUSH"
}, {
"hash" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335",
"triggerID" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"triggerType" : "PUSH"
}, {
"hash" : "fd1fd272a3257e50206333b3283374a1b8304894",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6338",
"triggerID" : "fd1fd272a3257e50206333b3283374a1b8304894",
"triggerType" : "PUSH"
}, {
"hash" : "1aa2d2fd21199cf18f8b7ea10e7fe229d5d3264e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1aa2d2fd21199cf18f8b7ea10e7fe229d5d3264e",
"triggerType" : "PUSH"
}, {
"hash" : "dab35428d0c43f93330d41d0ad21f218de5dba85",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dab35428d0c43f93330d41d0ad21f218de5dba85",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* e75a1af8faa1cc6fba9df9c71cc63467c68547ce UNKNOWN
* fd1fd272a3257e50206333b3283374a1b8304894 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6338)
* 1aa2d2fd21199cf18f8b7ea10e7fe229d5d3264e UNKNOWN
* dab35428d0c43f93330d41d0ad21f218de5dba85 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2153539528
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -378,6 +378,16 @@ public Comparable getOrderingValue(T record,
*/
public abstract HoodieRecord<T> constructHoodieRecord(BufferedRecord<T> bufferedRecord);
+ /**
+ * Constructs a new {@link HoodieRecord} based on the given record and schema.
+ *
+ * @param record The engine-specific row.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldName The name of the ordering field, if any.
+ * @return A new instance of {@link HoodieRecord}.
+ */
+ public abstract HoodieRecord<T> constructHoodieRecord(T record, Schema schema, Option<String> orderingFieldName);
Review Comment:
let's give it a try. In the end, I think we should introduce our engine agnostice data structures there to unify all the discrepencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985917751
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
}, {
"hash" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"triggerType" : "PUSH"
}, {
"hash" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335",
"triggerID" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"triggerType" : "PUSH"
}, {
"hash" : "fd1fd272a3257e50206333b3283374a1b8304894",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6338",
"triggerID" : "fd1fd272a3257e50206333b3283374a1b8304894",
"triggerType" : "PUSH"
}, {
"hash" : "1aa2d2fd21199cf18f8b7ea10e7fe229d5d3264e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1aa2d2fd21199cf18f8b7ea10e7fe229d5d3264e",
"triggerType" : "PUSH"
}, {
"hash" : "dab35428d0c43f93330d41d0ad21f218de5dba85",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6341",
"triggerID" : "dab35428d0c43f93330d41d0ad21f218de5dba85",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* e75a1af8faa1cc6fba9df9c71cc63467c68547ce UNKNOWN
* fd1fd272a3257e50206333b3283374a1b8304894 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6338)
* 1aa2d2fd21199cf18f8b7ea10e7fe229d5d3264e UNKNOWN
* dab35428d0c43f93330d41d0ad21f218de5dba85 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6341)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2153453714
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -384,10 +384,8 @@ public ClosableIterator<T> getClosableIterator() throws IOException {
* @return An iterator over the records that wraps the engine-specific record in a HoodieRecord.
*/
public ClosableIterator<HoodieRecord<T>> getClosableHoodieRecordIterator() throws IOException {
- return new CloseableMappingIterator<>(getClosableIterator(), nextRecord -> {
- BufferedRecord<T> bufferedRecord = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
Review Comment:
If we fix the `isDelete` flag here, the new introduced `#constructHoodieRecord` can be eliminated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2977512926
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* fd7cbdee9a399a48c8916638701916f5804be2d9 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241)
* 002a2f9920281401a0361b651ec6cde39624c935 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2977386495
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* fd7cbdee9a399a48c8916638701916f5804be2d9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241)
* 002a2f9920281401a0361b651ec6cde39624c935 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2977352234
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* fd7cbdee9a399a48c8916638701916f5804be2d9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151164771
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -110,6 +110,18 @@ public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRo
return new HoodieSparkRecord(hoodieKey, row, HoodieInternalRowUtils.getCachedSchema(schema), false);
}
+ @Override
+ public HoodieRecord<InternalRow> constructHoodieRecord(InternalRow row, Schema schema, Option<String> orderingFieldName) {
+ if (row instanceof HoodieInternalRow && ((HoodieInternalRow) row).isDeleteOperation()) {
+ return new HoodieEmptyRecord<>(
+ new HoodieKey(row.getUTF8String(HoodieRecord.RECORD_KEY_META_FIELD_ORD).toString(), partitionPath),
Review Comment:
The `HoodieSparkRecord` will be marked as a delete if the row is `null`. Using an empty record is the common pattern adopted across the engines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2978168684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 348ac1f28edef9731963d5c70335fd410e10df64 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151109547
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroReaderContext.java:
##########
@@ -120,7 +118,7 @@ public GenericRecord convertToAvroRecord(IndexedRecord record, Schema schema) {
@Override
public IndexedRecord getDeleteRow(IndexedRecord record, String recordKey) {
- throw new UnsupportedOperationException("Not supported for " + this.getClass().getSimpleName());
+ return new DeleteRecord(recordKey, record);
Review Comment:
The API is designed to return a record with the record keys set up ready in the record fields instead of a wrapper.
Things like `DeleteRecord.getRecordKey` is not a general API for readers..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2152444350
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1009,40 +1017,38 @@ private static boolean isDeleteRecord(HoodieTableMetaClient dataTableMetaClient,
}
}
- private static Map<String, HoodieRecord> getLogRecords(List<String> logFilePaths,
+ private static <T> ClosableIterator<HoodieRecord<T>> getLogRecords(List<String> logFilePaths,
HoodieTableMetaClient datasetMetaClient,
Option<Schema> writerSchemaOpt,
String latestCommitTimestamp,
- EngineType engineType) {
- if (writerSchemaOpt.isPresent()) {
+ String partitionPath,
+ HoodieReaderContext<T> readerContext) {
+ if (writerSchemaOpt.isPresent() && !logFilePaths.isEmpty()) {
+ List<HoodieLogFile> logFiles = logFilePaths.stream().map(HoodieLogFile::new).collect(Collectors.toList());
+ FileSlice fileSlice = new FileSlice(partitionPath, logFiles.get(0).getFileId(), logFiles.get(0).getDeltaCommitTime());
Review Comment:
Right now we don't rely on the instant time anywhere in the FileGroupReader but we can change this is there is a way to set it properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985146962
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
}, {
"hash" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* bb842430968a51bb4bc6532cea2332fae2cd6544 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324)
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* 19a487daecfc29d93bbbf3629872bf811f6a1666 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332)
* e75a1af8faa1cc6fba9df9c71cc63467c68547ce UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2154557912
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -378,6 +378,16 @@ public Comparable getOrderingValue(T record,
*/
public abstract HoodieRecord<T> constructHoodieRecord(BufferedRecord<T> bufferedRecord);
+ /**
+ * Constructs a new {@link HoodieRecord} based on the given record and schema.
+ *
+ * @param record The engine-specific row.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldName The name of the ordering field, if any.
+ * @return A new instance of {@link HoodieRecord}.
+ */
+ public abstract HoodieRecord<T> constructHoodieRecord(T record, Schema schema, Option<String> orderingFieldName);
Review Comment:
Pushed an update to add a check for whether the record is a delete operation in the reader context
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2992150146
> @the-other-tim-brown I think the `emitDeletes` support for HoodieRecord iterator brings in too much overhead than I thought, can we drop it in this PR, the delete keys fetching should be just used in the legacy code path, now we have streaming write to MDT, and the code should be removed in the future anyway(once the streaming write is stable).
>
> The `emitDeletes` is introduced mainly for streaming read scenarios with engine specific rows.
>
> Also can we revert the changes for size estimation into a serapate PR to make the review of the current one easier.
@danny0405 I don't understand what you are recommending here. Streaming write to the MDT is only used for Spark and I don't think there are plans to use it for other engines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985614339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
}, {
"hash" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"triggerType" : "PUSH"
}, {
"hash" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335",
"triggerID" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"triggerType" : "PUSH"
}, {
"hash" : "fd1fd272a3257e50206333b3283374a1b8304894",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6338",
"triggerID" : "fd1fd272a3257e50206333b3283374a1b8304894",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* e75a1af8faa1cc6fba9df9c71cc63467c68547ce UNKNOWN
* 049429375f5af2de03e3f64b3c912d3ea2c225b6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335)
* fd1fd272a3257e50206333b3283374a1b8304894 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6338)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2152447410
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/model/HoodieInternalRow.java:
##########
@@ -89,6 +92,26 @@ public HoodieInternalRow(UTF8String[] metaFields,
this.metaFields = metaFields;
this.sourceRow = sourceRow;
this.sourceContainsMetaFields = sourceContainsMetaFields;
+ this.isDeleteOperation = false;
+ }
+
+ private HoodieInternalRow(UTF8String recordKey,
Review Comment:
Yes, makes sense. I've cleaned up the constructors to all route through a common all-args constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2152944129
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/table/format/FlinkRowDataReaderContext.java:
##########
@@ -195,6 +195,16 @@ public HoodieRecord<RowData> constructHoodieRecord(BufferedRecord<RowData> buffe
return new HoodieFlinkRecord(hoodieKey, operation, bufferedRecord.getOrderingValue(), rowData);
}
+ @Override
+ public HoodieRecord<RowData> constructHoodieRecord(RowData rowData, Schema schema, Option<String> orderingFieldName) {
+ HoodieOperation operation = HoodieOperation.fromValue(rowData.getRowKind().toByteValue());
+ HoodieKey hoodieKey = new HoodieKey(getRecordKey(rowData, schema), partitionPath);
+ if (operation == HoodieOperation.DELETE) {
+ return new HoodieEmptyRecord<>(hoodieKey, HoodieOperation.DELETE, getOrderingValue(rowData, schema, orderingFieldName), HoodieRecord.HoodieRecordType.FLINK);
+ }
Review Comment:
We probably can use `HoodieDeleteHelper.createDeleteRecord` function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2155156953
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecord.java:
##########
@@ -41,12 +42,12 @@
*/
public class BufferedRecord<T> implements Serializable {
private final String recordKey;
- private final Comparable orderingValue;
+ private final Lazy<Comparable> orderingValue;
Review Comment:
When deriving the BufferedRecord from the record with `emitDeletes` we can run into issues trying to extract the ordering value since the record is null. This allows us to only perform that evaluation if required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2977343236
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2984488198
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* bb842430968a51bb4bc6532cea2332fae2cd6544 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2155833435
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -128,6 +130,7 @@ private HoodieFileGroupReader(HoodieReaderContext<T> readerContext, HoodieStorag
this.props = props;
this.start = start;
this.length = length;
+ this.emitDelete = emitDelete;
Review Comment:
I don't think we need this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown closed pull request #13445: [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil
URL: https://github.com/apache/hudi/pull/13445
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2978678773
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 348ac1f28edef9731963d5c70335fd410e10df64 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247)
* 2ed8e8da280f0937ed31c37cacf2c2e714bce27a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151163367
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -378,6 +378,16 @@ public Comparable getOrderingValue(T record,
*/
public abstract HoodieRecord<T> constructHoodieRecord(BufferedRecord<T> bufferedRecord);
+ /**
+ * Constructs a new {@link HoodieRecord} based on the given record and schema.
+ *
+ * @param record The engine-specific row.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldName The name of the ordering field, if any.
+ * @return A new instance of {@link HoodieRecord}.
+ */
+ public abstract HoodieRecord<T> constructHoodieRecord(T record, Schema schema, Option<String> orderingFieldName);
Review Comment:
the solution is already proposed here and it works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2978858475
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 2ed8e8da280f0937ed31c37cacf2c2e714bce27a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151380390
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/model/HoodieInternalRow.java:
##########
@@ -89,6 +92,26 @@ public HoodieInternalRow(UTF8String[] metaFields,
this.metaFields = metaFields;
this.sourceRow = sourceRow;
this.sourceContainsMetaFields = sourceContainsMetaFields;
+ this.isDeleteOperation = false;
+ }
+
+ private HoodieInternalRow(UTF8String recordKey,
Review Comment:
if this is anyway private, can you take in isDeleteOperation as an argument. and so we can set it to tru in L 114. Just that looking at constructor arguments, its not very apparent that this is for delete operation.
##########
hudi-common/src/main/java/org/apache/hudi/avro/DeleteIndexedRecord.java:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.avro;
+
+import org.apache.avro.Schema;
+import org.apache.avro.generic.IndexedRecord;
+
+public class DeleteIndexedRecord implements IndexedRecord {
+ private final String[] metaFields;
+ private final IndexedRecord record;
+
+ public DeleteIndexedRecord(String recordKey, String partitionPath, IndexedRecord record) {
Review Comment:
IndexedDeleteRecord
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/model/HoodieInternalRow.java:
##########
@@ -58,6 +58,8 @@ public class HoodieInternalRow extends InternalRow {
*/
private final UTF8String[] metaFields;
private final InternalRow sourceRow;
+ // indicates whether this row represents a delete operation. Used in the CDC read.
+ private final boolean isDeleteOperation;
Review Comment:
why not just `isDeleted`
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2376,35 +2383,29 @@ public static HoodieData<HoodieRecord> readRecordKeysFromFileSlices(HoodieEngine
final int parallelism = Math.min(partitionFileSlicePairs.size(), recordIndexMaxParallelism);
final StoragePath basePath = metaClient.getBasePath();
final StorageConfiguration<?> storageConf = metaClient.getStorageConf();
+ final Schema tableSchema;
+ try {
+ tableSchema = new TableSchemaResolver(metaClient).getTableAvroSchema();
+ } catch (Exception e) {
+ throw new HoodieException("Unable to resolve table schema for table", e);
+ }
+ ReaderContextFactory<T> readerContextFactory = engineContext.getReaderContextFactory(metaClient);
return engineContext.parallelize(partitionFileSlicePairs, parallelism).flatMap(partitionAndBaseFile -> {
final String partition = partitionAndBaseFile.getKey();
final FileSlice fileSlice = partitionAndBaseFile.getValue();
if (!fileSlice.getBaseFile().isPresent()) {
- List<String> logFilePaths = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator())
- .map(l -> l.getPath().toString()).collect(Collectors.toList());
- HoodieMergedLogRecordScanner mergedLogRecordScanner = HoodieMergedLogRecordScanner.newBuilder()
- .withStorage(metaClient.getStorage())
- .withBasePath(basePath)
- .withLogFilePaths(logFilePaths)
- .withReaderSchema(HoodieAvroUtils.getRecordKeySchema())
- .withLatestInstantTime(metaClient.getActiveTimeline().filterCompletedInstants().lastInstant().map(HoodieInstant::requestedTime).orElse(""))
- .withReverseReader(false)
- .withMaxMemorySizeInBytes(storageConf.getLong(
- MAX_MEMORY_FOR_COMPACTION.key(), DEFAULT_MAX_MEMORY_FOR_SPILLABLE_MAP_IN_BYTES))
- .withSpillableMapBasePath(FileIOUtils.getDefaultSpillableMapBasePath())
- .withPartition(fileSlice.getPartitionPath())
- .withOptimizedLogBlocksScan(storageConf.getBoolean(ENABLE_OPTIMIZED_LOG_BLOCKS_SCAN.key(), false))
- .withDiskMapType(storageConf.getEnum(SPILLABLE_DISK_MAP_TYPE.key(), SPILLABLE_DISK_MAP_TYPE.defaultValue()))
- .withBitCaskDiskMapCompressionEnabled(storageConf.getBoolean(
- DISK_MAP_BITCASK_COMPRESSION_ENABLED.key(), DISK_MAP_BITCASK_COMPRESSION_ENABLED.defaultValue()))
- .withRecordMerger(HoodieRecordUtils.createRecordMerger(
- metaClient.getBasePath().toString(),
- engineType,
- Collections.emptyList(), // TODO: support different merger classes, which is currently only known to write config
- metaClient.getTableConfig().getRecordMergeStrategyId()))
- .withTableMetaClient(metaClient)
+ HoodieFileGroupReader fileGroupReader = HoodieFileGroupReader.<T>newBuilder()
+ .withReaderContext(readerContextFactory.getContext())
+ .withHoodieTableMetaClient(metaClient)
+ .withFileSlice(fileSlice)
+ .withDataSchema(tableSchema)
+ .withRequestedSchema(HoodieAvroUtils.getRecordKeySchema())
+ .withLatestCommitTime(metaClient.getActiveTimeline().filterCompletedInstants().lastInstant().map(HoodieInstant::requestedTime).orElse(""))
Review Comment:
can we compute the latestInstant in driver and let the executor access it directly.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1009,40 +1017,38 @@ private static boolean isDeleteRecord(HoodieTableMetaClient dataTableMetaClient,
}
}
- private static Map<String, HoodieRecord> getLogRecords(List<String> logFilePaths,
+ private static <T> ClosableIterator<HoodieRecord<T>> getLogRecords(List<String> logFilePaths,
HoodieTableMetaClient datasetMetaClient,
Option<Schema> writerSchemaOpt,
String latestCommitTimestamp,
- EngineType engineType) {
- if (writerSchemaOpt.isPresent()) {
+ String partitionPath,
+ HoodieReaderContext<T> readerContext) {
+ if (writerSchemaOpt.isPresent() && !logFilePaths.isEmpty()) {
+ List<HoodieLogFile> logFiles = logFilePaths.stream().map(HoodieLogFile::new).collect(Collectors.toList());
+ FileSlice fileSlice = new FileSlice(partitionPath, logFiles.get(0).getFileId(), logFiles.get(0).getDeltaCommitTime());
Review Comment:
delta commit time of first log file may not be the file slice's base instant time right. but I see here we are setting it so.
Can you confirm this does not have any other side effects.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/DeleteRecord.java:
##########
@@ -107,7 +107,7 @@ public int hashCode() {
@Override
public String toString() {
- return "DeleteRecord {"
+ return "DeleteIndexedRecord {"
Review Comment:
should this be reverted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2154559779
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -171,9 +183,6 @@ public Comparable convertValueToEngineType(Comparable value) {
@Override
public InternalRow getDeleteRow(InternalRow record, String recordKey) {
- if (record != null) {
Review Comment:
I updated the handling here to simply wrap the row if it is non-null so we can still detect it is a delete but otherwise do not modify it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2155800910
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecordSerializer.java:
##########
@@ -34,49 +36,83 @@
*
*/
public class BufferedRecordSerializer<T> implements CustomSerializer<BufferedRecord<T>> {
- public static final int KRYO_SERIALIZER_INITIAL_BUFFER_SIZE = 1048576;
- private final Kryo kryo;
- // Caching ByteArrayOutputStream to avoid recreating it for every operation
- private final ByteArrayOutputStream baos;
- private final RecordSerializer<T> recordSerializer;
+ // Caching kryo serializer to avoid creating kryo instance for every serde operation
+ private static final ThreadLocal<InternalSerializerInstance> SERIALIZER_REF =
Review Comment:
The kryo creation happens once for each file group reader so the cost should be low?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2980644666
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 2ed8e8da280f0937ed31c37cacf2c2e714bce27a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257)
* 11a4df01b69b973f5fce11fbb574b1edba4ca19c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2993216350
> Streaming write to the MDT is only used for Spark and I don't think there are plans to use it for other engines.
That's true, for Flink and Java, there is no even a solution/plan to support the RLI there, that is why I said those codes like constructing RLI from files should be deemed as legacy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2977857823
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 002a2f9920281401a0361b651ec6cde39624c935 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243)
* 348ac1f28edef9731963d5c70335fd410e10df64 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2981016549
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 11a4df01b69b973f5fce11fbb574b1edba4ca19c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985568024
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
}, {
"hash" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"triggerType" : "PUSH"
}, {
"hash" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335",
"triggerID" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"triggerType" : "PUSH"
}, {
"hash" : "fd1fd272a3257e50206333b3283374a1b8304894",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd1fd272a3257e50206333b3283374a1b8304894",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* e75a1af8faa1cc6fba9df9c71cc63467c68547ce UNKNOWN
* 049429375f5af2de03e3f64b3c912d3ea2c225b6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335)
* fd1fd272a3257e50206333b3283374a1b8304894 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2978765155
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 348ac1f28edef9731963d5c70335fd410e10df64 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247)
* 2ed8e8da280f0937ed31c37cacf2c2e714bce27a Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151119580
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -171,9 +183,6 @@ public Comparable convertValueToEngineType(Comparable value) {
@Override
public InternalRow getDeleteRow(InternalRow record, String recordKey) {
- if (record != null) {
Review Comment:
The spark row doesn't have an operation type field like the flink row data does. Now this will always use an internal row to represent the row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985026767
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* bb842430968a51bb4bc6532cea2332fae2cd6544 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324)
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985320999
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
}, {
"hash" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"triggerType" : "PUSH"
}, {
"hash" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335",
"triggerID" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* 19a487daecfc29d93bbbf3629872bf811f6a1666 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332)
* e75a1af8faa1cc6fba9df9c71cc63467c68547ce UNKNOWN
* 049429375f5af2de03e3f64b3c912d3ea2c225b6 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985547716
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
}, {
"hash" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"triggerType" : "PUSH"
}, {
"hash" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335",
"triggerID" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* e75a1af8faa1cc6fba9df9c71cc63467c68547ce UNKNOWN
* 049429375f5af2de03e3f64b3c912d3ea2c225b6 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2977347851
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* fd7cbdee9a399a48c8916638701916f5804be2d9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2152436832
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/client/model/HoodieInternalRow.java:
##########
@@ -58,6 +58,8 @@ public class HoodieInternalRow extends InternalRow {
*/
private final UTF8String[] metaFields;
private final InternalRow sourceRow;
+ // indicates whether this row represents a delete operation. Used in the CDC read.
+ private final boolean isDeleteOperation;
Review Comment:
This was to better align with the operation type used in Flink
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151128477
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -378,6 +378,16 @@ public Comparable getOrderingValue(T record,
*/
public abstract HoodieRecord<T> constructHoodieRecord(BufferedRecord<T> bufferedRecord);
+ /**
+ * Constructs a new {@link HoodieRecord} based on the given record and schema.
+ *
+ * @param record The engine-specific row.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldName The name of the ordering field, if any.
+ * @return A new instance of {@link HoodieRecord}.
+ */
+ public abstract HoodieRecord<T> constructHoodieRecord(T record, Schema schema, Option<String> orderingFieldName);
Review Comment:
> If there is a way to do this across all engine types (not just Flink) then we can use that.
Let's figure out a solution for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151162216
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -171,9 +183,6 @@ public Comparable convertValueToEngineType(Comparable value) {
@Override
public InternalRow getDeleteRow(InternalRow record, String recordKey) {
- if (record != null) {
Review Comment:
When we read the data from the log block, it is not going to have an operation type set. We can determine it is a delete by inspecting the record for `_hoodie_is_delete` or by a field used to mark deletes or the merger will output an empty option to signal a delete in the case of custom mergers and payloads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151183064
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -378,6 +378,16 @@ public Comparable getOrderingValue(T record,
*/
public abstract HoodieRecord<T> constructHoodieRecord(BufferedRecord<T> bufferedRecord);
+ /**
+ * Constructs a new {@link HoodieRecord} based on the given record and schema.
+ *
+ * @param record The engine-specific row.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldName The name of the ordering field, if any.
+ * @return A new instance of {@link HoodieRecord}.
+ */
+ public abstract HoodieRecord<T> constructHoodieRecord(T record, Schema schema, Option<String> orderingFieldName);
Review Comment:
only other option I see is to make the HoodieReaderContext have a method for `isDeleteOperation(T record)` and then we can use that to set the value for `isDelete` in this [line](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java#L388), but that is just a roundabout way to get back to the same result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151084857
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -1009,40 +1017,38 @@ private static boolean isDeleteRecord(HoodieTableMetaClient dataTableMetaClient,
}
}
- private static Map<String, HoodieRecord> getLogRecords(List<String> logFilePaths,
+ private static <T> ClosableIterator<HoodieRecord<T>> getLogRecords(List<String> logFilePaths,
HoodieTableMetaClient datasetMetaClient,
Option<Schema> writerSchemaOpt,
String latestCommitTimestamp,
Review Comment:
Incorrect indention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151104904
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -110,6 +110,18 @@ public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRo
return new HoodieSparkRecord(hoodieKey, row, HoodieInternalRowUtils.getCachedSchema(schema), false);
}
+ @Override
+ public HoodieRecord<InternalRow> constructHoodieRecord(InternalRow row, Schema schema, Option<String> orderingFieldName) {
+ if (row instanceof HoodieInternalRow && ((HoodieInternalRow) row).isDeleteOperation()) {
+ return new HoodieEmptyRecord<>(
+ new HoodieKey(row.getUTF8String(HoodieRecord.RECORD_KEY_META_FIELD_ORD).toString(), partitionPath),
Review Comment:
if the `row` already been marked as `delete`, we can just construct `HoodieSparkRecord`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985127534
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* bb842430968a51bb4bc6532cea2332fae2cd6544 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324)
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2153448619
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -171,9 +183,6 @@ public Comparable convertValueToEngineType(Comparable value) {
@Override
public InternalRow getDeleteRow(InternalRow record, String recordKey) {
- if (record != null) {
Review Comment:
Returns a `InternalRow` with just record key and patition path set up there will lost the other payload fields, not sure if it works for cdc read scenarios(the deletes for retraction)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985151045
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
}, {
"hash" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* 19a487daecfc29d93bbbf3629872bf811f6a1666 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332)
* e75a1af8faa1cc6fba9df9c71cc63467c68547ce UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2154505324
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -110,6 +110,18 @@ public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRo
return new HoodieSparkRecord(hoodieKey, row, HoodieInternalRowUtils.getCachedSchema(schema), false);
}
+ @Override
+ public HoodieRecord<InternalRow> constructHoodieRecord(InternalRow row, Schema schema, Option<String> orderingFieldName) {
+ if (row instanceof HoodieInternalRow && ((HoodieInternalRow) row).isDeleteOperation()) {
+ return new HoodieEmptyRecord<>(
+ new HoodieKey(row.getUTF8String(HoodieRecord.RECORD_KEY_META_FIELD_ORD).toString(), partitionPath),
Review Comment:
@linliu-code what is the impact of using `EmptyHoodieRecordPayload` vs `HoodieEmptyRecord`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2153512087
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -384,10 +384,8 @@ public ClosableIterator<T> getClosableIterator() throws IOException {
* @return An iterator over the records that wraps the engine-specific record in a HoodieRecord.
*/
public ClosableIterator<HoodieRecord<T>> getClosableHoodieRecordIterator() throws IOException {
- return new CloseableMappingIterator<>(getClosableIterator(), nextRecord -> {
- BufferedRecord<T> bufferedRecord = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
Review Comment:
This is what I recommended to address this: https://github.com/apache/hudi/pull/13445/files/11a4df01b69b973f5fce11fbb574b1edba4ca19c#r2151183064
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2153510719
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -384,10 +384,8 @@ public ClosableIterator<T> getClosableIterator() throws IOException {
* @return An iterator over the records that wraps the engine-specific record in a HoodieRecord.
*/
public ClosableIterator<HoodieRecord<T>> getClosableHoodieRecordIterator() throws IOException {
- return new CloseableMappingIterator<>(getClosableIterator(), nextRecord -> {
- BufferedRecord<T> bufferedRecord = BufferedRecord.forRecordWithContext(nextRecord, readerContext.getSchemaHandler().getRequestedSchema(), readerContext, orderingFieldName, false);
Review Comment:
yes, just needs some way to set this properly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2977920872
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 002a2f9920281401a0361b651ec6cde39624c935 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243)
* 348ac1f28edef9731963d5c70335fd410e10df64 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151174324
##########
hudi-common/src/main/java/org/apache/hudi/avro/HoodieAvroReaderContext.java:
##########
@@ -120,7 +118,7 @@ public GenericRecord convertToAvroRecord(IndexedRecord record, Schema schema) {
@Override
public IndexedRecord getDeleteRow(IndexedRecord record, String recordKey) {
- throw new UnsupportedOperationException("Not supported for " + this.getClass().getSimpleName());
+ return new DeleteRecord(recordKey, record);
Review Comment:
Updated this to match the pattern with HoodieInternalRow
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2977390573
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* fd7cbdee9a399a48c8916638701916f5804be2d9 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241)
* 002a2f9920281401a0361b651ec6cde39624c935 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151110654
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -378,6 +378,16 @@ public Comparable getOrderingValue(T record,
*/
public abstract HoodieRecord<T> constructHoodieRecord(BufferedRecord<T> bufferedRecord);
+ /**
+ * Constructs a new {@link HoodieRecord} based on the given record and schema.
+ *
+ * @param record The engine-specific row.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldName The name of the ordering field, if any.
+ * @return A new instance of {@link HoodieRecord}.
+ */
+ public abstract HoodieRecord<T> constructHoodieRecord(T record, Schema schema, Option<String> orderingFieldName);
Review Comment:
I don't think we need this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151104385
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -171,9 +183,6 @@ public Comparable convertValueToEngineType(Comparable value) {
@Override
public InternalRow getDeleteRow(InternalRow record, String recordKey) {
- if (record != null) {
Review Comment:
why remove this? The record should already been marked as `delete` in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2980764452
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 2ed8e8da280f0937ed31c37cacf2c2e714bce27a Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257)
* 11a4df01b69b973f5fce11fbb574b1edba4ca19c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2986314023
@the-other-tim-brown I think the `emitDeletes` support for HoodieRecord iterator brings in too much overhead than I thought, can we drop it in this PR, the delete keys fetching should be just used in the legacy code path, now we have streaming write to MDT, and the code may be removed in the future anyway.
Also can we revert the changes for size estimation into a serapate PR to make the review of the current one easier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985781023
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
}, {
"hash" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"triggerType" : "PUSH"
}, {
"hash" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335",
"triggerID" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"triggerType" : "PUSH"
}, {
"hash" : "fd1fd272a3257e50206333b3283374a1b8304894",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6338",
"triggerID" : "fd1fd272a3257e50206333b3283374a1b8304894",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* e75a1af8faa1cc6fba9df9c71cc63467c68547ce UNKNOWN
* fd1fd272a3257e50206333b3283374a1b8304894 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6338)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2986073737
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
}, {
"hash" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"triggerType" : "PUSH"
}, {
"hash" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335",
"triggerID" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"triggerType" : "PUSH"
}, {
"hash" : "fd1fd272a3257e50206333b3283374a1b8304894",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6338",
"triggerID" : "fd1fd272a3257e50206333b3283374a1b8304894",
"triggerType" : "PUSH"
}, {
"hash" : "1aa2d2fd21199cf18f8b7ea10e7fe229d5d3264e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1aa2d2fd21199cf18f8b7ea10e7fe229d5d3264e",
"triggerType" : "PUSH"
}, {
"hash" : "dab35428d0c43f93330d41d0ad21f218de5dba85",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6341",
"triggerID" : "dab35428d0c43f93330d41d0ad21f218de5dba85",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* e75a1af8faa1cc6fba9df9c71cc63467c68547ce UNKNOWN
* 1aa2d2fd21199cf18f8b7ea10e7fe229d5d3264e UNKNOWN
* dab35428d0c43f93330d41d0ad21f218de5dba85 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6341)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2984217620
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 11a4df01b69b973f5fce11fbb574b1edba4ca19c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278)
* bb842430968a51bb4bc6532cea2332fae2cd6544 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2155796342
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -192,7 +192,7 @@ public SizeEstimator<BufferedRecord<T>> getRecordSizeEstimator() {
}
public CustomSerializer<BufferedRecord<T>> getRecordSerializer() {
- return new DefaultSerializer<>();
+ return new BufferedRecordSerializer<>();
Review Comment:
why this change? It seems not necessary based on the benchmark: https://github.com/apache/hudi/pull/13408#discussion_r2139076809
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2152654287
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2376,35 +2383,29 @@ public static HoodieData<HoodieRecord> readRecordKeysFromFileSlices(HoodieEngine
final int parallelism = Math.min(partitionFileSlicePairs.size(), recordIndexMaxParallelism);
final StoragePath basePath = metaClient.getBasePath();
final StorageConfiguration<?> storageConf = metaClient.getStorageConf();
+ final Schema tableSchema;
+ try {
+ tableSchema = new TableSchemaResolver(metaClient).getTableAvroSchema();
+ } catch (Exception e) {
+ throw new HoodieException("Unable to resolve table schema for table", e);
+ }
+ ReaderContextFactory<T> readerContextFactory = engineContext.getReaderContextFactory(metaClient);
return engineContext.parallelize(partitionFileSlicePairs, parallelism).flatMap(partitionAndBaseFile -> {
final String partition = partitionAndBaseFile.getKey();
final FileSlice fileSlice = partitionAndBaseFile.getValue();
if (!fileSlice.getBaseFile().isPresent()) {
- List<String> logFilePaths = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator())
- .map(l -> l.getPath().toString()).collect(Collectors.toList());
- HoodieMergedLogRecordScanner mergedLogRecordScanner = HoodieMergedLogRecordScanner.newBuilder()
- .withStorage(metaClient.getStorage())
- .withBasePath(basePath)
- .withLogFilePaths(logFilePaths)
- .withReaderSchema(HoodieAvroUtils.getRecordKeySchema())
- .withLatestInstantTime(metaClient.getActiveTimeline().filterCompletedInstants().lastInstant().map(HoodieInstant::requestedTime).orElse(""))
- .withReverseReader(false)
- .withMaxMemorySizeInBytes(storageConf.getLong(
- MAX_MEMORY_FOR_COMPACTION.key(), DEFAULT_MAX_MEMORY_FOR_SPILLABLE_MAP_IN_BYTES))
- .withSpillableMapBasePath(FileIOUtils.getDefaultSpillableMapBasePath())
- .withPartition(fileSlice.getPartitionPath())
- .withOptimizedLogBlocksScan(storageConf.getBoolean(ENABLE_OPTIMIZED_LOG_BLOCKS_SCAN.key(), false))
- .withDiskMapType(storageConf.getEnum(SPILLABLE_DISK_MAP_TYPE.key(), SPILLABLE_DISK_MAP_TYPE.defaultValue()))
- .withBitCaskDiskMapCompressionEnabled(storageConf.getBoolean(
- DISK_MAP_BITCASK_COMPRESSION_ENABLED.key(), DISK_MAP_BITCASK_COMPRESSION_ENABLED.defaultValue()))
- .withRecordMerger(HoodieRecordUtils.createRecordMerger(
- metaClient.getBasePath().toString(),
- engineType,
- Collections.emptyList(), // TODO: support different merger classes, which is currently only known to write config
- metaClient.getTableConfig().getRecordMergeStrategyId()))
- .withTableMetaClient(metaClient)
+ HoodieFileGroupReader fileGroupReader = HoodieFileGroupReader.<T>newBuilder()
+ .withReaderContext(readerContextFactory.getContext())
+ .withHoodieTableMetaClient(metaClient)
+ .withFileSlice(fileSlice)
+ .withDataSchema(tableSchema)
+ .withRequestedSchema(HoodieAvroUtils.getRecordKeySchema())
+ .withLatestCommitTime(metaClient.getActiveTimeline().filterCompletedInstants().lastInstant().map(HoodieInstant::requestedTime).orElse(""))
Review Comment:
Yes, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2152939664
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -850,6 +846,7 @@ public static HoodieData<HoodieRecord> convertMetadataToRecordIndexRecords(Hoodi
StorageConfiguration storageConfiguration = dataTableMetaClient.getStorageConf();
Option<Schema> writerSchemaOpt = tryResolveSchemaForTable(dataTableMetaClient);
Option<Schema> finalWriterSchemaOpt = writerSchemaOpt;
+ ReaderContextFactory<T> readerContextFactory = engineContext.getReaderContextFactory(dataTableMetaClient);
Review Comment:
Curious, should we consider initializing `InstantRange` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2153511772
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -850,6 +846,7 @@ public static HoodieData<HoodieRecord> convertMetadataToRecordIndexRecords(Hoodi
StorageConfiguration storageConfiguration = dataTableMetaClient.getStorageConf();
Option<Schema> writerSchemaOpt = tryResolveSchemaForTable(dataTableMetaClient);
Option<Schema> finalWriterSchemaOpt = writerSchemaOpt;
+ ReaderContextFactory<T> readerContextFactory = engineContext.getReaderContextFactory(dataTableMetaClient);
Review Comment:
I don't see it set in the current code, what would be the side-effect of not adding it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2977750919
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 002a2f9920281401a0361b651ec6cde39624c935 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2151125988
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -171,9 +183,6 @@ public Comparable convertValueToEngineType(Comparable value) {
@Override
public InternalRow getDeleteRow(InternalRow record, String recordKey) {
- if (record != null) {
Review Comment:
Are you saying the records written to log data block from spark is always not a delete? This is not true as when the customized delete marker is there.
Can we return `HoodieInternalRow` with `isDelete` set up in there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2152441367
##########
hudi-common/src/main/java/org/apache/hudi/common/model/DeleteRecord.java:
##########
@@ -107,7 +107,7 @@ public int hashCode() {
@Override
public String toString() {
- return "DeleteRecord {"
+ return "DeleteIndexedRecord {"
Review Comment:
Yes, this was a bad find and replace on my part
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2152919414
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -110,6 +110,18 @@ public HoodieRecord<InternalRow> constructHoodieRecord(BufferedRecord<InternalRo
return new HoodieSparkRecord(hoodieKey, row, HoodieInternalRowUtils.getCachedSchema(schema), false);
}
+ @Override
+ public HoodieRecord<InternalRow> constructHoodieRecord(InternalRow row, Schema schema, Option<String> orderingFieldName) {
+ if (row instanceof HoodieInternalRow && ((HoodieInternalRow) row).isDeleteOperation()) {
+ return new HoodieEmptyRecord<>(
+ new HoodieKey(row.getUTF8String(HoodieRecord.RECORD_KEY_META_FIELD_ORD).toString(), partitionPath),
Review Comment:
@the-other-tim-brown, @danny0405 , should we fix `HoodieDeleteHelper` as well, since for Avro, we use `HoodieAvroRecord` with `EmptyHoodieRecordPayload`; what about Flink, Hive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2153542902
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -378,6 +378,16 @@ public Comparable getOrderingValue(T record,
*/
public abstract HoodieRecord<T> constructHoodieRecord(BufferedRecord<T> bufferedRecord);
+ /**
+ * Constructs a new {@link HoodieRecord} based on the given record and schema.
+ *
+ * @param record The engine-specific row.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldName The name of the ordering field, if any.
+ * @return A new instance of {@link HoodieRecord}.
+ */
+ public abstract HoodieRecord<T> constructHoodieRecord(T record, Schema schema, Option<String> orderingFieldName);
Review Comment:
Another option is to have the FGReader return BufferedRecords directly which will already have this context for whether it represents a delete for that row. There are some other changes that may need to come with that like making the ordering value lookup lazy to avoid extra overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2153448619
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -171,9 +183,6 @@ public Comparable convertValueToEngineType(Comparable value) {
@Override
public InternalRow getDeleteRow(InternalRow record, String recordKey) {
- if (record != null) {
Review Comment:
Returns a `InternalRow` with just record key and patition path set up there will lost the other payload fields, not sure it it works for cdc read scenarios(the deletes for retraction)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2154458897
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -378,6 +378,16 @@ public Comparable getOrderingValue(T record,
*/
public abstract HoodieRecord<T> constructHoodieRecord(BufferedRecord<T> bufferedRecord);
+ /**
+ * Constructs a new {@link HoodieRecord} based on the given record and schema.
+ *
+ * @param record The engine-specific row.
+ * @param schema The Avro schema of the record.
+ * @param orderingFieldName The name of the ordering field, if any.
+ * @return A new instance of {@link HoodieRecord}.
+ */
+ public abstract HoodieRecord<T> constructHoodieRecord(T record, Schema schema, Option<String> orderingFieldName);
Review Comment:
The tricky part here is only when there are log files or merging we could have a `BufferedRecord`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985294490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
}, {
"hash" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"triggerType" : "PUSH"
}, {
"hash" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* 19a487daecfc29d93bbbf3629872bf811f6a1666 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332)
* e75a1af8faa1cc6fba9df9c71cc63467c68547ce UNKNOWN
* 049429375f5af2de03e3f64b3c912d3ea2c225b6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13445:
URL: https://github.com/apache/hudi/pull/13445#discussion_r2155799138
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/BufferedRecord.java:
##########
@@ -41,12 +42,12 @@
*/
public class BufferedRecord<T> implements Serializable {
private final String recordKey;
- private final Comparable orderingValue;
+ private final Lazy<Comparable> orderingValue;
Review Comment:
We already cache the `orderingVal` in `HoodieRecord`, the `Lazy` wrapper will increase the bytes to serialize for the spillableMap too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "the-other-tim-brown (via GitHub)" <gi...@apache.org>.
the-other-tim-brown commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2992646201
Started a new PR with the same end result of moving off deprecated code but smaller changeset https://github.com/apache/hudi/pull/13470
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2984144891
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 11a4df01b69b973f5fce11fbb574b1edba4ca19c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278)
* bb842430968a51bb4bc6532cea2332fae2cd6544 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985099554
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* bb842430968a51bb4bc6532cea2332fae2cd6544 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324)
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-9527] Switch to HoodieFileGroupReader in HoodieTableMetadataUtil [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13445:
URL: https://github.com/apache/hudi/pull/13445#issuecomment-2985812207
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173",
"triggerType" : "PUSH"
}, {
"hash" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6241",
"triggerID" : "fd7cbdee9a399a48c8916638701916f5804be2d9",
"triggerType" : "PUSH"
}, {
"hash" : "002a2f9920281401a0361b651ec6cde39624c935",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6243",
"triggerID" : "002a2f9920281401a0361b651ec6cde39624c935",
"triggerType" : "PUSH"
}, {
"hash" : "348ac1f28edef9731963d5c70335fd410e10df64",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6247",
"triggerID" : "348ac1f28edef9731963d5c70335fd410e10df64",
"triggerType" : "PUSH"
}, {
"hash" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6257",
"triggerID" : "2ed8e8da280f0937ed31c37cacf2c2e714bce27a",
"triggerType" : "PUSH"
}, {
"hash" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6278",
"triggerID" : "11a4df01b69b973f5fce11fbb574b1edba4ca19c",
"triggerType" : "PUSH"
}, {
"hash" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6324",
"triggerID" : "bb842430968a51bb4bc6532cea2332fae2cd6544",
"triggerType" : "PUSH"
}, {
"hash" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a",
"triggerType" : "PUSH"
}, {
"hash" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2ca6f2d1ef89f973291cbe2372b28647829faff3",
"triggerType" : "PUSH"
}, {
"hash" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "26f86c3b49c18facba107c80d9fafd2e8e82d62d",
"triggerType" : "PUSH"
}, {
"hash" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6332",
"triggerID" : "19a487daecfc29d93bbbf3629872bf811f6a1666",
"triggerType" : "PUSH"
}, {
"hash" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e75a1af8faa1cc6fba9df9c71cc63467c68547ce",
"triggerType" : "PUSH"
}, {
"hash" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6335",
"triggerID" : "049429375f5af2de03e3f64b3c912d3ea2c225b6",
"triggerType" : "PUSH"
}, {
"hash" : "fd1fd272a3257e50206333b3283374a1b8304894",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6338",
"triggerID" : "fd1fd272a3257e50206333b3283374a1b8304894",
"triggerType" : "PUSH"
}, {
"hash" : "1aa2d2fd21199cf18f8b7ea10e7fe229d5d3264e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1aa2d2fd21199cf18f8b7ea10e7fe229d5d3264e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2cb77f4b40bd4c8250c9aa67f3f22bcf76c173 UNKNOWN
* 57e41ca2b93ffa6c0891673e0a1f72f5cc57da3a UNKNOWN
* 2ca6f2d1ef89f973291cbe2372b28647829faff3 UNKNOWN
* 26f86c3b49c18facba107c80d9fafd2e8e82d62d UNKNOWN
* e75a1af8faa1cc6fba9df9c71cc63467c68547ce UNKNOWN
* fd1fd272a3257e50206333b3283374a1b8304894 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=6338)
* 1aa2d2fd21199cf18f8b7ea10e7fe229d5d3264e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org