You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2023/11/17 21:12:39 UTC
[PR] non-working. Want to review with team that this makes sense [hudi]
jonvex opened a new pull request, #10137:
URL: https://github.com/apache/hudi/pull/10137
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1824965339
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408132664
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/common/model/HoodieSparkRecord.java:
##########
@@ -449,7 +449,8 @@ private static void validateRow(InternalRow data, StructType schema) {
data instanceof HoodieInternalRow
|| data instanceof GenericInternalRow
|| data instanceof SpecificInternalRow
- || SparkAdapterSupport$.MODULE$.sparkAdapter().isColumnarBatchRow(data));
+ || SparkAdapterSupport$.MODULE$.sparkAdapter().isColumnarBatchRow(data))
+ || data instanceof JoinedRow;
Review Comment:
Bootstrap tables when we need to merge skeleton and base file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1843845749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
}, {
"hash" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347",
"triggerID" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"triggerType" : "PUSH"
}, {
"hash" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 993d0dc63f3f552b8bf5b52c113f3ae8ef53304c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241)
* 47e6224db0b980c0b3697b282fa724a67b2cca31 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345)
* c5f2e8da635bb7235d8b7719d7904df6c7489e0b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346)
* 8048ed72bc9a21e9d178445fdc39e1a379f183e0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347)
* 361e800774f0d6789a550fcc63c8764416ff54cc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1843794116
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
}, {
"hash" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 993d0dc63f3f552b8bf5b52c113f3ae8ef53304c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241)
* 47e6224db0b980c0b3697b282fa724a67b2cca31 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345)
* c5f2e8da635bb7235d8b7719d7904df6c7489e0b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346)
* 8048ed72bc9a21e9d178445fdc39e1a379f183e0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1843716815
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 993d0dc63f3f552b8bf5b52c113f3ae8ef53304c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241)
* 47e6224db0b980c0b3697b282fa724a67b2cca31 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345)
* c5f2e8da635bb7235d8b7719d7904df6c7489e0b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1845957310
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
}, {
"hash" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347",
"triggerID" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"triggerType" : "PUSH"
}, {
"hash" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350",
"triggerID" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"triggerType" : "PUSH"
}, {
"hash" : "830f88ca3021165497428d27e5e5fd19e5f3f41d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21373",
"triggerID" : "830f88ca3021165497428d27e5e5fd19e5f3f41d",
"triggerType" : "PUSH"
}, {
"hash" : "d6d0848e183d0a192fda67f647f81adcbf6f5922",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d6d0848e183d0a192fda67f647f81adcbf6f5922",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 361e800774f0d6789a550fcc63c8764416ff54cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350)
* 830f88ca3021165497428d27e5e5fd19e5f3f41d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21373)
* d6d0848e183d0a192fda67f647f81adcbf6f5922 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] non-working. Want to review with team that this makes sense [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1817170197
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 184858b13509316eeb7fe0a1198b7df98c3fc50d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] non-working. Want to review with team that this makes sense [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1817159471
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408151212
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -62,7 +63,7 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: Option[Partitioned
requiredSchema: Schema,
conf: Configuration): ClosableIterator[InternalRow] = {
val fileInfo = sparkAdapter.getSparkPartitionedFileUtils
- .createPartitionedFile(partitionValues, filePath, start, length)
+ .createPartitionedFile(InternalRow.empty, filePath, start, length)
Review Comment:
Ok so part of this PR is to clean up the philosophy of the fg reader.
We want to give the fg reader a requested schema, and we want the output to be an iterator with records in exactly that schema (maybe not with cdc, haven't given it much thought yet).
spark parquet file reader will append the partition values to the end of each record. Putting the logic for dealing with that inside of the fg reader is adding unnecessary stuff into the fg reader that is probably only relevant for spark.
Therefore I think it makes sense for HoodieFileGroupReaderBasedParquetFileFormat to be responsible for appending the partition columns to the end. In addition to being better organized in my opinion, this will also be more performant because it prevents adding several calls that append the partition col and then project the column away.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1832884937
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* dfa3bdee07f850efbacb55ecc84637339a953423 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225)
* 747894399d3bef0e05c561d9c67db61ab2536cf9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228)
* 31fe075c72fb189b9155e48ab3399e9199cc293a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828581987
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 2a9a36323053c5d35ae68cc5b2c38bab707b0f1a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159)
* 38b26030e7df02d865cfe4efd03a955124d2272a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1411320238
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +86,170 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
Review Comment:
Can we use this constructor by just passing in the query type and meta client so it can automatically figure out the file slice to read for a file group reader? Let's keep this constructor.
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -99,4 +104,49 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: Option[Partitioned
})
deserializer.deserialize(avroRecord).get.asInstanceOf[InternalRow]
}
+
+ override def mergeBootstrapReaders(skeletonFileIterator: ClosableIterator[InternalRow],
+ dataFileIterator: ClosableIterator[InternalRow]): ClosableIterator[InternalRow] = {
+ doBootstrapMerge(skeletonFileIterator.asInstanceOf[ClosableIterator[Any]],
+ dataFileIterator.asInstanceOf[ClosableIterator[Any]])
+ }
+
Review Comment:
remove redundant empty line.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +86,170 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? new HoodieSimpleFileGroupRecordBuffer<>(readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(), recordMerger, props)
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
+ this.baseFileIterator = makeBaseFileIterator();
scanLogFiles();
recordBuffer.setBaseFileIterator(baseFileIterator);
}
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ if (!HoodiePayloadProps.isProjectionCompatible(hoodieTableConfig.getPayloadClass())) {
+ return dataSchema;
+ }
+
+ List<Schema.Field> addedFields = new ArrayList<>();
+ for (String field : recordMerger.getMandatoryFieldsForMerging(hoodieTableConfig)) {
+ //need to match HoodieFileGroupReaderBasedParquetFileFormat for now
+ //if (!findNestedField(requestedSchema, field).isPresent()) {
+ if (requestedSchema.getField(field) == null) {
+ Option<Schema.Field> foundFieldOpt = findNestedField(dataSchema, field);
+ if (!foundFieldOpt.isPresent()) {
+ throw new IllegalArgumentException("Field: " + field + " does not exist in the table schema");
+ }
+ Schema.Field foundField = foundFieldOpt.get();
+ addedFields.add(foundField);
+ }
+ }
+
+ if (addedFields.isEmpty()) {
+ return maybeReorderForBootstrap(requestedSchema);
+ }
+
+ //return maybeReorderForBootstrap(appendFieldsToSchemaDedupNested(requestedSchema, addedFields));
Review Comment:
here too
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +86,170 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? new HoodieSimpleFileGroupRecordBuffer<>(readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(), recordMerger, props)
Review Comment:
Let's avoid adding the new record buffer implementation and make exsiting record buffer work.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -122,6 +125,23 @@ default boolean shouldFlush(HoodieRecord record, Schema schema, TypedProperties
return true;
}
+ default String[] getMandatoryFieldsForMerging(HoodieTableConfig cfg) {
+ ArrayList<String> requiredFields = new ArrayList<>();
+ if (cfg.populateMetaFields()) {
+ requiredFields.add(HoodieRecord.RECORD_KEY_METADATA_FIELD);
+ } else {
+ cfg.getRecordKeyFieldStream().forEach(requiredFields::add);
+ }
+ String preCombine = cfg.getPreCombineField();
+
+ //maybe throw exception otherwise
Review Comment:
Could you remove the comment here? I think we need to fix the enforcement of preCombine in a different PR. The reader does not have to fail here.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieLogFile.java:
##########
@@ -147,6 +148,10 @@ public String getFileExtension() {
return fileExtension;
}
+ public boolean isCDC() {
Review Comment:
I think this is ok given this is part of the log format.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -540,6 +541,15 @@ public Option<String[]> getRecordKeyFields() {
}
}
+ public Stream<String> getRecordKeyFieldStream() {
Review Comment:
Let's reuse the existing method?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -67,8 +71,8 @@
*/
public final class HoodieFileGroupReader<T> implements Closeable {
private final HoodieReaderContext<T> readerContext;
- private final Option<HoodieBaseFile> baseFilePath;
- private final Option<List<String>> logFilePathList;
+ private final Option<HoodieBaseFile> hoodieBaseFileOption;
+ private final List<HoodieLogFile> logFiles;
Review Comment:
Should the file slice instance be stored directly instead of base and log files, given the file slice is passed in?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +86,170 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? new HoodieSimpleFileGroupRecordBuffer<>(readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(), recordMerger, props)
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
+ this.baseFileIterator = makeBaseFileIterator();
scanLogFiles();
recordBuffer.setBaseFileIterator(baseFileIterator);
}
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ if (!HoodiePayloadProps.isProjectionCompatible(hoodieTableConfig.getPayloadClass())) {
+ return dataSchema;
+ }
+
+ List<Schema.Field> addedFields = new ArrayList<>();
+ for (String field : recordMerger.getMandatoryFieldsForMerging(hoodieTableConfig)) {
+ //need to match HoodieFileGroupReaderBasedParquetFileFormat for now
+ //if (!findNestedField(requestedSchema, field).isPresent()) {
Review Comment:
remove unused code
##########
hudi-common/src/test/java/org/apache/hudi/avro/TestAvroSchemaUtils.java:
##########
@@ -229,4 +230,75 @@ public void testIsCompatibleProjectionAllowed(boolean shouldValidate) {
public void testIsCompatiblePartitionDropCols(boolean shouldValidate) {
AvroSchemaUtils.checkSchemaCompatible(FULL_SCHEMA, SHORT_SCHEMA, shouldValidate, false, Collections.singleton("c"));
}
+
+ /* behavior is slightly modified currently
+ @Test
+ public void testAppendFieldsToSchemaDedupNested() {
Review Comment:
JIRA to track the disabled test.
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -84,6 +86,19 @@ trait HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
object HoodieSpark3CatalystPlanUtils extends SparkAdapterSupport {
+ def applyNewFileFormatChanges(scanOperation: LogicalPlan, logicalRelation: LogicalPlan, fs: HadoopFsRelation): LogicalPlan = {
+ val ff = fs.fileFormat.asInstanceOf[ParquetFileFormat with HoodieFormatTrait]
+ ff.isProjected = true
+ val tableSchema = fs.location.asInstanceOf[SparkHoodieTableFileIndex].schema
+ val resolvedSchema = logicalRelation.resolve(tableSchema, fs.sparkSession.sessionState.analyzer.resolver)
+ if (!fs.partitionSchema.fields.isEmpty && scanOperation.sameOutput(logicalRelation)) {
+ Project(resolvedSchema, scanOperation)
+ } else {
+ scanOperation
+ }
+ }
+
+
Review Comment:
remote: redundant line
##########
hudi-spark-datasource/hudi-spark2/src/main/scala/org/apache/spark/sql/HoodieSpark2CatalystPlanUtils.scala:
##########
@@ -25,9 +25,9 @@ import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
import org.apache.spark.sql.catalyst.optimizer.SimplifyCasts
import org.apache.spark.sql.catalyst.planning.PhysicalOperation
import org.apache.spark.sql.catalyst.plans.JoinType
-import org.apache.spark.sql.catalyst.plans.logical.{InsertIntoTable, Join, LogicalPlan, MergeIntoTable, Project}
+import org.apache.spark.sql.catalyst.plans.logical._
Review Comment:
nit: revert
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +86,170 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
Review Comment:
Keep only one of two as they serve the same purpose.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +86,170 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? new HoodieSimpleFileGroupRecordBuffer<>(readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(), recordMerger, props)
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
+ this.baseFileIterator = makeBaseFileIterator();
scanLogFiles();
recordBuffer.setBaseFileIterator(baseFileIterator);
}
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ if (!HoodiePayloadProps.isProjectionCompatible(hoodieTableConfig.getPayloadClass())) {
Review Comment:
This should be based on the merger strategy instead of payload type.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +86,170 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? new HoodieSimpleFileGroupRecordBuffer<>(readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(), recordMerger, props)
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
+ this.baseFileIterator = makeBaseFileIterator();
scanLogFiles();
recordBuffer.setBaseFileIterator(baseFileIterator);
}
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ if (!HoodiePayloadProps.isProjectionCompatible(hoodieTableConfig.getPayloadClass())) {
+ return dataSchema;
+ }
+
+ List<Schema.Field> addedFields = new ArrayList<>();
+ for (String field : recordMerger.getMandatoryFieldsForMerging(hoodieTableConfig)) {
+ //need to match HoodieFileGroupReaderBasedParquetFileFormat for now
+ //if (!findNestedField(requestedSchema, field).isPresent()) {
+ if (requestedSchema.getField(field) == null) {
+ Option<Schema.Field> foundFieldOpt = findNestedField(dataSchema, field);
+ if (!foundFieldOpt.isPresent()) {
+ throw new IllegalArgumentException("Field: " + field + " does not exist in the table schema");
+ }
+ Schema.Field foundField = foundFieldOpt.get();
+ addedFields.add(foundField);
+ }
+ }
+
+ if (addedFields.isEmpty()) {
+ return maybeReorderForBootstrap(requestedSchema);
+ }
+
+ //return maybeReorderForBootstrap(appendFieldsToSchemaDedupNested(requestedSchema, addedFields));
+ return maybeReorderForBootstrap(appendFieldsToSchema(requestedSchema, addedFields));
+ }
+
+ private Schema maybeReorderForBootstrap(Schema input) {
+ if (this.hoodieBaseFileOption.isPresent() && this.hoodieBaseFileOption.get().getBootstrapBaseFile().isPresent()) {
+ Pair<List<Schema.Field>,List<Schema.Field>> requiredFields = getDataAndMetaCols(input);
Review Comment:
Some code are not properly formatted. Do you have checkstyle enabled in IDE?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,166 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
Review Comment:
This is the read schema which can be different from the write schema.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,166 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
Review Comment:
@jonvex could you add docs in the code on these schemas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1824930533
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* f58d9cb5a301e1aff325afc9f0ef3f618c9fe655 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985)
* 0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828803008
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* c22d1db44cd389160fe06a6f20233fb0e22df322 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162)
* 91be81a12c0321d089c89ceeb7ed0ec8d18079ea Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408240703
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,167 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? new HoodieSimpleFileGroupRecordBuffer<>(readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(), recordMerger, props)
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
+ this.baseFileIterator = makeBaseFileIterator();
scanLogFiles();
recordBuffer.setBaseFileIterator(baseFileIterator);
}
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ //MergeOnReadSnapshotRelation.isProjectionCompatible. Should centralize the logic
Review Comment:
Completed the todo, but I think what we do with payloads seems like there is room for improvement. If you have better ideas lmk
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408353113
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -77,14 +78,18 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: Option[Partitioned
}
}).asInstanceOf[ClosableIterator[InternalRow]]
} else {
- if (baseFileReader.isEmpty) {
- throw new IllegalArgumentException("Base file reader is missing when instantiating "
- + "SparkFileFormatInternalRowReaderContext.");
+ val key = generateKey(dataSchema, requiredSchema)
+ if (!readerMaps.contains(key)) {
+ throw new IllegalStateException("schemas don't hash to a known reader")
}
- new CloseableInternalRowIterator(baseFileReader.get.apply(fileInfo))
+ new CloseableInternalRowIterator(readerMaps(key).apply(fileInfo))
}
}
+ private def generateKey(dataSchema: Schema, requestedSchema: Schema): Long = {
Review Comment:
The only time it would matter is if there are pushdown filters. This is just a temporary hacky solution for now. We will eventually build the reader inside of getFileRecordIterator. We need to incorporate an abstraction for filters when we do that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1830816005
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* b144ee0b098dcb17972152e07cd57412ad498d47 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1833149064
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 993d0dc63f3f552b8bf5b52c113f3ae8ef53304c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1831132621
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* b144ee0b098dcb17972152e07cd57412ad498d47 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186)
* b0b711e0c355320da652fa7f2d8669539873d4d6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1411097577
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,166 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
Review Comment:
Schemas with different meanings are floating in Hudi so far. We should try to make it consistent. Please take a note, and refactor it after landing this PR. (We may also do similar things for Flink/Presto/Trino/etc later).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1411506109
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -123,4 +125,11 @@ private Object getFieldValueFromInternalRow(InternalRow row, Schema recordSchema
return null;
}
}
+
+ @Override
+ public UnaryOperator<InternalRow> projectRecord(Schema from, Schema to) {
+ UnsafeProjection projection = HoodieInternalRowUtils.getCachedUnsafeProjection(AvroConversionUtils.convertAvroSchemaToStructType(from),
Review Comment:
HoodieCatalystExpressionUtils.generateUnsafeProjection is what I wanted to use, but the file contains both an object and a trait with the name HoodieCatalystExpressionUtils. When using scala2.11 and calling with java, it fails building with a no such method exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1845979329
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
}, {
"hash" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347",
"triggerID" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"triggerType" : "PUSH"
}, {
"hash" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350",
"triggerID" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"triggerType" : "PUSH"
}, {
"hash" : "830f88ca3021165497428d27e5e5fd19e5f3f41d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21373",
"triggerID" : "830f88ca3021165497428d27e5e5fd19e5f3f41d",
"triggerType" : "PUSH"
}, {
"hash" : "d6d0848e183d0a192fda67f647f81adcbf6f5922",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21375",
"triggerID" : "d6d0848e183d0a192fda67f647f81adcbf6f5922",
"triggerType" : "PUSH"
}, {
"hash" : "150a4b707903dd3405a5ad8235bdafc9a2b3fa9f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21376",
"triggerID" : "150a4b707903dd3405a5ad8235bdafc9a2b3fa9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* d6d0848e183d0a192fda67f647f81adcbf6f5922 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21375)
* 150a4b707903dd3405a5ad8235bdafc9a2b3fa9f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21376)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1832672177
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* bfc0a855cadb4f6329bd38a470ade931797c53ab Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223)
* dfa3bdee07f850efbacb55ecc84637339a953423 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828810016
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* c22d1db44cd389160fe06a6f20233fb0e22df322 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162)
* 91be81a12c0321d089c89ceeb7ed0ec8d18079ea Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164)
* a045da6d6c31c09a410449788002b75e8c335548 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1409754265
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,166 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
Review Comment:
does this align with the "requiredSchema" in ParquetFileFormat? Since I did not see "requestedSchema" before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1832880118
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* dfa3bdee07f850efbacb55ecc84637339a953423 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225)
* 747894399d3bef0e05c561d9c67db61ab2536cf9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228)
* 31fe075c72fb189b9155e48ab3399e9199cc293a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408285783
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedParquetFileFormat.scala:
##########
@@ -91,73 +95,69 @@ class HoodieFileGroupReaderBasedParquetFileFormat(tableState: HoodieTableState,
val outputSchema = StructType(requiredSchema.fields ++ partitionSchema.fields)
spark.conf.set("spark.sql.parquet.enableVectorizedReader", supportBatchResult)
val requiredSchemaWithMandatory = generateRequiredSchemaWithMandatory(requiredSchema, dataSchema, partitionSchema)
+ val isCount = requiredSchemaWithMandatory.isEmpty
val requiredSchemaSplits = requiredSchemaWithMandatory.fields.partition(f => HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(f.name))
val requiredMeta = StructType(requiredSchemaSplits._1)
val requiredWithoutMeta = StructType(requiredSchemaSplits._2)
val augmentedHadoopConf = FSUtils.buildInlineConf(hadoopConf)
- val (baseFileReader, preMergeBaseFileReader, _, _) = buildFileReaders(
+ val (baseFileReader, preMergeBaseFileReader, readerMaps) = buildFileReaders(
spark, dataSchema, partitionSchema, if (isIncremental) requiredSchemaWithMandatory else requiredSchema,
filters, options, augmentedHadoopConf, requiredSchemaWithMandatory, requiredWithoutMeta, requiredMeta)
+
+ val requestedAvroSchema = AvroConversionUtils.convertStructTypeToAvroSchema(requiredSchema, sanitizedTableName)
+ val dataAvroSchema = AvroConversionUtils.convertStructTypeToAvroSchema(dataSchema, sanitizedTableName)
+
val broadcastedHadoopConf = spark.sparkContext.broadcast(new SerializableConfiguration(augmentedHadoopConf))
+ val broadcastedDataSchema = spark.sparkContext.broadcast(dataAvroSchema)
+ val broadcastedRequestedSchema = spark.sparkContext.broadcast(requestedAvroSchema)
val props: TypedProperties = HoodieFileIndex.getConfigProperties(spark, options)
(file: PartitionedFile) => {
file.partitionValues match {
// Snapshot or incremental queries.
case fileSliceMapping: HoodiePartitionFileSliceMapping =>
val filePath = sparkAdapter.getSparkPartitionedFileUtils.getPathFromPartitionedFile(file)
- if (FSUtils.isLogFile(filePath)) {
- val partitionValues = fileSliceMapping.getPartitionValues
- val fileSlice = fileSliceMapping.getSlice(FSUtils.getFileId(filePath.getName).substring(1)).get
- buildFileGroupIterator(
- Option.empty[PartitionedFile => Iterator[InternalRow]],
- partitionValues,
- Option.empty[HoodieBaseFile],
- getLogFilesFromSlice(fileSlice),
- requiredSchemaWithMandatory,
- outputSchema,
- partitionSchema,
- broadcastedHadoopConf.value.value,
- -1,
- -1,
- shouldUseRecordPosition
- )
+ val filegroupName = if (FSUtils.isLogFile(filePath)) {
+ FSUtils.getFileId(filePath.getName).substring(1)
} else {
- fileSliceMapping.getSlice(FSUtils.getFileId(filePath.getName)) match {
- case Some(fileSlice) =>
+ FSUtils.getFileId(filePath.getName)
+ }
+ fileSliceMapping.getSlice(filegroupName) match {
+ case Some(fileSlice) if !isCount =>
+ if (requiredSchema.isEmpty && !fileSlice.getLogFiles.findAny().isPresent) {
val hoodieBaseFile = fileSlice.getBaseFile.get()
- val bootstrapFileOpt = hoodieBaseFile.getBootstrapBaseFile
- val partitionValues = fileSliceMapping.getPartitionValues
- val logFiles = getLogFilesFromSlice(fileSlice)
- if (requiredSchemaWithMandatory.isEmpty) {
- val baseFile = createPartitionedFile(partitionValues, hoodieBaseFile.getHadoopPath, 0, hoodieBaseFile.getFileLen)
- baseFileReader(baseFile)
- } else if (bootstrapFileOpt.isPresent) {
- // TODO: Use FileGroupReader here: HUDI-6942.
- throw new NotImplementedError("Not support reading bootstrap file")
- } else {
- if (logFiles.isEmpty) {
- throw new IllegalStateException(
- "should not be here since file slice should not have been broadcasted "
- + "since it has no log or data files")
- }
- buildFileGroupIterator(
- Option(preMergeBaseFileReader),
- partitionValues,
- Option(hoodieBaseFile),
- logFiles,
- requiredSchemaWithMandatory,
- outputSchema,
- partitionSchema,
- broadcastedHadoopConf.value.value,
- 0,
- hoodieBaseFile.getFileLen,
- shouldUseRecordPosition
- )
- }
- // TODO: Use FileGroupReader here: HUDI-6942.
- case _ => baseFileReader(file)
- }
+ baseFileReader(createPartitionedFile(fileSliceMapping.getPartitionValues, hoodieBaseFile.getHadoopPath, 0, hoodieBaseFile.getFileLen))
+ } else {
+ val readerContext: HoodieReaderContext[InternalRow] = new SparkFileFormatInternalRowReaderContext(
+ readerMaps)
+ val serializedHadoopConf = broadcastedHadoopConf.value.value
+ val metaClient: HoodieTableMetaClient = HoodieTableMetaClient
+ .builder().setConf(serializedHadoopConf).setBasePath(tableState.tablePath).build
+ val reader = new HoodieFileGroupReader[InternalRow](
+ readerContext,
+ serializedHadoopConf,
+ tableState.tablePath,
+ tableState.latestCommitTimestamp.get,
Review Comment:
good catch, I took that from what was there before and didn't notice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408299074
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/common/model/HoodieSparkRecord.java:
##########
@@ -449,7 +449,8 @@ private static void validateRow(InternalRow data, StructType schema) {
data instanceof HoodieInternalRow
|| data instanceof GenericInternalRow
|| data instanceof SpecificInternalRow
- || SparkAdapterSupport$.MODULE$.sparkAdapter().isColumnarBatchRow(data));
+ || SparkAdapterSupport$.MODULE$.sparkAdapter().isColumnarBatchRow(data))
+ || data instanceof JoinedRow;
Review Comment:
@codope, I also have a question about this: Does Hudi always assume the partition values are needed, even for non-partitioned tables?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1417819667
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +86,170 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
Review Comment:
discussed offline. Still need both for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1419353747
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +86,170 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
Review Comment:
@jonvex let's create a JIRA ticket to track this.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,119 +84,213 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+
+ // requestedSchema: the schema that the caller requests
+ private final Schema requestedSchema;
+
+ // requiredSchema: the requestedSchema with any additional columns required for merging etc
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
- this.recordMerger = readerContext.getRecordMerger(
- getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
+ this.recordMerger = readerContext.getRecordMerger(tableConfig.getRecordMergerStrategy());
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? null
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
- scanLogFiles();
- recordBuffer.setBaseFileIterator(baseFileIterator);
+ ClosableIterator<T> iter = makeBaseFileIterator();
+ if (logFiles.isEmpty()) {
+ this.baseFileIterator = CachingIterator.wrap(iter, readerContext);
+ } else {
+ this.baseFileIterator = iter;
+ scanLogFiles();
+ recordBuffer.setBaseFileIterator(baseFileIterator);
+ }
+ }
+
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ List<Schema.Field> addedFields = new ArrayList<>();
+ for (String field : recordMerger.getMandatoryFieldsForMerging(hoodieTableConfig)) {
+ if (requestedSchema.getField(field) == null) {
+ Option<Schema.Field> foundFieldOpt = findNestedField(dataSchema, field);
+ if (!foundFieldOpt.isPresent()) {
+ throw new IllegalArgumentException("Field: " + field + " does not exist in the table schema");
+ }
+ Schema.Field foundField = foundFieldOpt.get();
+ addedFields.add(foundField);
+ }
+ }
+
+ if (addedFields.isEmpty()) {
+ return maybeReorderForBootstrap(requestedSchema);
+ }
+
+ return maybeReorderForBootstrap(appendFieldsToSchema(requestedSchema, addedFields));
+ }
+
+ private Schema maybeReorderForBootstrap(Schema input) {
+ if (this.hoodieBaseFileOption.isPresent() && this.hoodieBaseFileOption.get().getBootstrapBaseFile().isPresent()) {
+ Pair<List<Schema.Field>,List<Schema.Field>> requiredFields = getDataAndMetaCols(input);
+ if (!(requiredFields.getLeft().isEmpty() || requiredFields.getRight().isEmpty())) {
+ return createSchemaFromFields(Stream.concat(requiredFields.getLeft().stream(), requiredFields.getRight().stream())
+ .collect(Collectors.toList()));
+ }
+ }
+ return input;
+ }
+
+ private static Pair<List<Schema.Field>,List<Schema.Field>> getDataAndMetaCols(Schema schema) {
+ Map<Boolean, List<Schema.Field>> fieldsByMeta = schema.getFields().stream()
+ .collect(Collectors.partitioningBy(f -> HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(f.name())));
+ return Pair.of(fieldsByMeta.getOrDefault(true, Collections.emptyList()),
+ fieldsByMeta.getOrDefault(false, Collections.emptyList()));
+ }
+
+ private Schema createSchemaFromFields(List<Schema.Field> fields) {
+ //fields have positions set, so we need to remove them due to avro setFields implementation
+ for (int i = 0; i < fields.size(); i++) {
+ Schema.Field curr = fields.get(i);
+ fields.set(i, new Schema.Field(curr.name(), curr.schema(), curr.doc(), curr.defaultVal()));
+ }
+ Schema newSchema = Schema.createRecord(dataSchema.getName(), dataSchema.getDoc(), dataSchema.getNamespace(), dataSchema.isError());
+ newSchema.setFields(fields);
+ return newSchema;
+ }
+
+ private ClosableIterator<T> makeBootstrapBaseFileIterator(HoodieBaseFile baseFile) {
+ BaseFile dataFile = baseFile.getBootstrapBaseFile().get();
+ Pair<List<Schema.Field>,List<Schema.Field>> requiredFields = getDataAndMetaCols(requiredSchema);
+ Pair<List<Schema.Field>,List<Schema.Field>> allFields = getDataAndMetaCols(dataSchema);
+
+ Option<ClosableIterator<T>> dataFileIterator = requiredFields.getRight().isEmpty() ? Option.empty() :
+ Option.of(readerContext.getFileRecordIterator(dataFile.getHadoopPath(), 0, dataFile.getFileLen(),
+ createSchemaFromFields(allFields.getRight()), createSchemaFromFields(requiredFields.getRight()), hadoopConf));
+
+ Option<ClosableIterator<T>> skeletonFileIterator = requiredFields.getLeft().isEmpty() ? Option.empty() :
+ Option.of(readerContext.getFileRecordIterator(baseFile.getHadoopPath(), 0, baseFile.getFileLen(),
+ createSchemaFromFields(allFields.getLeft()), createSchemaFromFields(requiredFields.getLeft()), hadoopConf));
+ if (!dataFileIterator.isPresent() && !skeletonFileIterator.isPresent()) {
+ throw new IllegalStateException("should not be here if only partition cols are required");
+ } else if (!dataFileIterator.isPresent()) {
+ return skeletonFileIterator.get();
+ } else if (!skeletonFileIterator.isPresent()) {
+ return dataFileIterator.get();
+ } else {
+ return readerContext.mergeBootstrapReaders(skeletonFileIterator.get(), dataFileIterator.get());
+ }
}
/**
* @return {@code true} if the next record exists; {@code false} otherwise.
* @throws IOException on reader error.
*/
public boolean hasNext() throws IOException {
- return recordBuffer.hasNext();
+ if (recordBuffer == null) {
+ return baseFileIterator.hasNext();
Review Comment:
@jonvex let's create a JIRA ticket to track this too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1845535768
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
}, {
"hash" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347",
"triggerID" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"triggerType" : "PUSH"
}, {
"hash" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350",
"triggerID" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"triggerType" : "PUSH"
}, {
"hash" : "830f88ca3021165497428d27e5e5fd19e5f3f41d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "830f88ca3021165497428d27e5e5fd19e5f3f41d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 361e800774f0d6789a550fcc63c8764416ff54cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350)
* 830f88ca3021165497428d27e5e5fd19e5f3f41d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408365392
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,166 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
Review Comment:
@yihua I wonder why do we need log record schema? Should it be stored in the log files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828749924
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 38b26030e7df02d865cfe4efd03a955124d2272a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160)
* c22d1db44cd389160fe06a6f20233fb0e22df322 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162)
* 91be81a12c0321d089c89ceeb7ed0ec8d18079ea UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1830634415
@jonvex , this PR is really hard for me to review since it combines so many different topics. I will focus on checking what I can understand, and rely on other folks to do the heavy lifting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1845550454
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
}, {
"hash" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347",
"triggerID" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"triggerType" : "PUSH"
}, {
"hash" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350",
"triggerID" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"triggerType" : "PUSH"
}, {
"hash" : "830f88ca3021165497428d27e5e5fd19e5f3f41d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21373",
"triggerID" : "830f88ca3021165497428d27e5e5fd19e5f3f41d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 361e800774f0d6789a550fcc63c8764416ff54cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350)
* 830f88ca3021165497428d27e5e5fd19e5f3f41d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21373)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1419429353
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +86,170 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
Review Comment:
https://issues.apache.org/jira/browse/HUDI-7192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1843969090
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
}, {
"hash" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347",
"triggerID" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"triggerType" : "PUSH"
}, {
"hash" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350",
"triggerID" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 8048ed72bc9a21e9d178445fdc39e1a379f183e0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347)
* 361e800774f0d6789a550fcc63c8764416ff54cc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1844163184
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
}, {
"hash" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347",
"triggerID" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"triggerType" : "PUSH"
}, {
"hash" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350",
"triggerID" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 361e800774f0d6789a550fcc63c8764416ff54cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828973086
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* a045da6d6c31c09a410449788002b75e8c335548 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165)
* 3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1843838045
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
}, {
"hash" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347",
"triggerID" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 993d0dc63f3f552b8bf5b52c113f3ae8ef53304c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241)
* 47e6224db0b980c0b3697b282fa724a67b2cca31 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345)
* c5f2e8da635bb7235d8b7719d7904df6c7489e0b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346)
* 8048ed72bc9a21e9d178445fdc39e1a379f183e0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1412419931
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -67,8 +71,8 @@
*/
public final class HoodieFileGroupReader<T> implements Closeable {
private final HoodieReaderContext<T> readerContext;
- private final Option<HoodieBaseFile> baseFilePath;
- private final Option<List<String>> logFilePathList;
+ private final Option<HoodieBaseFile> hoodieBaseFileOption;
+ private final List<HoodieLogFile> logFiles;
Review Comment:
I think it's cleaner this way, we access it several times
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828432894
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 2a9a36323053c5d35ae68cc5b2c38bab707b0f1a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828759007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 38b26030e7df02d865cfe4efd03a955124d2272a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160)
* c22d1db44cd389160fe06a6f20233fb0e22df322 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162)
* 91be81a12c0321d089c89ceeb7ed0ec8d18079ea Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1409955152
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -77,14 +78,18 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: Option[Partitioned
}
}).asInstanceOf[ClosableIterator[InternalRow]]
} else {
- if (baseFileReader.isEmpty) {
- throw new IllegalArgumentException("Base file reader is missing when instantiating "
- + "SparkFileFormatInternalRowReaderContext.");
+ val key = generateKey(dataSchema, requiredSchema)
+ if (!readerMaps.contains(key)) {
+ throw new IllegalStateException("schemas don't hash to a known reader")
}
- new CloseableInternalRowIterator(baseFileReader.get.apply(fileInfo))
+ new CloseableInternalRowIterator(readerMaps(key).apply(fileInfo))
}
}
+ private def generateKey(dataSchema: Schema, requestedSchema: Schema): Long = {
Review Comment:
Hi Jon, i really feel that if you can split this PR into smaller PRs, that would be much easier for reviewers to understand and easier for the CI to pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1833023754
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 31fe075c72fb189b9155e48ab3399e9199cc293a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231)
* 993d0dc63f3f552b8bf5b52c113f3ae8ef53304c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1834256493
azure failed due to network issue. Reran and it passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828978858
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* a045da6d6c31c09a410449788002b75e8c335548 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165)
* 3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1832765012
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* dfa3bdee07f850efbacb55ecc84637339a953423 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828871007
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 91be81a12c0321d089c89ceeb7ed0ec8d18079ea Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164)
* a045da6d6c31c09a410449788002b75e8c335548 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1411398279
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaUtils.java:
##########
@@ -218,17 +220,102 @@ private static boolean isProjectionOfInternal(Schema sourceSchema,
return atomicTypeEqualityPredicate.apply(sourceSchema, targetSchema);
}
+ public static Option<Schema.Field> findNestedField(Schema schema, String fieldName) {
+ return findNestedField(schema, fieldName.split("\\."), 0);
+ }
+
+ private static Option<Schema.Field> findNestedField(Schema schema, String[] fieldParts, int index) {
+ if (schema.getType().equals(Schema.Type.UNION)) {
+ Option<Schema.Field> notUnion = findNestedField(resolveNullableSchema(schema), fieldParts, index);
+ if (!notUnion.isPresent()) {
+ return Option.empty();
+ }
+ Schema.Field nu = notUnion.get();
+ return Option.of(new Schema.Field(nu.name(), nu.schema(), nu.doc(), nu.defaultVal()));
+ }
+ if (fieldParts.length <= index) {
+ return Option.empty();
+ }
+
+ Schema.Field foundField = schema.getField(fieldParts[index]);
+ if (foundField == null) {
+ return Option.empty();
+ }
+
+ if (index == fieldParts.length - 1) {
+ return Option.of(new Schema.Field(foundField.name(), foundField.schema(), foundField.doc(), foundField.defaultVal()));
+ }
+
+ Schema foundSchema = foundField.schema();
+ Option<Schema.Field> nestedPart = findNestedField(foundSchema, fieldParts, index + 1);
+ if (!nestedPart.isPresent()) {
+ return Option.empty();
+ }
+ //temporary, need to match HoodieFileGroupReaderBasedParquetFileFormat for now
Review Comment:
Jon, you used a different way to add comments from others.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1419013979
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -123,4 +125,11 @@ private Object getFieldValueFromInternalRow(InternalRow row, Schema recordSchema
return null;
}
}
+
+ @Override
+ public UnaryOperator<InternalRow> projectRecord(Schema from, Schema to) {
+ UnsafeProjection projection = HoodieInternalRowUtils.getCachedUnsafeProjection(AvroConversionUtils.convertAvroSchemaToStructType(from),
Review Comment:
I was referring to `AvroConversionUtils.convertAvroSchemaToStructType` which can be replaced by `HoodieInternalRowUtils.getCachedSchema(schema)` to avoid repeated parsing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828656724
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 38b26030e7df02d865cfe4efd03a955124d2272a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828593909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 2a9a36323053c5d35ae68cc5b2c38bab707b0f1a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159)
* 38b26030e7df02d865cfe4efd03a955124d2272a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1829022999
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* a045da6d6c31c09a410449788002b75e8c335548 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165)
* 3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166)
* 5450affcda38adf0edab0fd77c6b4214873261d3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408244066
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieSimpleFileGroupRecordBuffer.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.log.KeySpec;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+
+import java.io.IOException;
+import java.util.Map;
+
+public class HoodieSimpleFileGroupRecordBuffer<T> extends HoodieBaseFileGroupRecordBuffer<T> {
+ public HoodieSimpleFileGroupRecordBuffer(HoodieReaderContext<T> readerContext, Schema readerSchema, Schema baseFileSchema,
+ Option<String> partitionNameOverrideOpt, Option<String[]> partitionPathFieldOpt, HoodieRecordMerger recordMerger,
+ TypedProperties payloadProps) {
+ super(readerContext, readerSchema, baseFileSchema, partitionNameOverrideOpt, partitionPathFieldOpt, recordMerger, payloadProps);
+ }
+
+ @Override
+ protected boolean doHasNext() throws IOException {
+ if (baseFileIterator.hasNext()) {
+ nextRecord = readerContext.seal(baseFileIterator.next());
+ return true;
+ }
+ return false;
+ }
+
+ @Override
+ public BufferType getBufferType() {
+ return BufferType.SIMPLE;
+ }
+
+ @Override
+ public void processDataBlock(HoodieDataBlock dataBlock, Option<KeySpec> keySpecOpt) throws IOException {
+ throw new IllegalStateException("processDataBlock should never be called from HoodieSimpleFileGroupRecordBuffer");
Review Comment:
HoodieSimpleFileGroupRecordBuffer is used when there are no log files, so none of these methods should be called
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1832836486
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* dfa3bdee07f850efbacb55ecc84637339a953423 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225)
* 747894399d3bef0e05c561d9c67db61ab2536cf9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1411117772
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -123,4 +125,11 @@ private Object getFieldValueFromInternalRow(InternalRow row, Schema recordSchema
return null;
}
}
+
+ @Override
+ public UnaryOperator<InternalRow> projectRecord(Schema from, Schema to) {
+ UnsafeProjection projection = HoodieInternalRowUtils.getCachedUnsafeProjection(AvroConversionUtils.convertAvroSchemaToStructType(from),
Review Comment:
Use `HoodieInternalRowUtils.getCachedSchema(schema)` to avoid parsing Avro schema per record?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/common/model/HoodieSparkRecord.java:
##########
@@ -449,7 +449,8 @@ private static void validateRow(InternalRow data, StructType schema) {
data instanceof HoodieInternalRow
|| data instanceof GenericInternalRow
|| data instanceof SpecificInternalRow
- || SparkAdapterSupport$.MODULE$.sparkAdapter().isColumnarBatchRow(data));
+ || SparkAdapterSupport$.MODULE$.sparkAdapter().isColumnarBatchRow(data))
+ || data instanceof JoinedRow;
Review Comment:
Partition value is empty for non-partitioned tables.
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -77,14 +78,18 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: Option[Partitioned
}
}).asInstanceOf[ClosableIterator[InternalRow]]
} else {
- if (baseFileReader.isEmpty) {
- throw new IllegalArgumentException("Base file reader is missing when instantiating "
- + "SparkFileFormatInternalRowReaderContext.");
+ val key = generateKey(dataSchema, requiredSchema)
Review Comment:
nit: rename to `schemaPairHashKey`
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -49,8 +51,7 @@ import scala.collection.mutable
* not required for reading a file group with only log files.
* @param partitionValues The values for a partition in which the file group lives.
*/
-class SparkFileFormatInternalRowReaderContext(baseFileReader: Option[PartitionedFile => Iterator[InternalRow]],
- partitionValues: InternalRow) extends BaseSparkInternalRowReaderContext {
+class SparkFileFormatInternalRowReaderContext(readerMaps: mutable.Map[Long, PartitionedFile => Iterator[InternalRow]]) extends BaseSparkInternalRowReaderContext {
Review Comment:
Add docs on `readerMaps`
##########
hudi-common/src/main/java/org/apache/hudi/avro/AvroSchemaUtils.java:
##########
@@ -218,17 +220,102 @@ private static boolean isProjectionOfInternal(Schema sourceSchema,
return atomicTypeEqualityPredicate.apply(sourceSchema, targetSchema);
}
+ public static Option<Schema.Field> findNestedField(Schema schema, String fieldName) {
+ return findNestedField(schema, fieldName.split("\\."), 0);
+ }
+
+ private static Option<Schema.Field> findNestedField(Schema schema, String[] fieldParts, int index) {
+ if (schema.getType().equals(Schema.Type.UNION)) {
+ Option<Schema.Field> notUnion = findNestedField(resolveNullableSchema(schema), fieldParts, index);
+ if (!notUnion.isPresent()) {
+ return Option.empty();
+ }
+ Schema.Field nu = notUnion.get();
+ return Option.of(new Schema.Field(nu.name(), nu.schema(), nu.doc(), nu.defaultVal()));
+ }
+ if (fieldParts.length <= index) {
+ return Option.empty();
+ }
+
+ Schema.Field foundField = schema.getField(fieldParts[index]);
+ if (foundField == null) {
+ return Option.empty();
+ }
+
+ if (index == fieldParts.length - 1) {
+ return Option.of(new Schema.Field(foundField.name(), foundField.schema(), foundField.doc(), foundField.defaultVal()));
+ }
+
+ Schema foundSchema = foundField.schema();
+ Option<Schema.Field> nestedPart = findNestedField(foundSchema, fieldParts, index + 1);
+ if (!nestedPart.isPresent()) {
+ return Option.empty();
+ }
+ //temporary, need to match HoodieFileGroupReaderBasedParquetFileFormat for now
+ return nestedPart;
+ /*
+ boolean isUnion = false;
+ if (foundSchema.getType().equals(Schema.Type.UNION)) {
+ isUnion = true;
+ foundSchema = resolveNullableSchema(foundSchema);
+ }
+
+ Schema newSchema = Schema.createRecord(foundSchema.getName(), foundSchema.getDoc(), foundSchema.getNamespace(), false, Collections.singletonList(nestedPart.get()));
+ return Option.of(new Schema.Field(foundField.name(), isUnion ? createNullableSchema(newSchema) : newSchema, foundField.doc(), foundField.defaultVal()));
+ */
Review Comment:
Remove unused code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1411053061
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,166 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
Review Comment:
It actually really doesn't. Requested schema is what is requiredSchema in ParquetFileFormat
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1832681988
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* bfc0a855cadb4f6329bd38a470ade931797c53ab Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223)
* dfa3bdee07f850efbacb55ecc84637339a953423 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1831127365
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* b144ee0b098dcb17972152e07cd57412ad498d47 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186)
* b0b711e0c355320da652fa7f2d8669539873d4d6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1833017794
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 31fe075c72fb189b9155e48ab3399e9199cc293a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1419433198
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,119 +84,213 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+
+ // requestedSchema: the schema that the caller requests
+ private final Schema requestedSchema;
+
+ // requiredSchema: the requestedSchema with any additional columns required for merging etc
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
- this.recordMerger = readerContext.getRecordMerger(
- getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
+ this.recordMerger = readerContext.getRecordMerger(tableConfig.getRecordMergerStrategy());
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? null
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
- scanLogFiles();
- recordBuffer.setBaseFileIterator(baseFileIterator);
+ ClosableIterator<T> iter = makeBaseFileIterator();
+ if (logFiles.isEmpty()) {
+ this.baseFileIterator = CachingIterator.wrap(iter, readerContext);
+ } else {
+ this.baseFileIterator = iter;
+ scanLogFiles();
+ recordBuffer.setBaseFileIterator(baseFileIterator);
+ }
+ }
+
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ List<Schema.Field> addedFields = new ArrayList<>();
+ for (String field : recordMerger.getMandatoryFieldsForMerging(hoodieTableConfig)) {
+ if (requestedSchema.getField(field) == null) {
+ Option<Schema.Field> foundFieldOpt = findNestedField(dataSchema, field);
+ if (!foundFieldOpt.isPresent()) {
+ throw new IllegalArgumentException("Field: " + field + " does not exist in the table schema");
+ }
+ Schema.Field foundField = foundFieldOpt.get();
+ addedFields.add(foundField);
+ }
+ }
+
+ if (addedFields.isEmpty()) {
+ return maybeReorderForBootstrap(requestedSchema);
+ }
+
+ return maybeReorderForBootstrap(appendFieldsToSchema(requestedSchema, addedFields));
+ }
+
+ private Schema maybeReorderForBootstrap(Schema input) {
+ if (this.hoodieBaseFileOption.isPresent() && this.hoodieBaseFileOption.get().getBootstrapBaseFile().isPresent()) {
+ Pair<List<Schema.Field>,List<Schema.Field>> requiredFields = getDataAndMetaCols(input);
+ if (!(requiredFields.getLeft().isEmpty() || requiredFields.getRight().isEmpty())) {
+ return createSchemaFromFields(Stream.concat(requiredFields.getLeft().stream(), requiredFields.getRight().stream())
+ .collect(Collectors.toList()));
+ }
+ }
+ return input;
+ }
+
+ private static Pair<List<Schema.Field>,List<Schema.Field>> getDataAndMetaCols(Schema schema) {
+ Map<Boolean, List<Schema.Field>> fieldsByMeta = schema.getFields().stream()
+ .collect(Collectors.partitioningBy(f -> HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(f.name())));
+ return Pair.of(fieldsByMeta.getOrDefault(true, Collections.emptyList()),
+ fieldsByMeta.getOrDefault(false, Collections.emptyList()));
+ }
+
+ private Schema createSchemaFromFields(List<Schema.Field> fields) {
+ //fields have positions set, so we need to remove them due to avro setFields implementation
+ for (int i = 0; i < fields.size(); i++) {
+ Schema.Field curr = fields.get(i);
+ fields.set(i, new Schema.Field(curr.name(), curr.schema(), curr.doc(), curr.defaultVal()));
+ }
+ Schema newSchema = Schema.createRecord(dataSchema.getName(), dataSchema.getDoc(), dataSchema.getNamespace(), dataSchema.isError());
+ newSchema.setFields(fields);
+ return newSchema;
+ }
+
+ private ClosableIterator<T> makeBootstrapBaseFileIterator(HoodieBaseFile baseFile) {
+ BaseFile dataFile = baseFile.getBootstrapBaseFile().get();
+ Pair<List<Schema.Field>,List<Schema.Field>> requiredFields = getDataAndMetaCols(requiredSchema);
+ Pair<List<Schema.Field>,List<Schema.Field>> allFields = getDataAndMetaCols(dataSchema);
+
+ Option<ClosableIterator<T>> dataFileIterator = requiredFields.getRight().isEmpty() ? Option.empty() :
+ Option.of(readerContext.getFileRecordIterator(dataFile.getHadoopPath(), 0, dataFile.getFileLen(),
+ createSchemaFromFields(allFields.getRight()), createSchemaFromFields(requiredFields.getRight()), hadoopConf));
+
+ Option<ClosableIterator<T>> skeletonFileIterator = requiredFields.getLeft().isEmpty() ? Option.empty() :
+ Option.of(readerContext.getFileRecordIterator(baseFile.getHadoopPath(), 0, baseFile.getFileLen(),
+ createSchemaFromFields(allFields.getLeft()), createSchemaFromFields(requiredFields.getLeft()), hadoopConf));
+ if (!dataFileIterator.isPresent() && !skeletonFileIterator.isPresent()) {
+ throw new IllegalStateException("should not be here if only partition cols are required");
+ } else if (!dataFileIterator.isPresent()) {
+ return skeletonFileIterator.get();
+ } else if (!skeletonFileIterator.isPresent()) {
+ return dataFileIterator.get();
+ } else {
+ return readerContext.mergeBootstrapReaders(skeletonFileIterator.get(), dataFileIterator.get());
+ }
}
/**
* @return {@code true} if the next record exists; {@code false} otherwise.
* @throws IOException on reader error.
*/
public boolean hasNext() throws IOException {
- return recordBuffer.hasNext();
+ if (recordBuffer == null) {
+ return baseFileIterator.hasNext();
Review Comment:
https://issues.apache.org/jira/browse/HUDI-7193
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1843707123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 993d0dc63f3f552b8bf5b52c113f3ae8ef53304c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241)
* 47e6224db0b980c0b3697b282fa724a67b2cca31 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #10137:
URL: https://github.com/apache/hudi/pull/10137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] non-working. Want to review with team that this makes sense [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1817180840
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 184858b13509316eeb7fe0a1198b7df98c3fc50d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408167299
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -540,6 +541,15 @@ public Option<String[]> getRecordKeyFields() {
}
}
+ public Stream<String> getRecordKeyFieldStream() {
Review Comment:
If we did that we would go stream -> array -> stream. I think that will only be called once per file read, so the perf hit is probably negligible. But it bothers me lol
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1829027734
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* a045da6d6c31c09a410449788002b75e8c335548 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165)
* 3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166)
* 5450affcda38adf0edab0fd77c6b4214873261d3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408154172
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieLogFile.java:
##########
@@ -147,6 +148,10 @@ public String getFileExtension() {
return fileExtension;
}
+ public boolean isCDC() {
Review Comment:
idk. I like it here because I think future hudi contributors will be more likely to find it instead of creating duplicate code. Why do you think it should be in FSUtils?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1830620283
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 5450affcda38adf0edab0fd77c6b4214873261d3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169)
* b144ee0b098dcb17972152e07cd57412ad498d47 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1830581999
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 5450affcda38adf0edab0fd77c6b4214873261d3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169)
* b144ee0b098dcb17972152e07cd57412ad498d47 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408365392
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,166 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
Review Comment:
I wonder why do we need log record schema? Should it be stored in the log files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1832827213
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* dfa3bdee07f850efbacb55ecc84637339a953423 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225)
* 747894399d3bef0e05c561d9c67db61ab2536cf9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1833029928
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 31fe075c72fb189b9155e48ab3399e9199cc293a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231)
* 993d0dc63f3f552b8bf5b52c113f3ae8ef53304c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1411097577
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,166 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
Review Comment:
Schemas with different meanings are floating everywhere in Hudi repo. We should try to make it consistent. Please take a note, and refactor it after landing this PR. (We may also do similar things for Flink/Presto/Trino/etc later).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1843916237
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
}, {
"hash" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347",
"triggerID" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"triggerType" : "PUSH"
}, {
"hash" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350",
"triggerID" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 47e6224db0b980c0b3697b282fa724a67b2cca31 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345)
* c5f2e8da635bb7235d8b7719d7904df6c7489e0b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346)
* 8048ed72bc9a21e9d178445fdc39e1a379f183e0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347)
* 361e800774f0d6789a550fcc63c8764416ff54cc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1846236617
CI is green.
<img width="1648" alt="Screenshot 2023-12-07 at 14 59 26" src="https://github.com/apache/hudi/assets/2497195/13a096e5-3d67-4dd3-88dd-7a7e11bc27ad">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] non-working. Want to review with team that this makes sense [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1817271534
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* f58d9cb5a301e1aff325afc9f0ef3f618c9fe655 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] non-working. Want to review with team that this makes sense [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1817267096
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 184858b13509316eeb7fe0a1198b7df98c3fc50d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983)
* f58d9cb5a301e1aff325afc9f0ef3f618c9fe655 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828409625
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128)
* 2a9a36323053c5d35ae68cc5b2c38bab707b0f1a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1407078231
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/common/model/HoodieSparkRecord.java:
##########
@@ -449,7 +449,8 @@ private static void validateRow(InternalRow data, StructType schema) {
data instanceof HoodieInternalRow
|| data instanceof GenericInternalRow
|| data instanceof SpecificInternalRow
- || SparkAdapterSupport$.MODULE$.sparkAdapter().isColumnarBatchRow(data));
+ || SparkAdapterSupport$.MODULE$.sparkAdapter().isColumnarBatchRow(data))
+ || data instanceof JoinedRow;
Review Comment:
when is the data as `JoinedRow`?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,167 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? new HoodieSimpleFileGroupRecordBuffer<>(readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(), recordMerger, props)
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
+ this.baseFileIterator = makeBaseFileIterator();
scanLogFiles();
recordBuffer.setBaseFileIterator(baseFileIterator);
}
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ //MergeOnReadSnapshotRelation.isProjectionCompatible. Should centralize the logic
+ if (!hoodieTableConfig.getPayloadClass().equals(OverwriteWithLatestAvroPayload.class.getName())) {
+ return dataSchema;
+ }
+
+ List<Schema.Field> addedFields = new ArrayList<>();
+ for (String field : recordMerger.getMandatoryFieldsForMerging(hoodieTableConfig)) {
+ if (requestedSchema.getField(field) == null) {
+ Schema.Field foundField = dataSchema.getField(field);
+ if (foundField == null) {
+ throw new IllegalArgumentException("Field: " + field + " does not exist in the table schema");
+ }
+ addedFields.add(new Schema.Field(foundField.name(), foundField.schema(), foundField.doc(), foundField.defaultVal()));
+ }
+ }
+
+ if (addedFields.isEmpty()) {
+ return maybeReorderForBootstrap(requestedSchema);
+ }
+
+ return maybeReorderForBootstrap(AvroSchemaUtils.appendFieldsToSchema(requestedSchema, addedFields));
+ }
+
+ private Schema maybeReorderForBootstrap(Schema input) {
+ if (this.hoodieBaseFileOption.isPresent() && this.hoodieBaseFileOption.get().getBootstrapBaseFile().isPresent()) {
+ Pair<List<Schema.Field>,List<Schema.Field>> requiredFields = getDataAndMetaCols(input);
+ if (!(requiredFields.getLeft().isEmpty() || requiredFields.getRight().isEmpty())) {
+ return createSchemaFromFields(Stream.concat(requiredFields.getLeft().stream(), requiredFields.getRight().stream())
+ .collect(Collectors.toList()));
+ }
+ }
+ return input;
+ }
+
+ private static Pair<List<Schema.Field>,List<Schema.Field>> getDataAndMetaCols(Schema schema) {
Review Comment:
Would prefer to have this method in some util class and tested.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieSimpleFileGroupRecordBuffer.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.log.KeySpec;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+
+import java.io.IOException;
+import java.util.Map;
+
+public class HoodieSimpleFileGroupRecordBuffer<T> extends HoodieBaseFileGroupRecordBuffer<T> {
Review Comment:
add javadoc
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedParquetFileFormat.scala:
##########
@@ -41,10 +40,15 @@ import org.apache.spark.sql.types.{StructField, StructType}
import org.apache.spark.util.SerializableConfiguration
import scala.annotation.tailrec
-import scala.collection.JavaConverters._
import scala.collection.mutable
import scala.jdk.CollectionConverters.asScalaIteratorConverter
+trait HoodieFormatTrait {
+
+ //Used so that the planner only projects once and does not stack overflow
Review Comment:
```suggestion
// Used so that the planner only projects once and does not stack overflow
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -67,8 +71,8 @@
*/
public final class HoodieFileGroupReader<T> implements Closeable {
private final HoodieReaderContext<T> readerContext;
- private final Option<HoodieBaseFile> baseFilePath;
- private final Option<List<String>> logFilePathList;
+ private final Option<HoodieBaseFile> hoodieBaseFileOption;
+ private final List<HoodieLogFile> logFiles;
Review Comment:
Why remove `Option`?
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -122,6 +125,23 @@ default boolean shouldFlush(HoodieRecord record, Schema schema, TypedProperties
return true;
}
+ default String[] getMandatoryFieldsForMerging(HoodieTableConfig cfg) {
+ ArrayList<String> requiredFields = new ArrayList<>();
+ if (cfg.populateMetaFields()) {
+ requiredFields.add(HoodieRecord.RECORD_KEY_METADATA_FIELD);
+ } else {
+ cfg.getRecordKeyFieldStream().forEach(requiredFields::add);
+ }
+ String preCombine = cfg.getPreCombineField();
+
+ //maybe throw exception otherwise
Review Comment:
Should this be left to the caller? There are valid code paths with optional precombine field.
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieLogFile.java:
##########
@@ -147,6 +148,10 @@ public String getFileExtension() {
return fileExtension;
}
+ public boolean isCDC() {
Review Comment:
Should we make it a static method in `FSUtils`?
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -62,7 +63,7 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: Option[Partitioned
requiredSchema: Schema,
conf: Configuration): ClosableIterator[InternalRow] = {
val fileInfo = sparkAdapter.getSparkPartitionedFileUtils
- .createPartitionedFile(partitionValues, filePath, start, length)
+ .createPartitionedFile(InternalRow.empty, filePath, start, length)
Review Comment:
Why pass empty? I think `createPartitionedFile` needs the partition values, isn't it?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -540,6 +541,15 @@ public Option<String[]> getRecordKeyFields() {
}
}
+ public Stream<String> getRecordKeyFieldStream() {
Review Comment:
How about reusing the existing `getRecordKeyFields` method? and then use stream at the call site.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,167 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? new HoodieSimpleFileGroupRecordBuffer<>(readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(), recordMerger, props)
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
+ this.baseFileIterator = makeBaseFileIterator();
scanLogFiles();
recordBuffer.setBaseFileIterator(baseFileIterator);
}
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ //MergeOnReadSnapshotRelation.isProjectionCompatible. Should centralize the logic
+ if (!hoodieTableConfig.getPayloadClass().equals(OverwriteWithLatestAvroPayload.class.getName())) {
+ return dataSchema;
+ }
+
+ List<Schema.Field> addedFields = new ArrayList<>();
+ for (String field : recordMerger.getMandatoryFieldsForMerging(hoodieTableConfig)) {
+ if (requestedSchema.getField(field) == null) {
+ Schema.Field foundField = dataSchema.getField(field);
+ if (foundField == null) {
+ throw new IllegalArgumentException("Field: " + field + " does not exist in the table schema");
+ }
+ addedFields.add(new Schema.Field(foundField.name(), foundField.schema(), foundField.doc(), foundField.defaultVal()));
+ }
+ }
+
+ if (addedFields.isEmpty()) {
+ return maybeReorderForBootstrap(requestedSchema);
+ }
+
+ return maybeReorderForBootstrap(AvroSchemaUtils.appendFieldsToSchema(requestedSchema, addedFields));
+ }
+
+ private Schema maybeReorderForBootstrap(Schema input) {
+ if (this.hoodieBaseFileOption.isPresent() && this.hoodieBaseFileOption.get().getBootstrapBaseFile().isPresent()) {
+ Pair<List<Schema.Field>,List<Schema.Field>> requiredFields = getDataAndMetaCols(input);
+ if (!(requiredFields.getLeft().isEmpty() || requiredFields.getRight().isEmpty())) {
+ return createSchemaFromFields(Stream.concat(requiredFields.getLeft().stream(), requiredFields.getRight().stream())
+ .collect(Collectors.toList()));
+ }
+ }
+ return input;
+ }
+
+ private static Pair<List<Schema.Field>,List<Schema.Field>> getDataAndMetaCols(Schema schema) {
+ Map<Boolean, List<Schema.Field>> fieldsByMeta = schema.getFields().stream()
+ .collect(Collectors.partitioningBy(f -> HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(f.name())));
+ return Pair.of(fieldsByMeta.getOrDefault(true, Collections.emptyList()),
+ fieldsByMeta.getOrDefault(false, Collections.emptyList()));
+ }
+
+ private Schema createSchemaFromFields(List<Schema.Field> fields) {
Review Comment:
we need to test the path wherever these methods are exercised.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,167 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? new HoodieSimpleFileGroupRecordBuffer<>(readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(), recordMerger, props)
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
+ this.baseFileIterator = makeBaseFileIterator();
scanLogFiles();
recordBuffer.setBaseFileIterator(baseFileIterator);
}
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ //MergeOnReadSnapshotRelation.isProjectionCompatible. Should centralize the logic
Review Comment:
Please complete the TODOs here or remove it if it is no lponger necessary.
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -158,7 +156,7 @@ private void validateOutputFromFileGroupReader(Configuration hadoopConf,
SyncableFileSystemView fsView = viewManager.getFileSystemView(metaClient);
FileSlice fileSlice = fsView.getAllFileSlices(partitionPaths[0]).findFirst().get();
List<String> logFilePathList = getLogFileListFromFileSlice(fileSlice);
- Collections.sort(logFilePathList);
+ //Collections.sort(logFilePathList);
Review Comment:
Why removing the sorting?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedParquetFileFormat.scala:
##########
@@ -91,73 +95,69 @@ class HoodieFileGroupReaderBasedParquetFileFormat(tableState: HoodieTableState,
val outputSchema = StructType(requiredSchema.fields ++ partitionSchema.fields)
spark.conf.set("spark.sql.parquet.enableVectorizedReader", supportBatchResult)
val requiredSchemaWithMandatory = generateRequiredSchemaWithMandatory(requiredSchema, dataSchema, partitionSchema)
+ val isCount = requiredSchemaWithMandatory.isEmpty
val requiredSchemaSplits = requiredSchemaWithMandatory.fields.partition(f => HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(f.name))
val requiredMeta = StructType(requiredSchemaSplits._1)
val requiredWithoutMeta = StructType(requiredSchemaSplits._2)
val augmentedHadoopConf = FSUtils.buildInlineConf(hadoopConf)
- val (baseFileReader, preMergeBaseFileReader, _, _) = buildFileReaders(
+ val (baseFileReader, preMergeBaseFileReader, readerMaps) = buildFileReaders(
spark, dataSchema, partitionSchema, if (isIncremental) requiredSchemaWithMandatory else requiredSchema,
filters, options, augmentedHadoopConf, requiredSchemaWithMandatory, requiredWithoutMeta, requiredMeta)
+
+ val requestedAvroSchema = AvroConversionUtils.convertStructTypeToAvroSchema(requiredSchema, sanitizedTableName)
+ val dataAvroSchema = AvroConversionUtils.convertStructTypeToAvroSchema(dataSchema, sanitizedTableName)
+
val broadcastedHadoopConf = spark.sparkContext.broadcast(new SerializableConfiguration(augmentedHadoopConf))
+ val broadcastedDataSchema = spark.sparkContext.broadcast(dataAvroSchema)
+ val broadcastedRequestedSchema = spark.sparkContext.broadcast(requestedAvroSchema)
val props: TypedProperties = HoodieFileIndex.getConfigProperties(spark, options)
(file: PartitionedFile) => {
file.partitionValues match {
// Snapshot or incremental queries.
case fileSliceMapping: HoodiePartitionFileSliceMapping =>
val filePath = sparkAdapter.getSparkPartitionedFileUtils.getPathFromPartitionedFile(file)
- if (FSUtils.isLogFile(filePath)) {
- val partitionValues = fileSliceMapping.getPartitionValues
- val fileSlice = fileSliceMapping.getSlice(FSUtils.getFileId(filePath.getName).substring(1)).get
- buildFileGroupIterator(
- Option.empty[PartitionedFile => Iterator[InternalRow]],
- partitionValues,
- Option.empty[HoodieBaseFile],
- getLogFilesFromSlice(fileSlice),
- requiredSchemaWithMandatory,
- outputSchema,
- partitionSchema,
- broadcastedHadoopConf.value.value,
- -1,
- -1,
- shouldUseRecordPosition
- )
+ val filegroupName = if (FSUtils.isLogFile(filePath)) {
+ FSUtils.getFileId(filePath.getName).substring(1)
} else {
- fileSliceMapping.getSlice(FSUtils.getFileId(filePath.getName)) match {
- case Some(fileSlice) =>
+ FSUtils.getFileId(filePath.getName)
+ }
+ fileSliceMapping.getSlice(filegroupName) match {
+ case Some(fileSlice) if !isCount =>
+ if (requiredSchema.isEmpty && !fileSlice.getLogFiles.findAny().isPresent) {
val hoodieBaseFile = fileSlice.getBaseFile.get()
- val bootstrapFileOpt = hoodieBaseFile.getBootstrapBaseFile
- val partitionValues = fileSliceMapping.getPartitionValues
- val logFiles = getLogFilesFromSlice(fileSlice)
- if (requiredSchemaWithMandatory.isEmpty) {
- val baseFile = createPartitionedFile(partitionValues, hoodieBaseFile.getHadoopPath, 0, hoodieBaseFile.getFileLen)
- baseFileReader(baseFile)
- } else if (bootstrapFileOpt.isPresent) {
- // TODO: Use FileGroupReader here: HUDI-6942.
- throw new NotImplementedError("Not support reading bootstrap file")
- } else {
- if (logFiles.isEmpty) {
- throw new IllegalStateException(
- "should not be here since file slice should not have been broadcasted "
- + "since it has no log or data files")
- }
- buildFileGroupIterator(
- Option(preMergeBaseFileReader),
- partitionValues,
- Option(hoodieBaseFile),
- logFiles,
- requiredSchemaWithMandatory,
- outputSchema,
- partitionSchema,
- broadcastedHadoopConf.value.value,
- 0,
- hoodieBaseFile.getFileLen,
- shouldUseRecordPosition
- )
- }
- // TODO: Use FileGroupReader here: HUDI-6942.
- case _ => baseFileReader(file)
- }
+ baseFileReader(createPartitionedFile(fileSliceMapping.getPartitionValues, hoodieBaseFile.getHadoopPath, 0, hoodieBaseFile.getFileLen))
+ } else {
+ val readerContext: HoodieReaderContext[InternalRow] = new SparkFileFormatInternalRowReaderContext(
+ readerMaps)
+ val serializedHadoopConf = broadcastedHadoopConf.value.value
+ val metaClient: HoodieTableMetaClient = HoodieTableMetaClient
+ .builder().setConf(serializedHadoopConf).setBasePath(tableState.tablePath).build
+ val reader = new HoodieFileGroupReader[InternalRow](
+ readerContext,
+ serializedHadoopConf,
+ tableState.tablePath,
+ tableState.latestCommitTimestamp.get,
Review Comment:
i hope the latestCommitTimestamp is not empty/null. Have we considered that?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieSimpleFileGroupRecordBuffer.java:
##########
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.common.table.read;
+
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.engine.HoodieReaderContext;
+import org.apache.hudi.common.model.DeleteRecord;
+import org.apache.hudi.common.model.HoodieRecordMerger;
+import org.apache.hudi.common.table.log.KeySpec;
+import org.apache.hudi.common.table.log.block.HoodieDataBlock;
+import org.apache.hudi.common.table.log.block.HoodieDeleteBlock;
+import org.apache.hudi.common.util.Option;
+
+import org.apache.avro.Schema;
+
+import java.io.IOException;
+import java.util.Map;
+
+public class HoodieSimpleFileGroupRecordBuffer<T> extends HoodieBaseFileGroupRecordBuffer<T> {
+ public HoodieSimpleFileGroupRecordBuffer(HoodieReaderContext<T> readerContext, Schema readerSchema, Schema baseFileSchema,
+ Option<String> partitionNameOverrideOpt, Option<String[]> partitionPathFieldOpt, HoodieRecordMerger recordMerger,
+ TypedProperties payloadProps) {
+ super(readerContext, readerSchema, baseFileSchema, partitionNameOverrideOpt, partitionPathFieldOpt, recordMerger, payloadProps);
+ }
+
+ @Override
+ protected boolean doHasNext() throws IOException {
+ if (baseFileIterator.hasNext()) {
+ nextRecord = readerContext.seal(baseFileIterator.next());
+ return true;
+ }
+ return false;
+ }
+
+ @Override
+ public BufferType getBufferType() {
+ return BufferType.SIMPLE;
+ }
+
+ @Override
+ public void processDataBlock(HoodieDataBlock dataBlock, Option<KeySpec> keySpecOpt) throws IOException {
+ throw new IllegalStateException("processDataBlock should never be called from HoodieSimpleFileGroupRecordBuffer");
Review Comment:
should all these methods to process next data invoke super or throw an exception?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828421810
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128)
* 2a9a36323053c5d35ae68cc5b2c38bab707b0f1a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828818769
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* c22d1db44cd389160fe06a6f20233fb0e22df322 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162)
* 91be81a12c0321d089c89ceeb7ed0ec8d18079ea Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164)
* a045da6d6c31c09a410449788002b75e8c335548 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828669465
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 38b26030e7df02d865cfe4efd03a955124d2272a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160)
* c22d1db44cd389160fe06a6f20233fb0e22df322 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1411393845
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -62,7 +63,7 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: Option[Partitioned
requiredSchema: Schema,
conf: Configuration): ClosableIterator[InternalRow] = {
val fileInfo = sparkAdapter.getSparkPartitionedFileUtils
- .createPartitionedFile(partitionValues, filePath, start, length)
+ .createPartitionedFile(InternalRow.empty, filePath, start, length)
Review Comment:
Good, I thought about that before. Meanwhile, the partitionValues should not be passed from context also
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408362048
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,166 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
Review Comment:
What are the meanings for these two schemas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1832972346
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 747894399d3bef0e05c561d9c67db61ab2536cf9 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228)
* 31fe075c72fb189b9155e48ab3399e9199cc293a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1832622066
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* b0b711e0c355320da652fa7f2d8669539873d4d6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196)
* bfc0a855cadb4f6329bd38a470ade931797c53ab UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1411397747
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -99,4 +104,49 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: Option[Partitioned
})
deserializer.deserialize(avroRecord).get.asInstanceOf[InternalRow]
}
+
+ override def mergeBootstrapReaders(skeletonFileIterator: ClosableIterator[InternalRow],
+ dataFileIterator: ClosableIterator[InternalRow]): ClosableIterator[InternalRow] = {
+ doBootstrapMerge(skeletonFileIterator.asInstanceOf[ClosableIterator[Any]],
+ dataFileIterator.asInstanceOf[ClosableIterator[Any]])
+ }
+
+
+ protected def doBootstrapMerge(skeletonFileIterator: ClosableIterator[Any], dataFileIterator: ClosableIterator[Any]): ClosableIterator[InternalRow] = {
+ new ClosableIterator[Any] {
+ val combinedRow = new JoinedRow()
+
+ override def hasNext: Boolean = {
+ //If the iterators are out of sync it is probably due to filter pushdown
+ checkState(dataFileIterator.hasNext == skeletonFileIterator.hasNext,
+ "Bootstrap data-file iterator and skeleton-file iterator have to be in-sync!")
+ dataFileIterator.hasNext && skeletonFileIterator.hasNext
+ }
+
+ override def next(): Any = {
Review Comment:
This reminds me: do we need to support vectorization for other query types explicitly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1829055464
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166)
* 5450affcda38adf0edab0fd77c6b4214873261d3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1831207611
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* b0b711e0c355320da652fa7f2d8669539873d4d6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408161069
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieRecordMerger.java:
##########
@@ -122,6 +125,23 @@ default boolean shouldFlush(HoodieRecord record, Schema schema, TypedProperties
return true;
}
+ default String[] getMandatoryFieldsForMerging(HoodieTableConfig cfg) {
+ ArrayList<String> requiredFields = new ArrayList<>();
+ if (cfg.populateMetaFields()) {
+ requiredFields.add(HoodieRecord.RECORD_KEY_METADATA_FIELD);
+ } else {
+ cfg.getRecordKeyFieldStream().forEach(requiredFields::add);
+ }
+ String preCombine = cfg.getPreCombineField();
+
+ //maybe throw exception otherwise
Review Comment:
I decided not to throw an exception, because I think if we are going to enforce precombine being required, it should be done on the write side, not the read side. I don't think the caller should have to worry about potential nulls in the returning array. Do you think it would be better if we let it NPE? I don't have a strong opinion on what the correct thing to do here is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408169067
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -67,8 +71,8 @@
*/
public final class HoodieFileGroupReader<T> implements Closeable {
private final HoodieReaderContext<T> readerContext;
- private final Option<HoodieBaseFile> baseFilePath;
- private final Option<List<String>> logFilePathList;
+ private final Option<HoodieBaseFile> hoodieBaseFileOption;
+ private final List<HoodieLogFile> logFiles;
Review Comment:
It seemed unnecessary to me because we can just call list.isEmpty()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408334975
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -77,14 +78,18 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: Option[Partitioned
}
}).asInstanceOf[ClosableIterator[InternalRow]]
} else {
- if (baseFileReader.isEmpty) {
- throw new IllegalArgumentException("Base file reader is missing when instantiating "
- + "SparkFileFormatInternalRowReaderContext.");
+ val key = generateKey(dataSchema, requiredSchema)
+ if (!readerMaps.contains(key)) {
+ throw new IllegalStateException("schemas don't hash to a known reader")
}
- new CloseableInternalRowIterator(baseFileReader.get.apply(fileInfo))
+ new CloseableInternalRowIterator(readerMaps(key).apply(fileInfo))
}
}
+ private def generateKey(dataSchema: Schema, requestedSchema: Schema): Long = {
Review Comment:
Do we have any cases that different readers use the same data schema and requested Schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1829200902
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 5450affcda38adf0edab0fd77c6b4214873261d3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408250539
##########
hudi-common/src/test/java/org/apache/hudi/common/table/read/TestHoodieFileGroupReaderBase.java:
##########
@@ -158,7 +156,7 @@ private void validateOutputFromFileGroupReader(Configuration hadoopConf,
SyncableFileSystemView fsView = viewManager.getFileSystemView(metaClient);
FileSlice fileSlice = fsView.getAllFileSlices(partitionPaths[0]).findFirst().get();
List<String> logFilePathList = getLogFileListFromFileSlice(fileSlice);
- Collections.sort(logFilePathList);
+ //Collections.sort(logFilePathList);
Review Comment:
The filegroup reader now takes in a fileslice, so sorting here does nothing. In the reader we now sort with
```
fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828680284
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 38b26030e7df02d865cfe4efd03a955124d2272a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160)
* c22d1db44cd389160fe06a6f20233fb0e22df322 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "jonvex (via GitHub)" <gi...@apache.org>.
jonvex commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1408392073
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,166 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
Review Comment:
requested schema - the schema the caller wants (ie when next() is called, what will the schema of the record be)
required schema - the schema that we need to read from the file, for example, mor needs additional fields for merging
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #10137:
URL: https://github.com/apache/hudi/pull/10137#discussion_r1419016929
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/BaseSparkInternalRowReaderContext.java:
##########
@@ -123,4 +125,11 @@ private Object getFieldValueFromInternalRow(InternalRow row, Schema recordSchema
return null;
}
}
+
+ @Override
+ public UnaryOperator<InternalRow> projectRecord(Schema from, Schema to) {
+ UnsafeProjection projection = HoodieInternalRowUtils.generateUnsafeProjectionAlias(AvroConversionUtils.convertAvroSchemaToStructType(from),
Review Comment:
I was referring to AvroConversionUtils.convertAvroSchemaToStructType which can be replaced by HoodieInternalRowUtils.getCachedSchema(schema) to avoid repeated parsing.
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -77,14 +77,18 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: Option[Partitioned
}
}).asInstanceOf[ClosableIterator[InternalRow]]
} else {
- if (baseFileReader.isEmpty) {
- throw new IllegalArgumentException("Base file reader is missing when instantiating "
- + "SparkFileFormatInternalRowReaderContext.");
+ val key = schemaPairHashKey(dataSchema, requiredSchema)
Review Comment:
```suggestion
Val schemaPairHashKey = generateSchemaPairHashKey(dataSchema, requiredSchema)
```
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +85,167 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? new HoodieSimpleFileGroupRecordBuffer<>(readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(), recordMerger, props)
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
+ this.baseFileIterator = makeBaseFileIterator();
scanLogFiles();
recordBuffer.setBaseFileIterator(baseFileIterator);
}
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ //MergeOnReadSnapshotRelation.isProjectionCompatible. Should centralize the logic
+ if (!hoodieTableConfig.getPayloadClass().equals(OverwriteWithLatestAvroPayload.class.getName())) {
+ return dataSchema;
+ }
+
+ List<Schema.Field> addedFields = new ArrayList<>();
+ for (String field : recordMerger.getMandatoryFieldsForMerging(hoodieTableConfig)) {
+ if (requestedSchema.getField(field) == null) {
+ Schema.Field foundField = dataSchema.getField(field);
+ if (foundField == null) {
+ throw new IllegalArgumentException("Field: " + field + " does not exist in the table schema");
+ }
+ addedFields.add(new Schema.Field(foundField.name(), foundField.schema(), foundField.doc(), foundField.defaultVal()));
+ }
+ }
+
+ if (addedFields.isEmpty()) {
+ return maybeReorderForBootstrap(requestedSchema);
+ }
+
+ return maybeReorderForBootstrap(AvroSchemaUtils.appendFieldsToSchema(requestedSchema, addedFields));
+ }
+
+ private Schema maybeReorderForBootstrap(Schema input) {
+ if (this.hoodieBaseFileOption.isPresent() && this.hoodieBaseFileOption.get().getBootstrapBaseFile().isPresent()) {
+ Pair<List<Schema.Field>,List<Schema.Field>> requiredFields = getDataAndMetaCols(input);
+ if (!(requiredFields.getLeft().isEmpty() || requiredFields.getRight().isEmpty())) {
+ return createSchemaFromFields(Stream.concat(requiredFields.getLeft().stream(), requiredFields.getRight().stream())
+ .collect(Collectors.toList()));
+ }
+ }
+ return input;
+ }
+
+ private static Pair<List<Schema.Field>,List<Schema.Field>> getDataAndMetaCols(Schema schema) {
+ Map<Boolean, List<Schema.Field>> fieldsByMeta = schema.getFields().stream()
+ .collect(Collectors.partitioningBy(f -> HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(f.name())));
+ return Pair.of(fieldsByMeta.getOrDefault(true, Collections.emptyList()),
+ fieldsByMeta.getOrDefault(false, Collections.emptyList()));
+ }
+
+ private Schema createSchemaFromFields(List<Schema.Field> fields) {
Review Comment:
@jonvex could you follow up on this separately if not done in this PR?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,119 +84,213 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+
+ // requestedSchema: the schema that the caller requests
+ private final Schema requestedSchema;
+
+ // requiredSchema: the requestedSchema with any additional columns required for merging etc
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
- this.recordMerger = readerContext.getRecordMerger(
- getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
+ this.recordMerger = readerContext.getRecordMerger(tableConfig.getRecordMergerStrategy());
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? null
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
- scanLogFiles();
- recordBuffer.setBaseFileIterator(baseFileIterator);
+ ClosableIterator<T> iter = makeBaseFileIterator();
+ if (logFiles.isEmpty()) {
+ this.baseFileIterator = CachingIterator.wrap(iter, readerContext);
+ } else {
+ this.baseFileIterator = iter;
+ scanLogFiles();
+ recordBuffer.setBaseFileIterator(baseFileIterator);
+ }
+ }
+
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ List<Schema.Field> addedFields = new ArrayList<>();
+ for (String field : recordMerger.getMandatoryFieldsForMerging(hoodieTableConfig)) {
+ if (requestedSchema.getField(field) == null) {
+ Option<Schema.Field> foundFieldOpt = findNestedField(dataSchema, field);
+ if (!foundFieldOpt.isPresent()) {
+ throw new IllegalArgumentException("Field: " + field + " does not exist in the table schema");
+ }
+ Schema.Field foundField = foundFieldOpt.get();
+ addedFields.add(foundField);
+ }
+ }
+
+ if (addedFields.isEmpty()) {
+ return maybeReorderForBootstrap(requestedSchema);
+ }
+
+ return maybeReorderForBootstrap(appendFieldsToSchema(requestedSchema, addedFields));
+ }
+
+ private Schema maybeReorderForBootstrap(Schema input) {
+ if (this.hoodieBaseFileOption.isPresent() && this.hoodieBaseFileOption.get().getBootstrapBaseFile().isPresent()) {
+ Pair<List<Schema.Field>,List<Schema.Field>> requiredFields = getDataAndMetaCols(input);
+ if (!(requiredFields.getLeft().isEmpty() || requiredFields.getRight().isEmpty())) {
+ return createSchemaFromFields(Stream.concat(requiredFields.getLeft().stream(), requiredFields.getRight().stream())
+ .collect(Collectors.toList()));
+ }
+ }
+ return input;
+ }
+
+ private static Pair<List<Schema.Field>,List<Schema.Field>> getDataAndMetaCols(Schema schema) {
+ Map<Boolean, List<Schema.Field>> fieldsByMeta = schema.getFields().stream()
+ .collect(Collectors.partitioningBy(f -> HoodieRecord.HOODIE_META_COLUMNS_WITH_OPERATION.contains(f.name())));
+ return Pair.of(fieldsByMeta.getOrDefault(true, Collections.emptyList()),
+ fieldsByMeta.getOrDefault(false, Collections.emptyList()));
+ }
+
+ private Schema createSchemaFromFields(List<Schema.Field> fields) {
+ //fields have positions set, so we need to remove them due to avro setFields implementation
+ for (int i = 0; i < fields.size(); i++) {
+ Schema.Field curr = fields.get(i);
+ fields.set(i, new Schema.Field(curr.name(), curr.schema(), curr.doc(), curr.defaultVal()));
+ }
+ Schema newSchema = Schema.createRecord(dataSchema.getName(), dataSchema.getDoc(), dataSchema.getNamespace(), dataSchema.isError());
+ newSchema.setFields(fields);
+ return newSchema;
+ }
+
+ private ClosableIterator<T> makeBootstrapBaseFileIterator(HoodieBaseFile baseFile) {
+ BaseFile dataFile = baseFile.getBootstrapBaseFile().get();
+ Pair<List<Schema.Field>,List<Schema.Field>> requiredFields = getDataAndMetaCols(requiredSchema);
+ Pair<List<Schema.Field>,List<Schema.Field>> allFields = getDataAndMetaCols(dataSchema);
+
+ Option<ClosableIterator<T>> dataFileIterator = requiredFields.getRight().isEmpty() ? Option.empty() :
+ Option.of(readerContext.getFileRecordIterator(dataFile.getHadoopPath(), 0, dataFile.getFileLen(),
+ createSchemaFromFields(allFields.getRight()), createSchemaFromFields(requiredFields.getRight()), hadoopConf));
+
+ Option<ClosableIterator<T>> skeletonFileIterator = requiredFields.getLeft().isEmpty() ? Option.empty() :
+ Option.of(readerContext.getFileRecordIterator(baseFile.getHadoopPath(), 0, baseFile.getFileLen(),
+ createSchemaFromFields(allFields.getLeft()), createSchemaFromFields(requiredFields.getLeft()), hadoopConf));
+ if (!dataFileIterator.isPresent() && !skeletonFileIterator.isPresent()) {
+ throw new IllegalStateException("should not be here if only partition cols are required");
+ } else if (!dataFileIterator.isPresent()) {
+ return skeletonFileIterator.get();
+ } else if (!skeletonFileIterator.isPresent()) {
+ return dataFileIterator.get();
+ } else {
+ return readerContext.mergeBootstrapReaders(skeletonFileIterator.get(), dataFileIterator.get());
+ }
}
/**
* @return {@code true} if the next record exists; {@code false} otherwise.
* @throws IOException on reader error.
*/
public boolean hasNext() throws IOException {
- return recordBuffer.hasNext();
+ if (recordBuffer == null) {
+ return baseFileIterator.hasNext();
Review Comment:
We should revisit whether this can be directly put into the record buffer.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/read/HoodieFileGroupReader.java:
##########
@@ -81,82 +86,170 @@ public final class HoodieFileGroupReader<T> implements Closeable {
private ClosableIterator<T> baseFileIterator;
private HoodieRecordMerger recordMerger;
- public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
- HoodieTableMetaClient metaClient,
- String fileGroupId,
- TypedProperties props,
- HoodieTimeline timeline,
- HoodieTableQueryType queryType,
- Option<String> instantTime,
- Option<String> startInstantTime,
- boolean shouldUseRecordPosition) throws Exception {
- // This constructor is a placeholder now to allow automatically fetching the correct list of
- // base and log files for a file group.
- // Derive base and log files and call the corresponding constructor.
- this(readerContext, metaClient.getHadoopConf(), metaClient.getBasePathV2().toString(),
- instantTime.get(), Option.empty(), Option.empty(),
- new TableSchemaResolver(metaClient).getTableAvroSchema(),
- props, 0, Long.MAX_VALUE, shouldUseRecordPosition);
- }
+ private final Schema dataSchema;
+ private final Schema requestedSchema;
+
+ private final Schema requiredSchema;
+
+ private final HoodieTableConfig hoodieTableConfig;
+
+ private final Option<UnaryOperator<T>> outputConverter;
public HoodieFileGroupReader(HoodieReaderContext<T> readerContext,
Configuration hadoopConf,
String tablePath,
String latestCommitTime,
- Option<HoodieBaseFile> baseFilePath,
- Option<List<String>> logFilePathList,
- Schema avroSchema,
+ FileSlice fileSlice,
+ Schema dataSchema,
+ Schema requestedSchema,
TypedProperties props,
+ HoodieTableConfig tableConfig,
long start,
long length,
boolean shouldUseRecordPosition) {
this.readerContext = readerContext;
this.hadoopConf = hadoopConf;
- this.baseFilePath = baseFilePath;
- this.logFilePathList = logFilePathList;
+ this.hoodieBaseFileOption = fileSlice.getBaseFile();
+ this.logFiles = fileSlice.getLogFiles().sorted(HoodieLogFile.getLogFileComparator()).collect(Collectors.toList());
this.props = props;
this.start = start;
this.length = length;
this.recordMerger = readerContext.getRecordMerger(
getStringWithAltKeys(props, RECORD_MERGER_STRATEGY, RECORD_MERGER_STRATEGY.defaultValue()));
this.readerState.tablePath = tablePath;
this.readerState.latestCommitTime = latestCommitTime;
- this.readerState.baseFileAvroSchema = avroSchema;
- this.readerState.logRecordAvroSchema = avroSchema;
+ this.dataSchema = dataSchema;
+ this.requestedSchema = requestedSchema;
+ this.hoodieTableConfig = tableConfig;
+ this.requiredSchema = generateRequiredSchema();
+ if (!requestedSchema.equals(requiredSchema)) {
+ this.outputConverter = Option.of(readerContext.projectRecord(requiredSchema, requestedSchema));
+ } else {
+ this.outputConverter = Option.empty();
+ }
+ this.readerState.baseFileAvroSchema = requiredSchema;
+ this.readerState.logRecordAvroSchema = requiredSchema;
this.readerState.mergeProps.putAll(props);
- String filePath = baseFilePath.isPresent()
- ? baseFilePath.get().getPath()
- : logFilePathList.get().get(0);
- String partitionPath = FSUtils.getRelativePartitionPath(
- new Path(tablePath), new Path(filePath).getParent());
- Option<String> partitionNameOpt = StringUtils.isNullOrEmpty(partitionPath)
- ? Option.empty() : Option.of(partitionPath);
- Option<Object> partitionConfigValue = ConfigUtils.getRawValueWithAltKeys(props, PARTITION_FIELDS);
- Option<String[]> partitionPathFieldOpt = partitionConfigValue.isPresent()
- ? Option.of(Arrays.stream(partitionConfigValue.get().toString().split(","))
- .filter(p -> p.length() > 0).collect(Collectors.toList()).toArray(new String[] {}))
- : Option.empty();
- this.recordBuffer = shouldUseRecordPosition
+ this.recordBuffer = this.logFiles.isEmpty()
+ ? new HoodieSimpleFileGroupRecordBuffer<>(readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(), recordMerger, props)
+ : shouldUseRecordPosition
? new HoodiePositionBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props)
: new HoodieKeyBasedFileGroupRecordBuffer<>(
- readerContext, avroSchema, avroSchema, partitionNameOpt, partitionPathFieldOpt,
+ readerContext, requiredSchema, requiredSchema, Option.empty(), Option.empty(),
recordMerger, props);
+
+
}
/**
* Initialize internal iterators on the base and log files.
*/
public void initRecordIterators() {
- this.baseFileIterator = baseFilePath.isPresent()
- ? readerContext.getFileRecordIterator(
- baseFilePath.get().getHadoopPath(), start, length, readerState.baseFileAvroSchema, readerState.baseFileAvroSchema, hadoopConf)
- : new EmptyIterator<>();
+ this.baseFileIterator = makeBaseFileIterator();
scanLogFiles();
recordBuffer.setBaseFileIterator(baseFileIterator);
}
+ private ClosableIterator<T> makeBaseFileIterator() {
+ if (!hoodieBaseFileOption.isPresent()) {
+ return new EmptyIterator<>();
+ }
+
+ HoodieBaseFile baseFile = hoodieBaseFileOption.get();
+ if (baseFile.getBootstrapBaseFile().isPresent()) {
+ return makeBootstrapBaseFileIterator(baseFile);
+ }
+
+ return readerContext.getFileRecordIterator(baseFile.getHadoopPath(), start, length,
+ dataSchema, requiredSchema, hadoopConf);
+ }
+
+ private Schema generateRequiredSchema() {
+ //might need to change this if other queries than mor have mandatory fields
+ if (logFiles.isEmpty()) {
+ return requestedSchema;
+ }
+
+ if (!HoodiePayloadProps.isProjectionCompatible(hoodieTableConfig.getPayloadClass())) {
+ return dataSchema;
+ }
+
+ List<Schema.Field> addedFields = new ArrayList<>();
+ for (String field : recordMerger.getMandatoryFieldsForMerging(hoodieTableConfig)) {
+ //need to match HoodieFileGroupReaderBasedParquetFileFormat for now
+ //if (!findNestedField(requestedSchema, field).isPresent()) {
+ if (requestedSchema.getField(field) == null) {
+ Option<Schema.Field> foundFieldOpt = findNestedField(dataSchema, field);
+ if (!foundFieldOpt.isPresent()) {
+ throw new IllegalArgumentException("Field: " + field + " does not exist in the table schema");
+ }
+ Schema.Field foundField = foundFieldOpt.get();
+ addedFields.add(foundField);
+ }
+ }
+
+ if (addedFields.isEmpty()) {
+ return maybeReorderForBootstrap(requestedSchema);
+ }
+
+ //return maybeReorderForBootstrap(appendFieldsToSchemaDedupNested(requestedSchema, addedFields));
+ return maybeReorderForBootstrap(appendFieldsToSchema(requestedSchema, addedFields));
+ }
+
+ private Schema maybeReorderForBootstrap(Schema input) {
+ if (this.hoodieBaseFileOption.isPresent() && this.hoodieBaseFileOption.get().getBootstrapBaseFile().isPresent()) {
+ Pair<List<Schema.Field>,List<Schema.Field>> requiredFields = getDataAndMetaCols(input);
Review Comment:
@jonvex could you check this?
##########
hudi-common/src/main/java/org/apache/hudi/common/engine/HoodieReaderContext.java:
##########
@@ -188,4 +189,8 @@ public Map<String, Object> updateSchemaAndResetOrderingValInMetadata(Map<String,
meta.put(INTERNAL_META_SCHEMA, schema);
return meta;
}
+
+ public abstract ClosableIterator<T> mergeBootstrapReaders(ClosableIterator<T> skeletonFileIterator, ClosableIterator<T> dataFileIterator);
+
+ public abstract UnaryOperator<T> projectRecord(Schema from, Schema to);
Review Comment:
docs here
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/SparkFileFormatInternalRowReaderContext.scala:
##########
@@ -62,7 +63,7 @@ class SparkFileFormatInternalRowReaderContext(baseFileReader: Option[Partitioned
requiredSchema: Schema,
conf: Configuration): ClosableIterator[InternalRow] = {
val fileInfo = sparkAdapter.getSparkPartitionedFileUtils
- .createPartitionedFile(partitionValues, filePath, start, length)
+ .createPartitionedFile(InternalRow.empty, filePath, start, length)
Review Comment:
@jonvex nit: add a comment/doc in the code to explain why the partition value is empty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1828920258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* a045da6d6c31c09a410449788002b75e8c335548 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1845968108
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
}, {
"hash" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347",
"triggerID" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"triggerType" : "PUSH"
}, {
"hash" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350",
"triggerID" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"triggerType" : "PUSH"
}, {
"hash" : "830f88ca3021165497428d27e5e5fd19e5f3f41d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21373",
"triggerID" : "830f88ca3021165497428d27e5e5fd19e5f3f41d",
"triggerType" : "PUSH"
}, {
"hash" : "d6d0848e183d0a192fda67f647f81adcbf6f5922",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21375",
"triggerID" : "d6d0848e183d0a192fda67f647f81adcbf6f5922",
"triggerType" : "PUSH"
}, {
"hash" : "150a4b707903dd3405a5ad8235bdafc9a2b3fa9f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "150a4b707903dd3405a5ad8235bdafc9a2b3fa9f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 361e800774f0d6789a550fcc63c8764416ff54cc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350)
* 830f88ca3021165497428d27e5e5fd19e5f3f41d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21373)
* d6d0848e183d0a192fda67f647f81adcbf6f5922 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21375)
* 150a4b707903dd3405a5ad8235bdafc9a2b3fa9f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1843768074
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 993d0dc63f3f552b8bf5b52c113f3ae8ef53304c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241)
* 47e6224db0b980c0b3697b282fa724a67b2cca31 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345)
* c5f2e8da635bb7235d8b7719d7904df6c7489e0b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7137] Implement Bootstrap for new FG reader [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #10137:
URL: https://github.com/apache/hudi/pull/10137#issuecomment-1843852567
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "77205b47c45501a0d9de1ebc74d5bb8c960cd95a",
"triggerType" : "PUSH"
}, {
"hash" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20983",
"triggerID" : "184858b13509316eeb7fe0a1198b7df98c3fc50d",
"triggerType" : "PUSH"
}, {
"hash" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20985",
"triggerID" : "f58d9cb5a301e1aff325afc9f0ef3f618c9fe655",
"triggerType" : "PUSH"
}, {
"hash" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21128",
"triggerID" : "0b7f47a1c2c9a1d36b0f69b89d88cc1a40e391d6",
"triggerType" : "PUSH"
}, {
"hash" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21159",
"triggerID" : "2a9a36323053c5d35ae68cc5b2c38bab707b0f1a",
"triggerType" : "PUSH"
}, {
"hash" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21160",
"triggerID" : "38b26030e7df02d865cfe4efd03a955124d2272a",
"triggerType" : "PUSH"
}, {
"hash" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21162",
"triggerID" : "c22d1db44cd389160fe06a6f20233fb0e22df322",
"triggerType" : "PUSH"
}, {
"hash" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21164",
"triggerID" : "91be81a12c0321d089c89ceeb7ed0ec8d18079ea",
"triggerType" : "PUSH"
}, {
"hash" : "a045da6d6c31c09a410449788002b75e8c335548",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21165",
"triggerID" : "a045da6d6c31c09a410449788002b75e8c335548",
"triggerType" : "PUSH"
}, {
"hash" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21166",
"triggerID" : "3ae4d30b5af0842ac2fbd4380f70f2f573c0b7d2",
"triggerType" : "PUSH"
}, {
"hash" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21169",
"triggerID" : "5450affcda38adf0edab0fd77c6b4214873261d3",
"triggerType" : "PUSH"
}, {
"hash" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21186",
"triggerID" : "b144ee0b098dcb17972152e07cd57412ad498d47",
"triggerType" : "PUSH"
}, {
"hash" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21196",
"triggerID" : "b0b711e0c355320da652fa7f2d8669539873d4d6",
"triggerType" : "PUSH"
}, {
"hash" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21223",
"triggerID" : "bfc0a855cadb4f6329bd38a470ade931797c53ab",
"triggerType" : "PUSH"
}, {
"hash" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21225",
"triggerID" : "dfa3bdee07f850efbacb55ecc84637339a953423",
"triggerType" : "PUSH"
}, {
"hash" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21228",
"triggerID" : "747894399d3bef0e05c561d9c67db61ab2536cf9",
"triggerType" : "PUSH"
}, {
"hash" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21231",
"triggerID" : "31fe075c72fb189b9155e48ab3399e9199cc293a",
"triggerType" : "PUSH"
}, {
"hash" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241",
"triggerID" : "993d0dc63f3f552b8bf5b52c113f3ae8ef53304c",
"triggerType" : "PUSH"
}, {
"hash" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345",
"triggerID" : "47e6224db0b980c0b3697b282fa724a67b2cca31",
"triggerType" : "PUSH"
}, {
"hash" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346",
"triggerID" : "c5f2e8da635bb7235d8b7719d7904df6c7489e0b",
"triggerType" : "PUSH"
}, {
"hash" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347",
"triggerID" : "8048ed72bc9a21e9d178445fdc39e1a379f183e0",
"triggerType" : "PUSH"
}, {
"hash" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350",
"triggerID" : "361e800774f0d6789a550fcc63c8764416ff54cc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 77205b47c45501a0d9de1ebc74d5bb8c960cd95a UNKNOWN
* 993d0dc63f3f552b8bf5b52c113f3ae8ef53304c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21241)
* 47e6224db0b980c0b3697b282fa724a67b2cca31 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21345)
* c5f2e8da635bb7235d8b7719d7904df6c7489e0b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21346)
* 8048ed72bc9a21e9d178445fdc39e1a379f183e0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21347)
* 361e800774f0d6789a550fcc63c8764416ff54cc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21350)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org