You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2023/08/31 01:52:32 UTC
[GitHub] [hudi] yihua opened a new pull request, #9581: [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes
yihua opened a new pull request, #9581:
URL: https://github.com/apache/hudi/pull/9581
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
### Impact
_Describe any public API or user-facing feature change or any performance impact._
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9581: [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1700415371
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1208189ffb60441f9544933a2446ad194509c391 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1745317180
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
}, {
"hash" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* b1581950ca129d2753c53ec77c0bf046701b2c92 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206)
* 20921d8e6034d38eaefb739c021f4324b94f6803 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1752540717
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
}, {
"hash" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207",
"triggerID" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"triggerType" : "PUSH"
}, {
"hash" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208",
"triggerID" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1752515071",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* 4ce9d07dea4d5ab3088037461f9fa3b884b64eb6 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1745488652
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
}, {
"hash" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207",
"triggerID" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"triggerType" : "PUSH"
}, {
"hash" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208",
"triggerID" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* 1a11ff678d2345105879a6faa951c18d94dfa1ba Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1744462777
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* e286659cb1e1cb69126b8ec09d4e2a62969ce9d4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587)
* b71be9bbde91fb6afeb52dfc102ba789963b66ff Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1744656800
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* a3ece047efd3ef23736d5162211c4176f31468e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204)
* b1581950ca129d2753c53ec77c0bf046701b2c92 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9581:
URL: https://github.com/apache/hudi/pull/9581#discussion_r1344501029
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDeleteBlock.java:
##########
@@ -65,17 +69,44 @@ public class HoodieDeleteBlock extends HoodieLogBlock {
private static final Lazy<HoodieDeleteRecord.Builder> HOODIE_DELETE_RECORD_BUILDER_STUB =
Lazy.lazily(HoodieDeleteRecord::newBuilder);
+ private final boolean writeRecordPositions;
+ // Records to delete, sorted based on the record position if writing record position to the log block header
private DeleteRecord[] recordsToDelete;
- public HoodieDeleteBlock(DeleteRecord[] recordsToDelete, Map<HeaderMetadataType, String> header) {
- this(Option.empty(), null, false, Option.empty(), header, new HashMap<>());
- this.recordsToDelete = recordsToDelete;
+ public HoodieDeleteBlock(List<Pair<DeleteRecord, Long>> recordsToDelete,
+ boolean writeRecordPositions,
+ Map<HeaderMetadataType, String> header) {
+ this(Option.empty(), null, false, Option.empty(), header, new HashMap<>(), writeRecordPositions);
+ if (writeRecordPositions) {
+ recordsToDelete.sort((o1, o2) -> {
+ long v1 = o1.getRight();
+ long v2 = o2.getRight();
+ return Long.compare(v1, v2);
+ });
+ if (recordsToDelete.get(0).getRight() > -1L) {
+ addRecordPositionsToHeader(
Review Comment:
record position can be invalid (-1) when:
1. current location is not known (new inserts going to log file).
2. when base format is hfile (record position not supported for hfile).
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##########
@@ -89,10 +90,11 @@ public class HoodieAppendHandle<T, I, K, O> extends HoodieWriteHandle<T, I, K, O
private static final int NUMBER_OF_RECORDS_TO_ESTIMATE_RECORD_SIZE = 100;
protected final String fileId;
+ private final boolean writeRecordPositions;
Review Comment:
Renamed and moved to a level above in `HoodieWriteConfig`.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDataBlock.java:
##########
@@ -70,14 +75,31 @@ public abstract class HoodieDataBlock extends HoodieLogBlock {
* NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)
*/
public HoodieDataBlock(List<HoodieRecord> records,
+ boolean writeRecordPositions,
Map<HeaderMetadataType, String> header,
Map<HeaderMetadataType, String> footer,
String keyFieldName) {
super(header, footer, Option.empty(), Option.empty(), null, false);
+ if (writeRecordPositions) {
+ records.sort((o1, o2) -> {
Review Comment:
I've enabled writing record positions for some of the existing tests which includes the following scenarios:
1. MOR table with compaction.
2. MOR table with clustering.
3. MOR table with clustering and no base file (before clustering).
4. COW/MOR table with different index types.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -173,18 +173,18 @@ public static <R> HoodieRecord<R> tagRecord(HoodieRecord<R> record, HoodieRecord
*
* @param filePath - File to filter keys from
* @param candidateRecordKeys - Candidate keys to filter
- * @return List of candidate keys that are available in the file
+ * @return List of pairs of candidate keys and positions that are available in the file
*/
- public static List<String> filterKeysFromFile(Path filePath, List<String> candidateRecordKeys,
- Configuration configuration) throws HoodieIndexException {
+ public static List<Pair<String, Long>> filterKeysFromFile(Path filePath, List<String> candidateRecordKeys,
+ Configuration configuration) throws HoodieIndexException {
ValidationUtils.checkArgument(FSUtils.isBaseFile(filePath));
- List<String> foundRecordKeys = new ArrayList<>();
+ List<Pair<String, Long>> foundRecordKeys = new ArrayList<>();
try (HoodieFileReader fileReader = HoodieFileReaderFactory.getReaderFactory(HoodieRecordType.AVRO)
.getFileReader(configuration, filePath)) {
// Load all rowKeys from the file, to double-confirm
if (!candidateRecordKeys.isEmpty()) {
HoodieTimer timer = HoodieTimer.start();
- Set<String> fileRowKeys = fileReader.filterRowKeys(new TreeSet<>(candidateRecordKeys));
+ Set<Pair<String, Long>> fileRowKeys = fileReader.filterRowKeys(new TreeSet<>(candidateRecordKeys));
Review Comment:
Keys should be sorted for HFile reader. For others, it doesn't matter as long as it is a set (for efficient contains check). Incorporated above suggestion - here we just pass a set and init `SortedSet` in `HoodieHFileAvroReader` only instead of building `TreeSet` for every other format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9581:
URL: https://github.com/apache/hudi/pull/9581#discussion_r1349935738
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -752,6 +752,14 @@ public class HoodieWriteConfig extends HoodieConfig {
+ "The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`."
+ "By default, no Hudi client init callback is executed.");
+ public static final ConfigProperty<Boolean> WRITE_RECORD_POSITIONS = ConfigProperty
+ .key("hoodie.write.record.positions")
+ .defaultValue(false)
+ .markAdvanced()
+ .sinceVersion("1.0.0")
+ .withDocumentation("Whether to write record positions to the block header for data blocks containing updates and delete blocks. "
+ + "The record positions can be used to improve the performance of merging records from base and log files.");
Review Comment:
Those are separate. Using the record positions for merging will be done in the filegroup reader in a subsequent PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #9581:
URL: https://github.com/apache/hudi/pull/9581
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9581: [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1700264208
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1208189ffb60441f9544933a2446ad194509c391 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9581:
URL: https://github.com/apache/hudi/pull/9581#discussion_r1349935195
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/ListBasedHoodieBloomIndexHelper.java:
##########
@@ -64,22 +64,23 @@ public HoodiePairData<HoodieKey, HoodieRecordLocation> findMatchingFilesForRecor
List<HoodieKeyLookupResult> keyLookupResults =
CollectionUtils.toStream(
- new HoodieBloomIndexCheckFunction<Pair<HoodieFileGroupId, String>>(hoodieTable, config, Pair::getLeft, Pair::getRight)
- .apply(fileComparisonPairList.iterator())
- )
+ new HoodieBloomIndexCheckFunction<Pair<HoodieFileGroupId, String>>(hoodieTable, config, Pair::getLeft, Pair::getRight)
+ .apply(fileComparisonPairList.iterator())
+ )
Review Comment:
will revert.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1753135409
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
}, {
"hash" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207",
"triggerID" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"triggerType" : "PUSH"
}, {
"hash" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208",
"triggerID" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "1752515071",
"triggerType" : "MANUAL"
}, {
"hash" : "bda7b26fbc477b408e8658eb6e52badeb5c653e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20242",
"triggerID" : "bda7b26fbc477b408e8658eb6e52badeb5c653e0",
"triggerType" : "PUSH"
}, {
"hash" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20249",
"triggerID" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* bda7b26fbc477b408e8658eb6e52badeb5c653e0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20242)
* 391fc1099c8b51221a0448d8ac9d74e3013c1cd1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1752515071
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #9581:
URL: https://github.com/apache/hudi/pull/9581#discussion_r1349221039
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -173,18 +173,18 @@ public static <R> HoodieRecord<R> tagRecord(HoodieRecord<R> record, HoodieRecord
*
* @param filePath - File to filter keys from
* @param candidateRecordKeys - Candidate keys to filter
- * @return List of candidate keys that are available in the file
+ * @return List of pairs of candidate keys and positions that are available in the file
*/
- public static List<String> filterKeysFromFile(Path filePath, List<String> candidateRecordKeys,
- Configuration configuration) throws HoodieIndexException {
+ public static List<Pair<String, Long>> filterKeysFromFile(Path filePath, List<String> candidateRecordKeys,
+ Configuration configuration) throws HoodieIndexException {
ValidationUtils.checkArgument(FSUtils.isBaseFile(filePath));
- List<String> foundRecordKeys = new ArrayList<>();
+ List<Pair<String, Long>> foundRecordKeys = new ArrayList<>();
try (HoodieFileReader fileReader = HoodieFileReaderFactory.getReaderFactory(HoodieRecordType.AVRO)
.getFileReader(configuration, filePath)) {
// Load all rowKeys from the file, to double-confirm
if (!candidateRecordKeys.isEmpty()) {
HoodieTimer timer = HoodieTimer.start();
- Set<String> fileRowKeys = fileReader.filterRowKeys(new TreeSet<>(candidateRecordKeys));
+ Set<Pair<String, Long>> fileRowKeys = fileReader.filterRowKeys(candidateRecordKeys.stream().collect(Collectors.toSet()));
Review Comment:
Is this change solely for performance purpose?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1751335152
LGTM. The GH CI failure is pretty annoying though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1754287454
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 391fc1099c8b51221a0448d8ac9d74e3013c1cd1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1744751092
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* b1581950ca129d2753c53ec77c0bf046701b2c92 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9581:
URL: https://github.com/apache/hudi/pull/9581#discussion_r1349936513
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDeleteBlock.java:
##########
@@ -65,17 +69,44 @@ public class HoodieDeleteBlock extends HoodieLogBlock {
private static final Lazy<HoodieDeleteRecord.Builder> HOODIE_DELETE_RECORD_BUILDER_STUB =
Lazy.lazily(HoodieDeleteRecord::newBuilder);
+ private final boolean writeRecordPositions;
+ // Records to delete, sorted based on the record position if writing record position to the log block header
private DeleteRecord[] recordsToDelete;
- public HoodieDeleteBlock(DeleteRecord[] recordsToDelete, Map<HeaderMetadataType, String> header) {
- this(Option.empty(), null, false, Option.empty(), header, new HashMap<>());
- this.recordsToDelete = recordsToDelete;
+ public HoodieDeleteBlock(List<Pair<DeleteRecord, Long>> recordsToDelete,
+ boolean writeRecordPositions,
+ Map<HeaderMetadataType, String> header) {
+ this(Option.empty(), null, false, Option.empty(), header, new HashMap<>(), writeRecordPositions);
+ if (writeRecordPositions) {
+ recordsToDelete.sort((o1, o2) -> {
+ long v1 = o1.getRight();
+ long v2 = o2.getRight();
+ return Long.compare(v1, v2);
+ });
+ if (recordsToDelete.get(0).getRight() > -1L) {
Review Comment:
makes sense.. refactored accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1752646021
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
}, {
"hash" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207",
"triggerID" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"triggerType" : "PUSH"
}, {
"hash" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208",
"triggerID" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "1752515071",
"triggerType" : "MANUAL"
}, {
"hash" : "bda7b26fbc477b408e8658eb6e52badeb5c653e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20242",
"triggerID" : "bda7b26fbc477b408e8658eb6e52badeb5c653e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* bda7b26fbc477b408e8658eb6e52badeb5c653e0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20242)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1754300024
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20249",
"triggerID" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"triggerType" : "PUSH"
}, {
"hash" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1752515071",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 391fc1099c8b51221a0448d8ac9d74e3013c1cd1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1752568707
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
}, {
"hash" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207",
"triggerID" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"triggerType" : "PUSH"
}, {
"hash" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208",
"triggerID" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "1752515071",
"triggerType" : "MANUAL"
}, {
"hash" : "bda7b26fbc477b408e8658eb6e52badeb5c653e0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20242",
"triggerID" : "bda7b26fbc477b408e8658eb6e52badeb5c653e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* 4ce9d07dea4d5ab3088037461f9fa3b884b64eb6 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241)
* bda7b26fbc477b408e8658eb6e52badeb5c653e0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20242)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9581: [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1701927483
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* e286659cb1e1cb69126b8ec09d4e2a62969ce9d4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9581: [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1700297752
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1208189ffb60441f9544933a2446ad194509c391 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1744555123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* b71be9bbde91fb6afeb52dfc102ba789963b66ff Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203)
* a3ece047efd3ef23736d5162211c4176f31468e1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1753211609
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
}, {
"hash" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207",
"triggerID" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"triggerType" : "PUSH"
}, {
"hash" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208",
"triggerID" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "1752515071",
"triggerType" : "MANUAL"
}, {
"hash" : "bda7b26fbc477b408e8658eb6e52badeb5c653e0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20242",
"triggerID" : "bda7b26fbc477b408e8658eb6e52badeb5c653e0",
"triggerType" : "PUSH"
}, {
"hash" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20249",
"triggerID" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* 391fc1099c8b51221a0448d8ac9d74e3013c1cd1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #9581:
URL: https://github.com/apache/hudi/pull/9581#discussion_r1349217024
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -752,6 +752,14 @@ public class HoodieWriteConfig extends HoodieConfig {
+ "The class must be a subclass of `org.apache.hudi.callback.HoodieClientInitCallback`."
+ "By default, no Hudi client init callback is executed.");
+ public static final ConfigProperty<Boolean> WRITE_RECORD_POSITIONS = ConfigProperty
+ .key("hoodie.write.record.positions")
+ .defaultValue(false)
+ .markAdvanced()
+ .sinceVersion("1.0.0")
+ .withDocumentation("Whether to write record positions to the block header for data blocks containing updates and delete blocks. "
+ + "The record positions can be used to improve the performance of merging records from base and log files.");
Review Comment:
After we turn on this config, does it mean Hudi will definitely do position-based merging? Or writing positions and position-based merging are separate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1753117528
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
}, {
"hash" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207",
"triggerID" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"triggerType" : "PUSH"
}, {
"hash" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208",
"triggerID" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "1752515071",
"triggerType" : "MANUAL"
}, {
"hash" : "bda7b26fbc477b408e8658eb6e52badeb5c653e0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20242",
"triggerID" : "bda7b26fbc477b408e8658eb6e52badeb5c653e0",
"triggerType" : "PUSH"
}, {
"hash" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* bda7b26fbc477b408e8658eb6e52badeb5c653e0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20242)
* 391fc1099c8b51221a0448d8ac9d74e3013c1cd1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #9581: [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #9581:
URL: https://github.com/apache/hudi/pull/9581#discussion_r1313612529
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDeleteBlock.java:
##########
@@ -65,17 +69,44 @@ public class HoodieDeleteBlock extends HoodieLogBlock {
private static final Lazy<HoodieDeleteRecord.Builder> HOODIE_DELETE_RECORD_BUILDER_STUB =
Lazy.lazily(HoodieDeleteRecord::newBuilder);
+ private final boolean writeRecordPositions;
+ // Records to delete, sorted based on the record position if writing record position to the log block header
private DeleteRecord[] recordsToDelete;
- public HoodieDeleteBlock(DeleteRecord[] recordsToDelete, Map<HeaderMetadataType, String> header) {
- this(Option.empty(), null, false, Option.empty(), header, new HashMap<>());
- this.recordsToDelete = recordsToDelete;
+ public HoodieDeleteBlock(List<Pair<DeleteRecord, Long>> recordsToDelete,
+ boolean writeRecordPositions,
+ Map<HeaderMetadataType, String> header) {
+ this(Option.empty(), null, false, Option.empty(), header, new HashMap<>(), writeRecordPositions);
+ if (writeRecordPositions) {
+ recordsToDelete.sort((o1, o2) -> {
+ long v1 = o1.getRight();
+ long v2 = o2.getRight();
+ return Long.compare(v1, v2);
+ });
+ if (recordsToDelete.get(0).getRight() > -1L) {
+ addRecordPositionsToHeader(
Review Comment:
may I know under what circumstances, we will get -1 values here?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDataBlock.java:
##########
@@ -70,14 +75,31 @@ public abstract class HoodieDataBlock extends HoodieLogBlock {
* NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)
*/
public HoodieDataBlock(List<HoodieRecord> records,
+ boolean writeRecordPositions,
Map<HeaderMetadataType, String> header,
Map<HeaderMetadataType, String> footer,
String keyFieldName) {
super(header, footer, Option.empty(), Option.empty(), null, false);
+ if (writeRecordPositions) {
+ records.sort((o1, o2) -> {
Review Comment:
where do we guard the below condition.
in case of async compaction, latest parquet file may not exists only.
So, a new log file being added to latest file slice should not encode the write positions.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieLogBlock.java:
##########
@@ -137,6 +140,20 @@ public Roaring64NavigableMap getRecordPositions() throws IOException {
return LogReaderUtils.decodeRecordPositionsHeader(logBlockHeader.get(HeaderMetadataType.RECORD_POSITIONS));
}
+ protected void addRecordPositionsToHeader(Set<Long> positionSet, int numRecords) {
+ if (positionSet.size() == numRecords) {
Review Comment:
I see. its here.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDataBlock.java:
##########
@@ -70,14 +75,31 @@ public abstract class HoodieDataBlock extends HoodieLogBlock {
* NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)
*/
public HoodieDataBlock(List<HoodieRecord> records,
+ boolean writeRecordPositions,
Map<HeaderMetadataType, String> header,
Map<HeaderMetadataType, String> footer,
String keyFieldName) {
super(header, footer, Option.empty(), Option.empty(), null, false);
+ if (writeRecordPositions) {
+ records.sort((o1, o2) -> {
Review Comment:
where is the logic where we avoid writing record positions if there are duplicate records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9581: [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1701802062
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1208189ffb60441f9544933a2446ad194509c391 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565)
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* e286659cb1e1cb69126b8ec09d4e2a62969ce9d4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9581: [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1701789789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1208189ffb60441f9544933a2446ad194509c391 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565)
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #9581:
URL: https://github.com/apache/hudi/pull/9581#discussion_r1349225544
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/bloom/ListBasedHoodieBloomIndexHelper.java:
##########
@@ -64,22 +64,23 @@ public HoodiePairData<HoodieKey, HoodieRecordLocation> findMatchingFilesForRecor
List<HoodieKeyLookupResult> keyLookupResults =
CollectionUtils.toStream(
- new HoodieBloomIndexCheckFunction<Pair<HoodieFileGroupId, String>>(hoodieTable, config, Pair::getLeft, Pair::getRight)
- .apply(fileComparisonPairList.iterator())
- )
+ new HoodieBloomIndexCheckFunction<Pair<HoodieFileGroupId, String>>(hoodieTable, config, Pair::getLeft, Pair::getRight)
+ .apply(fileComparisonPairList.iterator())
+ )
Review Comment:
Accidental change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1754620283
The docker tests are flaky due to namenode shutting down. It has got nothing to do with the changes in this code. I am trying to fix that in HUDI-6917, #9825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9581:
URL: https://github.com/apache/hudi/pull/9581#discussion_r1349937000
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2029,6 +2029,10 @@ public long getLogFileDataBlockMaxSize() {
return getLong(HoodieStorageConfig.LOGFILE_DATA_BLOCK_MAX_SIZE);
}
+ public boolean shouldLogFileWriteRecordPositions() {
+ return getBoolean(HoodieStorageConfig.LOGFILE_WRITE_RECORD_POSITIONS);
Review Comment:
renamed and moved to write config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1754305774
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20249",
"triggerID" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"triggerType" : "PUSH"
}, {
"hash" : "391fc1099c8b51221a0448d8ac9d74e3013c1cd1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20249",
"triggerID" : "1752515071",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 391fc1099c8b51221a0448d8ac9d74e3013c1cd1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20249)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1745336244
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
}, {
"hash" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207",
"triggerID" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"triggerType" : "PUSH"
}, {
"hash" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* b1581950ca129d2753c53ec77c0bf046701b2c92 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206)
* 20921d8e6034d38eaefb739c021f4324b94f6803 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207)
* 1a11ff678d2345105879a6faa951c18d94dfa1ba UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #9581:
URL: https://github.com/apache/hudi/pull/9581#discussion_r1349221039
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -173,18 +173,18 @@ public static <R> HoodieRecord<R> tagRecord(HoodieRecord<R> record, HoodieRecord
*
* @param filePath - File to filter keys from
* @param candidateRecordKeys - Candidate keys to filter
- * @return List of candidate keys that are available in the file
+ * @return List of pairs of candidate keys and positions that are available in the file
*/
- public static List<String> filterKeysFromFile(Path filePath, List<String> candidateRecordKeys,
- Configuration configuration) throws HoodieIndexException {
+ public static List<Pair<String, Long>> filterKeysFromFile(Path filePath, List<String> candidateRecordKeys,
+ Configuration configuration) throws HoodieIndexException {
ValidationUtils.checkArgument(FSUtils.isBaseFile(filePath));
- List<String> foundRecordKeys = new ArrayList<>();
+ List<Pair<String, Long>> foundRecordKeys = new ArrayList<>();
try (HoodieFileReader fileReader = HoodieFileReaderFactory.getReaderFactory(HoodieRecordType.AVRO)
.getFileReader(configuration, filePath)) {
// Load all rowKeys from the file, to double-confirm
if (!candidateRecordKeys.isEmpty()) {
HoodieTimer timer = HoodieTimer.start();
- Set<String> fileRowKeys = fileReader.filterRowKeys(new TreeSet<>(candidateRecordKeys));
+ Set<Pair<String, Long>> fileRowKeys = fileReader.filterRowKeys(candidateRecordKeys.stream().collect(Collectors.toSet()));
Review Comment:
Is this change solely for performance purpose? Meanwhile, not sure if we should create a new interface method to capture the `position` info.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #9581:
URL: https://github.com/apache/hudi/pull/9581#discussion_r1349935956
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -173,18 +173,18 @@ public static <R> HoodieRecord<R> tagRecord(HoodieRecord<R> record, HoodieRecord
*
* @param filePath - File to filter keys from
* @param candidateRecordKeys - Candidate keys to filter
- * @return List of candidate keys that are available in the file
+ * @return List of pairs of candidate keys and positions that are available in the file
*/
- public static List<String> filterKeysFromFile(Path filePath, List<String> candidateRecordKeys,
- Configuration configuration) throws HoodieIndexException {
+ public static List<Pair<String, Long>> filterKeysFromFile(Path filePath, List<String> candidateRecordKeys,
+ Configuration configuration) throws HoodieIndexException {
ValidationUtils.checkArgument(FSUtils.isBaseFile(filePath));
- List<String> foundRecordKeys = new ArrayList<>();
+ List<Pair<String, Long>> foundRecordKeys = new ArrayList<>();
try (HoodieFileReader fileReader = HoodieFileReaderFactory.getReaderFactory(HoodieRecordType.AVRO)
.getFileReader(configuration, filePath)) {
// Load all rowKeys from the file, to double-confirm
if (!candidateRecordKeys.isEmpty()) {
HoodieTimer timer = HoodieTimer.start();
- Set<String> fileRowKeys = fileReader.filterRowKeys(new TreeSet<>(candidateRecordKeys));
+ Set<Pair<String, Long>> fileRowKeys = fileReader.filterRowKeys(candidateRecordKeys.stream().collect(Collectors.toSet()));
Review Comment:
We will use it when merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1752481011
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
}, {
"hash" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207",
"triggerID" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"triggerType" : "PUSH"
}, {
"hash" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208",
"triggerID" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* 1a11ff678d2345105879a6faa951c18d94dfa1ba Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208)
* 4ce9d07dea4d5ab3088037461f9fa3b884b64eb6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1752554282
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
}, {
"hash" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207",
"triggerID" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"triggerType" : "PUSH"
}, {
"hash" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208",
"triggerID" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"triggerType" : "PUSH"
}, {
"hash" : "4ce9d07dea4d5ab3088037461f9fa3b884b64eb6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241",
"triggerID" : "1752515071",
"triggerType" : "MANUAL"
}, {
"hash" : "bda7b26fbc477b408e8658eb6e52badeb5c653e0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bda7b26fbc477b408e8658eb6e52badeb5c653e0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* 4ce9d07dea4d5ab3088037461f9fa3b884b64eb6 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20241)
* bda7b26fbc477b408e8658eb6e52badeb5c653e0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9581: [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1701842618
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 1208189ffb60441f9544933a2446ad194509c391 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565)
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* e286659cb1e1cb69126b8ec09d4e2a62969ce9d4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1744443328
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* e286659cb1e1cb69126b8ec09d4e2a62969ce9d4 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587)
* b71be9bbde91fb6afeb52dfc102ba789963b66ff UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1744573109
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* a3ece047efd3ef23736d5162211c4176f31468e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1744672834
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* a3ece047efd3ef23736d5162211c4176f31468e1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204)
* b1581950ca129d2753c53ec77c0bf046701b2c92 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9581:
URL: https://github.com/apache/hudi/pull/9581#issuecomment-1745398875
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "1208189ffb60441f9544933a2446ad194509c391",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19565",
"triggerID" : "1208189ffb60441f9544933a2446ad194509c391",
"triggerType" : "PUSH"
}, {
"hash" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f",
"triggerType" : "PUSH"
}, {
"hash" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19587",
"triggerID" : "e286659cb1e1cb69126b8ec09d4e2a62969ce9d4",
"triggerType" : "PUSH"
}, {
"hash" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20203",
"triggerID" : "b71be9bbde91fb6afeb52dfc102ba789963b66ff",
"triggerType" : "PUSH"
}, {
"hash" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20204",
"triggerID" : "a3ece047efd3ef23736d5162211c4176f31468e1",
"triggerType" : "PUSH"
}, {
"hash" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20206",
"triggerID" : "b1581950ca129d2753c53ec77c0bf046701b2c92",
"triggerType" : "PUSH"
}, {
"hash" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207",
"triggerID" : "20921d8e6034d38eaefb739c021f4324b94f6803",
"triggerType" : "PUSH"
}, {
"hash" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208",
"triggerID" : "1a11ff678d2345105879a6faa951c18d94dfa1ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50e495ed1223eaf19ec6f0fd1f00ed13bb3c487f UNKNOWN
* 20921d8e6034d38eaefb739c021f4324b94f6803 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20207)
* 1a11ff678d2345105879a6faa951c18d94dfa1ba Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=20208)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on a diff in pull request #9581: [HUDI-6795] Implement writing record_positions to log blocks for updates and deletes
Posted by "vinothchandar (via GitHub)" <gi...@apache.org>.
vinothchandar commented on code in PR #9581:
URL: https://github.com/apache/hudi/pull/9581#discussion_r1323803839
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:
##########
@@ -173,18 +173,18 @@ public static <R> HoodieRecord<R> tagRecord(HoodieRecord<R> record, HoodieRecord
*
* @param filePath - File to filter keys from
* @param candidateRecordKeys - Candidate keys to filter
- * @return List of candidate keys that are available in the file
+ * @return List of pairs of candidate keys and positions that are available in the file
*/
- public static List<String> filterKeysFromFile(Path filePath, List<String> candidateRecordKeys,
- Configuration configuration) throws HoodieIndexException {
+ public static List<Pair<String, Long>> filterKeysFromFile(Path filePath, List<String> candidateRecordKeys,
+ Configuration configuration) throws HoodieIndexException {
ValidationUtils.checkArgument(FSUtils.isBaseFile(filePath));
- List<String> foundRecordKeys = new ArrayList<>();
+ List<Pair<String, Long>> foundRecordKeys = new ArrayList<>();
try (HoodieFileReader fileReader = HoodieFileReaderFactory.getReaderFactory(HoodieRecordType.AVRO)
.getFileReader(configuration, filePath)) {
// Load all rowKeys from the file, to double-confirm
if (!candidateRecordKeys.isEmpty()) {
HoodieTimer timer = HoodieTimer.start();
- Set<String> fileRowKeys = fileReader.filterRowKeys(new TreeSet<>(candidateRecordKeys));
+ Set<Pair<String, Long>> fileRowKeys = fileReader.filterRowKeys(new TreeSet<>(candidateRecordKeys));
Review Comment:
Use `TreeSet` here as well. does the ordering inside `fileRowKeys` matter? if so, then better use `SortedSet` here. or a list?
##########
hudi-common/src/main/java/org/apache/hudi/common/config/HoodieStorageConfig.java:
##########
@@ -106,6 +106,15 @@ public class HoodieStorageConfig extends HoodieConfig {
+ "to be appended to a log file. This helps to make sure the data appended to the log file is broken up "
+ "into sizable blocks to prevent from OOM errors. This size should be greater than the JVM memory.");
+ public static final ConfigProperty<Boolean> LOGFILE_WRITE_RECORD_POSITIONS = ConfigProperty
Review Comment:
this is not a storage config? i.e its about a higher layer functionality around updates. Move this to where the core write configs are? and call it `WRITE_RECORD_POSITIONS`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##########
@@ -89,10 +90,11 @@ public class HoodieAppendHandle<T, I, K, O> extends HoodieWriteHandle<T, I, K, O
private static final int NUMBER_OF_RECORDS_TO_ESTIMATE_RECORD_SIZE = 100;
protected final String fileId;
+ private final boolean writeRecordPositions;
Review Comment:
I like this better than even LOG_RECORD_POSITIONS as the config name/
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##########
@@ -89,10 +90,11 @@ public class HoodieAppendHandle<T, I, K, O> extends HoodieWriteHandle<T, I, K, O
private static final int NUMBER_OF_RECORDS_TO_ESTIMATE_RECORD_SIZE = 100;
protected final String fileId;
+ private final boolean writeRecordPositions;
// Buffer for holding records in memory before they are flushed to disk
private final List<HoodieRecord> recordList = new ArrayList<>();
// Buffer for holding records (to be deleted) in memory before they are flushed to disk
- private final List<DeleteRecord> recordsToDelete = new ArrayList<>();
+ private final List<Pair<DeleteRecord, Long>> recordsToDelete = new ArrayList<>();
Review Comment:
rename variable.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##########
@@ -461,11 +464,13 @@ protected void appendDataAndDeleteBlocks(Map<HeaderMetadataType, String> header,
? HoodieRecord.RECORD_KEY_METADATA_FIELD
: hoodieTable.getMetaClient().getTableConfig().getRecordKeyFieldProp();
- blocks.add(getBlock(config, pickLogDataBlockFormat(), recordList, getUpdatedHeader(header, blockSequenceNumber++, taskContextSupplier.getAttemptIdSupplier().get()), keyField));
+ blocks.add(getBlock(config, pickLogDataBlockFormat(), recordList, writeRecordPositions,
Review Comment:
name booleans with either a `should` or `is` . e,g `shouldWriteRecordPositions`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieKeyLocationFetchHandle.java:
##########
@@ -51,26 +51,28 @@ public HoodieKeyLocationFetchHandle(HoodieWriteConfig config, HoodieTable<T, I,
this.keyGeneratorOpt = keyGeneratorOpt;
}
- private List<HoodieKey> fetchHoodieKeys(HoodieBaseFile baseFile) {
+ private List<Pair<HoodieKey, Long>> fetchHoodieKeysAndPositions(HoodieBaseFile baseFile) {
Review Comment:
while you are at it. rename: `fetchRecordKeysWithPositions`
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/config/HoodieWriteConfig.java:
##########
@@ -2029,6 +2029,10 @@ public long getLogFileDataBlockMaxSize() {
return getLong(HoodieStorageConfig.LOGFILE_DATA_BLOCK_MAX_SIZE);
}
+ public boolean shouldLogFileWriteRecordPositions() {
+ return getBoolean(HoodieStorageConfig.LOGFILE_WRITE_RECORD_POSITIONS);
Review Comment:
This is more of a core write config, than it is a log file config. Can we uplevel to "LOG_RECORD_POSITIONS" i.e should we log record positions for updates/deletes or not.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDataBlock.java:
##########
@@ -70,14 +75,31 @@ public abstract class HoodieDataBlock extends HoodieLogBlock {
* NOTE: This ctor is used on the write-path (ie when records ought to be written into the log)
*/
public HoodieDataBlock(List<HoodieRecord> records,
+ boolean writeRecordPositions,
Map<HeaderMetadataType, String> header,
Map<HeaderMetadataType, String> footer,
String keyFieldName) {
super(header, footer, Option.empty(), Option.empty(), null, false);
+ if (writeRecordPositions) {
+ records.sort((o1, o2) -> {
Review Comment:
+1 valid scenarios to think through. lets also consider clustering and other scenarios where the record position is accurate.?
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDeleteBlock.java:
##########
@@ -65,17 +69,44 @@ public class HoodieDeleteBlock extends HoodieLogBlock {
private static final Lazy<HoodieDeleteRecord.Builder> HOODIE_DELETE_RECORD_BUILDER_STUB =
Lazy.lazily(HoodieDeleteRecord::newBuilder);
+ private final boolean writeRecordPositions;
+ // Records to delete, sorted based on the record position if writing record position to the log block header
private DeleteRecord[] recordsToDelete;
- public HoodieDeleteBlock(DeleteRecord[] recordsToDelete, Map<HeaderMetadataType, String> header) {
- this(Option.empty(), null, false, Option.empty(), header, new HashMap<>());
- this.recordsToDelete = recordsToDelete;
+ public HoodieDeleteBlock(List<Pair<DeleteRecord, Long>> recordsToDelete,
+ boolean writeRecordPositions,
+ Map<HeaderMetadataType, String> header) {
+ this(Option.empty(), null, false, Option.empty(), header, new HashMap<>(), writeRecordPositions);
+ if (writeRecordPositions) {
+ recordsToDelete.sort((o1, o2) -> {
+ long v1 = o1.getRight();
+ long v2 = o2.getRight();
+ return Long.compare(v1, v2);
+ });
+ if (recordsToDelete.get(0).getRight() > -1L) {
Review Comment:
if you can to check if its not invalid, can we use the constant you define before and call a helper like HoodieRecordLocation.isValidPosition() or sth?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org