You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/18 07:00:32 UTC
[GitHub] [hudi] codope opened a new pull request, #5342: [HUDI-3899] Drop index to delete pending index instants from timeline
codope opened a new pull request, #5342:
URL: https://github.com/apache/hudi/pull/5342
## What is the purpose of the pull request
Currently, drop index deletes the partition but leaves any pending instant on the timeline. If we do not delete the pending index instants, then compaction may not kick in the MDT. This PR fixes the behavior.
## Brief change log
- Update dropIndex API in `HoodieBackedTableMetadataWriter`
- Add UT for the scenario
## Verify this pull request
- Added a UT, which schedules column_stats index but then calls drop index, so the indexing.requested should be deleted.
- Manually verified for indexing.inflight by killing the regular writer.
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline if applicable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101855445
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095",
"triggerID" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"triggerType" : "PUSH"
}, {
"hash" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8108",
"triggerID" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"triggerType" : "PUSH"
}, {
"hash" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114",
"triggerID" : "1101771568",
"triggerType" : "MANUAL"
}, {
"hash" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114",
"triggerID" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1101771568",
"triggerType" : "MANUAL"
}, {
"hash" : "9d6268b7a98f5f0bb2b13a6745314cf48aa4dea8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9d6268b7a98f5f0bb2b13a6745314cf48aa4dea8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94b2045d1b9d20edadba53ae06af79540f73e22b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114)
* 9d6268b7a98f5f0bb2b13a6745314cf48aa4dea8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline if applicable
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101771568
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline if applicable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101776072
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095",
"triggerID" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"triggerType" : "PUSH"
}, {
"hash" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8108",
"triggerID" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1101771568",
"triggerType" : "MANUAL"
}, {
"hash" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114",
"triggerID" : "1101771568",
"triggerType" : "MANUAL"
}, {
"hash" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114",
"triggerID" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 94b2045d1b9d20edadba53ae06af79540f73e22b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101498424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095",
"triggerID" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"triggerType" : "PUSH"
}, {
"hash" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9bb3e7a690559977a4e30804e621fb85f96bf670 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095)
* f10e6ece4f254320ed20b28c721b8e82f201e163 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline if applicable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101856951
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095",
"triggerID" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"triggerType" : "PUSH"
}, {
"hash" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8108",
"triggerID" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"triggerType" : "PUSH"
}, {
"hash" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114",
"triggerID" : "1101771568",
"triggerType" : "MANUAL"
}, {
"hash" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114",
"triggerID" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1101771568",
"triggerType" : "MANUAL"
}, {
"hash" : "9d6268b7a98f5f0bb2b13a6745314cf48aa4dea8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8118",
"triggerID" : "9d6268b7a98f5f0bb2b13a6745314cf48aa4dea8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 94b2045d1b9d20edadba53ae06af79540f73e22b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114)
* 9d6268b7a98f5f0bb2b13a6745314cf48aa4dea8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8118)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline if applicable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101773545
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095",
"triggerID" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"triggerType" : "PUSH"
}, {
"hash" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8108",
"triggerID" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"triggerType" : "PUSH"
}, {
"hash" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1101771568",
"triggerType" : "MANUAL"
}, {
"hash" : "",
"status" : "CANCELED",
"url" : "TBD",
"triggerID" : "1101771568",
"triggerType" : "MANUAL"
}, {
"hash" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* Unknown: [CANCELED](TBD)
* 94b2045d1b9d20edadba53ae06af79540f73e22b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline if applicable
Posted by GitBox <gi...@apache.org>.
nsivabalan merged PR #5342:
URL: https://github.com/apache/hudi/pull/5342
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101162432
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095",
"triggerID" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9bb3e7a690559977a4e30804e621fb85f96bf670 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101733790
There is some flakiness around TestHoodieIndexer.testTwoIndexersOneCreateOneDropPartition. I tried running it locally for 30 times and could not reproduce. have re-triggered CI run. lets see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101261772
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095",
"triggerID" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9bb3e7a690559977a4e30804e621fb85f96bf670 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101501037
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095",
"triggerID" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"triggerType" : "PUSH"
}, {
"hash" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8108",
"triggerID" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9bb3e7a690559977a4e30804e621fb85f96bf670 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095)
* f10e6ece4f254320ed20b28c721b8e82f201e163 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8108)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline if applicable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101818896
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095",
"triggerID" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"triggerType" : "PUSH"
}, {
"hash" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8108",
"triggerID" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"triggerType" : "PUSH"
}, {
"hash" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114",
"triggerID" : "1101771568",
"triggerType" : "MANUAL"
}, {
"hash" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114",
"triggerID" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1101771568",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 94b2045d1b9d20edadba53ae06af79540f73e22b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101160120
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9bb3e7a690559977a4e30804e621fb85f96bf670 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101557586
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095",
"triggerID" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"triggerType" : "PUSH"
}, {
"hash" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8108",
"triggerID" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f10e6ece4f254320ed20b28c721b8e82f201e163 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8108)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on code in PR #5342:
URL: https://github.com/apache/hudi/pull/5342#discussion_r852164219
##########
hudi-utilities/src/test/java/org/apache/hudi/utilities/TestHoodieIndexer.java:
##########
@@ -132,29 +139,84 @@ public void testIndexerWithNotAllIndexesEnabled() {
assertNoWriteErrors(statuses);
// validate table config
- assertTrue(HoodieTableMetadataUtil.getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(FILES.getPartitionPath()));
- assertTrue(HoodieTableMetadataUtil.getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(BLOOM_FILTERS.getPartitionPath()));
+ assertTrue(getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(FILES.getPartitionPath()));
+ assertTrue(getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(BLOOM_FILTERS.getPartitionPath()));
// build indexer config which has only column_stats enabled (files is enabled by default)
HoodieIndexer.Config config = new HoodieIndexer.Config();
String propsPath = Objects.requireNonNull(getClass().getClassLoader().getResource("delta-streamer-config/indexer.properties")).getPath();
config.basePath = basePath;
config.tableName = tableName;
config.indexTypes = "COLUMN_STATS";
- config.runningMode = "scheduleAndExecute";
+ config.runningMode = SCHEDULE_AND_EXECUTE;
config.propsFilePath = propsPath;
// start the indexer and validate column_stats index is also complete
HoodieIndexer indexer = new HoodieIndexer(jsc, config);
assertEquals(0, indexer.start(0));
// validate table config
- assertTrue(HoodieTableMetadataUtil.getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(FILES.getPartitionPath()));
- assertTrue(HoodieTableMetadataUtil.getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(BLOOM_FILTERS.getPartitionPath()));
- assertTrue(HoodieTableMetadataUtil.getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(COLUMN_STATS.getPartitionPath()));
+ assertTrue(getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(FILES.getPartitionPath()));
+ assertTrue(getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(BLOOM_FILTERS.getPartitionPath()));
+ assertTrue(getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(COLUMN_STATS.getPartitionPath()));
// validate metadata partitions actually exist
- assertTrue(HoodieTableMetadataUtil.metadataPartitionExists(basePath, context, FILES));
- assertTrue(HoodieTableMetadataUtil.metadataPartitionExists(basePath, context, COLUMN_STATS));
- assertTrue(HoodieTableMetadataUtil.metadataPartitionExists(basePath, context, BLOOM_FILTERS));
+ assertTrue(metadataPartitionExists(basePath, context, FILES));
+ assertTrue(metadataPartitionExists(basePath, context, COLUMN_STATS));
+ assertTrue(metadataPartitionExists(basePath, context, BLOOM_FILTERS));
+ }
+
+ @Test
+ public void testIndexerDropPartitionDeletesInstantFromTimeline() throws Exception {
+ initTestDataGenerator();
+ String tableName = "indexer_test";
+ HoodieWriteConfig.Builder writeConfigBuilder = getWriteConfigBuilder(basePath, tableName);
+ // enable files on the regular write client
+ HoodieMetadataConfig.Builder metadataConfigBuilder = getMetadataConfigBuilder(true, false).withMetadataIndexBloomFilter(true);
+ HoodieWriteConfig writeConfig = writeConfigBuilder.withMetadataConfig(metadataConfigBuilder.build()).build();
+ // do one upsert with synchronous metadata update
+ SparkRDDWriteClient writeClient = new SparkRDDWriteClient(context, writeConfig);
+ String instant = "0001";
+ writeClient.startCommitWithTime(instant);
+ List<HoodieRecord> records = dataGen.generateInserts(instant, 100);
+ JavaRDD<WriteStatus> result = writeClient.upsert(jsc.parallelize(records, 1), instant);
+ List<WriteStatus> statuses = result.collect();
+ assertNoWriteErrors(statuses);
+
+ // validate partitions built successfully
+ assertTrue(getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(FILES.getPartitionPath()));
+ assertTrue(metadataPartitionExists(basePath, context, FILES));
+ assertTrue(getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(BLOOM_FILTERS.getPartitionPath()));
+ assertTrue(metadataPartitionExists(basePath, context, BLOOM_FILTERS));
+
+ // build indexer config which has only column_stats enabled (files is enabled by default)
+ HoodieIndexer.Config config = new HoodieIndexer.Config();
+ String propsPath = Objects.requireNonNull(getClass().getClassLoader().getResource("delta-streamer-config/indexer.properties")).getPath();
+ config.basePath = basePath;
+ config.tableName = tableName;
+ config.indexTypes = "COLUMN_STATS";
+ config.runningMode = SCHEDULE;
+ config.propsFilePath = propsPath;
+
+ // schedule indexing and validate column_stats index is also initialized
+ HoodieIndexer indexer = new HoodieIndexer(jsc, config);
+ assertEquals(0, indexer.start(0));
+ Option<HoodieInstant> indexInstantInTimeline = metaClient.reloadActiveTimeline().filterPendingIndexTimeline().lastInstant();
+ assertTrue(indexInstantInTimeline.isPresent());
+ assertEquals(REQUESTED, indexInstantInTimeline.get().getState());
+ assertTrue(metadataPartitionExists(basePath, context, COLUMN_STATS));
+
+ // drop column_stats and validate indexing.requested is also removed from the timeline
+ config.runningMode = DROP_INDEX;
+ indexer = new HoodieIndexer(jsc, config);
+ assertEquals(0, indexer.start(0));
+ indexInstantInTimeline = metaClient.reloadActiveTimeline().filterPendingIndexTimeline().lastInstant();
+ assertFalse(indexInstantInTimeline.isPresent());
+ assertFalse(metadataPartitionExists(basePath, context, COLUMN_STATS));
+
+ // check other partitions are intact
+ assertTrue(getCompletedMetadataPartitions(reload(metaClient).getTableConfig()).contains(FILES.getPartitionPath()));
Review Comment:
if we have any other tests for DropPartitions, can we add assertion that for a fully built out MDT partition, timeline files should not be removed. ie. if index instants are completed, dropPartition should not touch the timeline files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #5342: [HUDI-3899] Drop index to delete pending index instants from timeline if applicable
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5342:
URL: https://github.com/apache/hudi/pull/5342#issuecomment-1101912976
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8095",
"triggerID" : "9bb3e7a690559977a4e30804e621fb85f96bf670",
"triggerType" : "PUSH"
}, {
"hash" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8108",
"triggerID" : "f10e6ece4f254320ed20b28c721b8e82f201e163",
"triggerType" : "PUSH"
}, {
"hash" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114",
"triggerID" : "1101771568",
"triggerType" : "MANUAL"
}, {
"hash" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8114",
"triggerID" : "94b2045d1b9d20edadba53ae06af79540f73e22b",
"triggerType" : "PUSH"
}, {
"hash" : "",
"status" : "DELETED",
"url" : "TBD",
"triggerID" : "1101771568",
"triggerType" : "MANUAL"
}, {
"hash" : "9d6268b7a98f5f0bb2b13a6745314cf48aa4dea8",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8118",
"triggerID" : "9d6268b7a98f5f0bb2b13a6745314cf48aa4dea8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9d6268b7a98f5f0bb2b13a6745314cf48aa4dea8 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8118)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org