You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/12/10 16:33:15 UTC
[GitHub] [hudi] yihua opened a new pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
yihua opened a new pull request #4278:
URL: https://github.com/apache/hudi/pull/4278
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991395953
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 347001542a65afa8dc4ce42f4b80fbedf8e8036d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182)
* 25a3f9b1e23db855abe6f352c391033fdef4d460 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991414894
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25a3f9b1e23db855abe6f352c391033fdef4d460 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184)
* 6b705b1a371fe8af7fb65795a1083ca13e9e7348 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991127663
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfa6aa245ac894cf9187ca73e741daed595cc27c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168)
* c5ae7055a92e92d7f01f082d364d36cead5f8569 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991123083
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfa6aa245ac894cf9187ca73e741daed595cc27c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991193330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991426720
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b705b1a371fe8af7fb65795a1083ca13e9e7348 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187)
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991129712
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfa6aa245ac894cf9187ca73e741daed595cc27c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168)
* c5ae7055a92e92d7f01f082d364d36cead5f8569 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
yihua merged pull request #4278:
URL: https://github.com/apache/hudi/pull/4278
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991427887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b705b1a371fe8af7fb65795a1083ca13e9e7348 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187)
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991407294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25a3f9b1e23db855abe6f352c391033fdef4d460 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991468392
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
}, {
"hash" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4194",
"triggerID" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* 432c2aff71bf918d22a3c6e81f23b11f5297d3b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4194)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991453789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
}, {
"hash" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4194",
"triggerID" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190)
* 432c2aff71bf918d22a3c6e81f23b11f5297d3b0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4194)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991361399
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 347001542a65afa8dc4ce42f4b80fbedf8e8036d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#discussion_r781513914
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java
##########
@@ -143,15 +149,19 @@ public HoodieArchivedTimeline reload() {
}
private HoodieInstant readCommit(GenericRecord record, boolean loadDetails) {
- final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
+ final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
final String action = record.get(ACTION_TYPE_KEY).toString();
+ String actionState = record.get(ACTION_STATE).toString();
if (loadDetails) {
getMetadataKey(action).map(key -> {
Object actionData = record.get(key);
- if (action.equals(HoodieTimeline.COMPACTION_ACTION)) {
- this.readCommits.put(instantTime, HoodieAvroUtils.indexedRecordToBytes((IndexedRecord)actionData));
- } else {
- this.readCommits.put(instantTime, actionData.toString().getBytes(StandardCharsets.UTF_8));
+ // Only puts instant details if the instant is completed and there is action data
+ if (HoodieInstant.State.COMPLETED.toString().equals(actionState) && actionData != null) {
Review comment:
Yeah, that's a good point. The archival should not have inflight instant. To avoid breaking changes, I used a different way of fixing the `readCommit`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991127663
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfa6aa245ac894cf9187ca73e741daed595cc27c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168)
* c5ae7055a92e92d7f01f082d364d36cead5f8569 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991407294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25a3f9b1e23db855abe6f352c391033fdef4d460 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991427887
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b705b1a371fe8af7fb65795a1083ca13e9e7348 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187)
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991415363
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25a3f9b1e23db855abe6f352c391033fdef4d460 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184)
* 6b705b1a371fe8af7fb65795a1083ca13e9e7348 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#discussion_r781511307
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieRepairTool.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.FileSystemBackedTableMetadata;
+import org.apache.hudi.table.repair.RepairUtils;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+public class HoodieRepairTool {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieRepairTool.class);
+ private static final String BACKUP_DIR_PREFIX = "hoodie_repair_backup_";
+ // Repair config
+ private final Config cfg;
+ // Properties with source, hoodie client, key generator etc.
+ private TypedProperties props;
+ // Spark context
+ private final JavaSparkContext jsc;
+ private final HoodieTableMetaClient metaClient;
+ private final FileSystemBackedTableMetadata tableMetadata;
+
+ public HoodieRepairTool(JavaSparkContext jsc, Config cfg) {
+ if (cfg.propsFilePath != null) {
+ cfg.propsFilePath = FSUtils.addSchemeIfLocalPath(cfg.propsFilePath).toString();
+ }
+ this.jsc = jsc;
+ this.cfg = cfg;
+ this.props = cfg.propsFilePath == null
+ ? UtilHelpers.buildProperties(cfg.configs)
+ : readConfigFromFileSystem(jsc, cfg);
+ this.metaClient = HoodieTableMetaClient.builder()
+ .setConf(jsc.hadoopConfiguration()).setBasePath(cfg.basePath)
+ .setLoadActiveTimelineOnLoad(true)
+ .build();
+ this.tableMetadata = new FileSystemBackedTableMetadata(
+ new HoodieSparkEngineContext(jsc),
+ new SerializableConfiguration(jsc.hadoopConfiguration()),
+ cfg.basePath, cfg.assumeDatePartitioning);
+ }
+
+ public void run() {
+ Option<String> startingInstantOption = Option.ofNullable(cfg.startingInstantTime);
+ Option<String> endingInstantOption = Option.ofNullable(cfg.endingInstantTime);
+
+ if (startingInstantOption.isPresent() && endingInstantOption.isPresent()) {
+ LOG.info(String.format("Start repairing completed instants between %s and %s (inclusive)",
+ startingInstantOption.get(), endingInstantOption.get()));
+ } else if (startingInstantOption.isPresent()) {
+ LOG.info(String.format("Start repairing completed instants from %s (inclusive)",
+ startingInstantOption.get()));
+ } else if (endingInstantOption.isPresent()) {
+ LOG.info(String.format("Start repairing completed instants till %s (inclusive)",
+ endingInstantOption.get()));
+ } else {
+ LOG.info("Start repairing all completed instants");
+ }
+
+ try {
+ Mode mode = Mode.valueOf(cfg.runningMode.toUpperCase());
+ switch (mode) {
+ case REPAIR:
+ LOG.info(" ****** The repair tool is in REPAIR mode, dangling data and logs files "
Review comment:
This is intentional to alert the user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991395953
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 347001542a65afa8dc4ce42f4b80fbedf8e8036d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182)
* 25a3f9b1e23db855abe6f352c391033fdef4d460 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991129712
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfa6aa245ac894cf9187ca73e741daed595cc27c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168)
* c5ae7055a92e92d7f01f082d364d36cead5f8569 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991415363
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25a3f9b1e23db855abe6f352c391033fdef4d460 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184)
* 6b705b1a371fe8af7fb65795a1083ca13e9e7348 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991193330
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5ae7055a92e92d7f01f082d364d36cead5f8569 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
yihua merged pull request #4278:
URL: https://github.com/apache/hudi/pull/4278
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991361399
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 347001542a65afa8dc4ce42f4b80fbedf8e8036d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991426720
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b705b1a371fe8af7fb65795a1083ca13e9e7348 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187)
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#discussion_r767032764
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -316,9 +316,20 @@ public synchronized HoodieArchivedTimeline getArchivedTimeline() {
return archivedTimeline;
}
+ /**
+ * Get the archived instants as a timeline with all instant details.
+ * This is costly operation, as all data from the archived files are read.
+ *
+ * @return archived instant timeline
+ */
+ public synchronized HoodieArchivedTimeline getArchivedTimelineWithInstantDetails() {
Review comment:
why this needs to be synchronized ?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieRepairTool.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.FileSystemBackedTableMetadata;
+import org.apache.hudi.table.repair.RepairUtils;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+public class HoodieRepairTool {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieRepairTool.class);
+ private static final String BACKUP_DIR_PREFIX = "hoodie_repair_backup_";
+ // Repair config
+ private final Config cfg;
+ // Properties with source, hoodie client, key generator etc.
+ private TypedProperties props;
+ // Spark context
+ private final JavaSparkContext jsc;
+ private final HoodieTableMetaClient metaClient;
+ private final FileSystemBackedTableMetadata tableMetadata;
+
+ public HoodieRepairTool(JavaSparkContext jsc, Config cfg) {
+ if (cfg.propsFilePath != null) {
+ cfg.propsFilePath = FSUtils.addSchemeIfLocalPath(cfg.propsFilePath).toString();
+ }
+ this.jsc = jsc;
+ this.cfg = cfg;
+ this.props = cfg.propsFilePath == null
+ ? UtilHelpers.buildProperties(cfg.configs)
+ : readConfigFromFileSystem(jsc, cfg);
+ this.metaClient = HoodieTableMetaClient.builder()
+ .setConf(jsc.hadoopConfiguration()).setBasePath(cfg.basePath)
+ .setLoadActiveTimelineOnLoad(true)
+ .build();
+ this.tableMetadata = new FileSystemBackedTableMetadata(
+ new HoodieSparkEngineContext(jsc),
+ new SerializableConfiguration(jsc.hadoopConfiguration()),
+ cfg.basePath, cfg.assumeDatePartitioning);
+ }
+
+ public void run() {
+ Option<String> startingInstantOption = Option.ofNullable(cfg.startingInstantTime);
+ Option<String> endingInstantOption = Option.ofNullable(cfg.endingInstantTime);
+
+ if (startingInstantOption.isPresent() && endingInstantOption.isPresent()) {
+ LOG.info(String.format("Start repairing completed instants between %s and %s (inclusive)",
+ startingInstantOption.get(), endingInstantOption.get()));
+ } else if (startingInstantOption.isPresent()) {
+ LOG.info(String.format("Start repairing completed instants from %s (inclusive)",
+ startingInstantOption.get()));
+ } else if (endingInstantOption.isPresent()) {
+ LOG.info(String.format("Start repairing completed instants till %s (inclusive)",
+ endingInstantOption.get()));
+ } else {
+ LOG.info("Start repairing all completed instants");
+ }
+
+ try {
+ Mode mode = Mode.valueOf(cfg.runningMode.toUpperCase());
+ switch (mode) {
+ case REPAIR:
+ LOG.info(" ****** The repair tool is in REPAIR mode, dangling data and logs files "
Review comment:
is "****" intentional or did you plan to remove it
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieRepairTool.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.FileSystemBackedTableMetadata;
+import org.apache.hudi.table.repair.RepairUtils;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+public class HoodieRepairTool {
Review comment:
java docs.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/repair/RepairUtils.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.table.repair;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.table.timeline.HoodieTimeline.COMMIT_ACTION;
+import static org.apache.hudi.common.table.timeline.HoodieTimeline.DELTA_COMMIT_ACTION;
+import static org.apache.hudi.common.table.timeline.HoodieTimeline.REPLACE_COMMIT_ACTION;
+
+public final class RepairUtils {
+ public static Map<String, List<String>> tagInstantsOfBaseAndLogFiles(
+ String basePath, String baseFileExtension, List<Path> allPaths) {
+ // Instant time -> Set of base and log file paths
+ Map<String, List<String>> instantToFilesMap = new HashMap<>();
+ allPaths.forEach(path -> {
+ String instantTime = path.toString().endsWith(baseFileExtension)
+ ? FSUtils.getCommitTime(path.getName()) : FSUtils.getBaseCommitTimeFromLogPath(path);
+ instantToFilesMap.computeIfAbsent(instantTime, k -> new ArrayList<>());
+ instantToFilesMap.get(instantTime).add(
+ FSUtils.getRelativePartitionPath(new Path(basePath), path));
+ });
+ return instantToFilesMap;
+ }
+
+ public static Option<Set<String>> getBaseAndLogFilePathsFromTimeline(
+ HoodieTimeline timeline, HoodieInstant instant) throws IOException {
+ if (!instant.isCompleted()) {
+ throw new HoodieException("Cannot get base and log file paths from "
+ + "instant not completed: " + instant.getTimestamp());
+ }
+
+ switch (instant.getAction()) {
+ case COMMIT_ACTION:
+ case DELTA_COMMIT_ACTION:
+ final HoodieCommitMetadata commitMetadata =
+ HoodieCommitMetadata.fromBytes(
+ timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+ return Option.of(commitMetadata.getPartitionToWriteStats().values().stream().flatMap(List::stream)
+ .map(HoodieWriteStat::getPath).collect(Collectors.toSet()));
+ case REPLACE_COMMIT_ACTION:
+ final HoodieReplaceCommitMetadata replaceCommitMetadata =
+ HoodieReplaceCommitMetadata.fromBytes(
+ timeline.getInstantDetails(instant).get(), HoodieReplaceCommitMetadata.class);
+ return Option.of(replaceCommitMetadata.getPartitionToWriteStats().values().stream().flatMap(List::stream)
+ .map(HoodieWriteStat::getPath).collect(Collectors.toSet()));
+ default:
+ return Option.empty();
Review comment:
should we consider other actions like rollback and savepoint etc.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java
##########
@@ -143,15 +149,19 @@ public HoodieArchivedTimeline reload() {
}
private HoodieInstant readCommit(GenericRecord record, boolean loadDetails) {
- final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
+ final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
final String action = record.get(ACTION_TYPE_KEY).toString();
+ String actionState = record.get(ACTION_STATE).toString();
if (loadDetails) {
getMetadataKey(action).map(key -> {
Object actionData = record.get(key);
- if (action.equals(HoodieTimeline.COMPACTION_ACTION)) {
- this.readCommits.put(instantTime, HoodieAvroUtils.indexedRecordToBytes((IndexedRecord)actionData));
- } else {
- this.readCommits.put(instantTime, actionData.toString().getBytes(StandardCharsets.UTF_8));
+ // Only puts instant details if the instant is completed and there is action data
+ if (HoodieInstant.State.COMPLETED.toString().equals(actionState) && actionData != null) {
Review comment:
looked at the code to ensure that archival will not move past the earlist inflight instant in timline. But just inacse someone wants to double check, leaving a comment here. if this assumption is wrong, this fix has to be reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991396490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 347001542a65afa8dc4ce42f4b80fbedf8e8036d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182)
* 25a3f9b1e23db855abe6f352c391033fdef4d460 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991396490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 347001542a65afa8dc4ce42f4b80fbedf8e8036d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182)
* 25a3f9b1e23db855abe6f352c391033fdef4d460 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991414894
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25a3f9b1e23db855abe6f352c391033fdef4d460 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184)
* 6b705b1a371fe8af7fb65795a1083ca13e9e7348 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991360258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5ae7055a92e92d7f01f082d364d36cead5f8569 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170)
* 347001542a65afa8dc4ce42f4b80fbedf8e8036d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot edited a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot edited a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991468392
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
}, {
"hash" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4194",
"triggerID" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* 432c2aff71bf918d22a3c6e81f23b11f5297d3b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4194)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991360258
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5ae7055a92e92d7f01f082d364d36cead5f8569 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170)
* 347001542a65afa8dc4ce42f4b80fbedf8e8036d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991435391
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#discussion_r781512566
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -316,9 +316,20 @@ public synchronized HoodieArchivedTimeline getArchivedTimeline() {
return archivedTimeline;
}
+ /**
+ * Get the archived instants as a timeline with all instant details.
+ * This is costly operation, as all data from the archived files are read.
+ *
+ * @return archived instant timeline
+ */
+ public synchronized HoodieArchivedTimeline getArchivedTimelineWithInstantDetails() {
Review comment:
This method is no longer added. FYI, it followed the same conversion of other methods getting the timeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991193330
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c5ae7055a92e92d7f01f082d364d36cead5f8569 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991425623
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25a3f9b1e23db855abe6f352c391033fdef4d460 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184)
* 6b705b1a371fe8af7fb65795a1083ca13e9e7348 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187)
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991432196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b705b1a371fe8af7fb65795a1083ca13e9e7348 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187)
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991432196
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6b705b1a371fe8af7fb65795a1083ca13e9e7348 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187)
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991435391
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991453789
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
}, {
"hash" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4194",
"triggerID" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190)
* 432c2aff71bf918d22a3c6e81f23b11f5297d3b0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4194)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991120863
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfa6aa245ac894cf9187ca73e741daed595cc27c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#discussion_r767032764
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableMetaClient.java
##########
@@ -316,9 +316,20 @@ public synchronized HoodieArchivedTimeline getArchivedTimeline() {
return archivedTimeline;
}
+ /**
+ * Get the archived instants as a timeline with all instant details.
+ * This is costly operation, as all data from the archived files are read.
+ *
+ * @return archived instant timeline
+ */
+ public synchronized HoodieArchivedTimeline getArchivedTimelineWithInstantDetails() {
Review comment:
why this needs to be synchronized ?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieRepairTool.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.FileSystemBackedTableMetadata;
+import org.apache.hudi.table.repair.RepairUtils;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+public class HoodieRepairTool {
+
+ private static final Logger LOG = LogManager.getLogger(HoodieRepairTool.class);
+ private static final String BACKUP_DIR_PREFIX = "hoodie_repair_backup_";
+ // Repair config
+ private final Config cfg;
+ // Properties with source, hoodie client, key generator etc.
+ private TypedProperties props;
+ // Spark context
+ private final JavaSparkContext jsc;
+ private final HoodieTableMetaClient metaClient;
+ private final FileSystemBackedTableMetadata tableMetadata;
+
+ public HoodieRepairTool(JavaSparkContext jsc, Config cfg) {
+ if (cfg.propsFilePath != null) {
+ cfg.propsFilePath = FSUtils.addSchemeIfLocalPath(cfg.propsFilePath).toString();
+ }
+ this.jsc = jsc;
+ this.cfg = cfg;
+ this.props = cfg.propsFilePath == null
+ ? UtilHelpers.buildProperties(cfg.configs)
+ : readConfigFromFileSystem(jsc, cfg);
+ this.metaClient = HoodieTableMetaClient.builder()
+ .setConf(jsc.hadoopConfiguration()).setBasePath(cfg.basePath)
+ .setLoadActiveTimelineOnLoad(true)
+ .build();
+ this.tableMetadata = new FileSystemBackedTableMetadata(
+ new HoodieSparkEngineContext(jsc),
+ new SerializableConfiguration(jsc.hadoopConfiguration()),
+ cfg.basePath, cfg.assumeDatePartitioning);
+ }
+
+ public void run() {
+ Option<String> startingInstantOption = Option.ofNullable(cfg.startingInstantTime);
+ Option<String> endingInstantOption = Option.ofNullable(cfg.endingInstantTime);
+
+ if (startingInstantOption.isPresent() && endingInstantOption.isPresent()) {
+ LOG.info(String.format("Start repairing completed instants between %s and %s (inclusive)",
+ startingInstantOption.get(), endingInstantOption.get()));
+ } else if (startingInstantOption.isPresent()) {
+ LOG.info(String.format("Start repairing completed instants from %s (inclusive)",
+ startingInstantOption.get()));
+ } else if (endingInstantOption.isPresent()) {
+ LOG.info(String.format("Start repairing completed instants till %s (inclusive)",
+ endingInstantOption.get()));
+ } else {
+ LOG.info("Start repairing all completed instants");
+ }
+
+ try {
+ Mode mode = Mode.valueOf(cfg.runningMode.toUpperCase());
+ switch (mode) {
+ case REPAIR:
+ LOG.info(" ****** The repair tool is in REPAIR mode, dangling data and logs files "
Review comment:
is "****" intentional or did you plan to remove it
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieRepairTool.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.FileSystemBackedTableMetadata;
+import org.apache.hudi.table.repair.RepairUtils;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+public class HoodieRepairTool {
Review comment:
java docs.
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/repair/RepairUtils.java
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.table.repair;
+
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.model.HoodieCommitMetadata;
+import org.apache.hudi.common.model.HoodieReplaceCommitMetadata;
+import org.apache.hudi.common.model.HoodieWriteStat;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.table.timeline.HoodieInstant;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.PathFilter;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+import static org.apache.hudi.common.table.timeline.HoodieTimeline.COMMIT_ACTION;
+import static org.apache.hudi.common.table.timeline.HoodieTimeline.DELTA_COMMIT_ACTION;
+import static org.apache.hudi.common.table.timeline.HoodieTimeline.REPLACE_COMMIT_ACTION;
+
+public final class RepairUtils {
+ public static Map<String, List<String>> tagInstantsOfBaseAndLogFiles(
+ String basePath, String baseFileExtension, List<Path> allPaths) {
+ // Instant time -> Set of base and log file paths
+ Map<String, List<String>> instantToFilesMap = new HashMap<>();
+ allPaths.forEach(path -> {
+ String instantTime = path.toString().endsWith(baseFileExtension)
+ ? FSUtils.getCommitTime(path.getName()) : FSUtils.getBaseCommitTimeFromLogPath(path);
+ instantToFilesMap.computeIfAbsent(instantTime, k -> new ArrayList<>());
+ instantToFilesMap.get(instantTime).add(
+ FSUtils.getRelativePartitionPath(new Path(basePath), path));
+ });
+ return instantToFilesMap;
+ }
+
+ public static Option<Set<String>> getBaseAndLogFilePathsFromTimeline(
+ HoodieTimeline timeline, HoodieInstant instant) throws IOException {
+ if (!instant.isCompleted()) {
+ throw new HoodieException("Cannot get base and log file paths from "
+ + "instant not completed: " + instant.getTimestamp());
+ }
+
+ switch (instant.getAction()) {
+ case COMMIT_ACTION:
+ case DELTA_COMMIT_ACTION:
+ final HoodieCommitMetadata commitMetadata =
+ HoodieCommitMetadata.fromBytes(
+ timeline.getInstantDetails(instant).get(), HoodieCommitMetadata.class);
+ return Option.of(commitMetadata.getPartitionToWriteStats().values().stream().flatMap(List::stream)
+ .map(HoodieWriteStat::getPath).collect(Collectors.toSet()));
+ case REPLACE_COMMIT_ACTION:
+ final HoodieReplaceCommitMetadata replaceCommitMetadata =
+ HoodieReplaceCommitMetadata.fromBytes(
+ timeline.getInstantDetails(instant).get(), HoodieReplaceCommitMetadata.class);
+ return Option.of(replaceCommitMetadata.getPartitionToWriteStats().values().stream().flatMap(List::stream)
+ .map(HoodieWriteStat::getPath).collect(Collectors.toSet()));
+ default:
+ return Option.empty();
Review comment:
should we consider other actions like rollback and savepoint etc.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/table/timeline/HoodieArchivedTimeline.java
##########
@@ -143,15 +149,19 @@ public HoodieArchivedTimeline reload() {
}
private HoodieInstant readCommit(GenericRecord record, boolean loadDetails) {
- final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
+ final String instantTime = record.get(HoodiePartitionMetadata.COMMIT_TIME_KEY).toString();
final String action = record.get(ACTION_TYPE_KEY).toString();
+ String actionState = record.get(ACTION_STATE).toString();
if (loadDetails) {
getMetadataKey(action).map(key -> {
Object actionData = record.get(key);
- if (action.equals(HoodieTimeline.COMPACTION_ACTION)) {
- this.readCommits.put(instantTime, HoodieAvroUtils.indexedRecordToBytes((IndexedRecord)actionData));
- } else {
- this.readCommits.put(instantTime, actionData.toString().getBytes(StandardCharsets.UTF_8));
+ // Only puts instant details if the instant is completed and there is action data
+ if (HoodieInstant.State.COMPLETED.toString().equals(actionState) && actionData != null) {
Review comment:
looked at the code to ensure that archival will not move past the earlist inflight instant in timline. But just inacse someone wants to double check, leaving a comment here. if this assumption is wrong, this fix has to be reverted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991436367
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
}, {
"hash" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190)
* 432c2aff71bf918d22a3c6e81f23b11f5297d3b0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991425623
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 25a3f9b1e23db855abe6f352c391033fdef4d460 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184)
* 6b705b1a371fe8af7fb65795a1083ca13e9e7348 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187)
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991436367
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
}, {
"hash" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190)
* 432c2aff71bf918d22a3c6e81f23b11f5297d3b0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991468392
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
}, {
"hash" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4182",
"triggerID" : "347001542a65afa8dc4ce42f4b80fbedf8e8036d",
"triggerType" : "PUSH"
}, {
"hash" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4184",
"triggerID" : "25a3f9b1e23db855abe6f352c391033fdef4d460",
"triggerType" : "PUSH"
}, {
"hash" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4187",
"triggerID" : "6b705b1a371fe8af7fb65795a1083ca13e9e7348",
"triggerType" : "PUSH"
}, {
"hash" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e8c56862a65de258d657f029cd15466f7e4e41f7",
"triggerType" : "PUSH"
}, {
"hash" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4190",
"triggerID" : "ee2a811b2f3630472bf4aeb00e8e39d9384a0b7f",
"triggerType" : "PUSH"
}, {
"hash" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4194",
"triggerID" : "432c2aff71bf918d22a3c6e81f23b11f5297d3b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e8c56862a65de258d657f029cd15466f7e4e41f7 UNKNOWN
* 432c2aff71bf918d22a3c6e81f23b11f5297d3b0 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4194)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991360258
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991123083
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfa6aa245ac894cf9187ca73e741daed595cc27c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991120863
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfa6aa245ac894cf9187ca73e741daed595cc27c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991166361
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfa6aa245ac894cf9187ca73e741daed595cc27c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168)
* c5ae7055a92e92d7f01f082d364d36cead5f8569 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#issuecomment-991166361
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168",
"triggerID" : "bfa6aa245ac894cf9187ca73e741daed595cc27c",
"triggerType" : "PUSH"
}, {
"hash" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170",
"triggerID" : "c5ae7055a92e92d7f01f082d364d36cead5f8569",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bfa6aa245ac894cf9187ca73e741daed595cc27c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4168)
* c5ae7055a92e92d7f01f082d364d36cead5f8569 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=4170)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4278: [HUDI-2906] Add a repair util to clean up dangling data and log files
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4278:
URL: https://github.com/apache/hudi/pull/4278#discussion_r781510897
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodieRepairTool.java
##########
@@ -0,0 +1,295 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hudi.utilities;
+
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.SerializableConfiguration;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieArchivedTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.exception.HoodieIOException;
+import org.apache.hudi.metadata.FileSystemBackedTableMetadata;
+import org.apache.hudi.table.repair.RepairUtils;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.security.SecureRandom;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import scala.Tuple2;
+
+public class HoodieRepairTool {
Review comment:
This is resolved in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org