You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/24 23:52:58 UTC
[GitHub] [hudi] nsivabalan opened a new pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
nsivabalan opened a new pull request #4110:
URL: https://github.com/apache/hudi/pull/4110
## What is the purpose of the pull request
Listing based rollback of delta commits in MOR table, uses latest file slice to fetch list of log files to be rolledback. With lazy cleaning in multi-writer flow, rollback could happen very late. i.e compaction could have triggered by then. And so we can't use latestFileSlice for rollback. Fixed to use getLatestFileSliceOnOrBefore the commit being rolledback.
## Brief change log
- Fixed rollback utils to use getLatestFileSliceOnOrBefore the commit being rolledback instead of getLatestfileSlice while collecting files to be rolledback for delta commits in MOR.
## Verify this pull request
- Added test to
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#issuecomment-978502984
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718",
"triggerID" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eebcd6c249ec16e99685917a72a4acc6e42acef0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#issuecomment-978497479
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eebcd6c249ec16e99685917a72a4acc6e42acef0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#issuecomment-979247096
LGTM .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#issuecomment-979467181
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718",
"triggerID" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"triggerType" : "PUSH"
}, {
"hash" : "554fd911028098207297f4de171c5d119b587b7c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3764",
"triggerID" : "554fd911028098207297f4de171c5d119b587b7c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 554fd911028098207297f4de171c5d119b587b7c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3764)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#issuecomment-978732519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718",
"triggerID" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eebcd6c249ec16e99685917a72a4acc6e42acef0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#issuecomment-979442126
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718",
"triggerID" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"triggerType" : "PUSH"
}, {
"hash" : "554fd911028098207297f4de171c5d119b587b7c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "554fd911028098207297f4de171c5d119b587b7c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eebcd6c249ec16e99685917a72a4acc6e42acef0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718)
* 554fd911028098207297f4de171c5d119b587b7c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#issuecomment-979442932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718",
"triggerID" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"triggerType" : "PUSH"
}, {
"hash" : "554fd911028098207297f4de171c5d119b587b7c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3764",
"triggerID" : "554fd911028098207297f4de171c5d119b587b7c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eebcd6c249ec16e99685917a72a4acc6e42acef0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718)
* 554fd911028098207297f4de171c5d119b587b7c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3764)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#issuecomment-979442126
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718",
"triggerID" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"triggerType" : "PUSH"
}, {
"hash" : "554fd911028098207297f4de171c5d119b587b7c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "554fd911028098207297f4de171c5d119b587b7c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eebcd6c249ec16e99685917a72a4acc6e42acef0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718)
* 554fd911028098207297f4de171c5d119b587b7c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#issuecomment-978502984
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718",
"triggerID" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eebcd6c249ec16e99685917a72a4acc6e42acef0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a change in pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
codope commented on a change in pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#discussion_r757074542
##########
File path: hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/RollbackUtils.java
##########
@@ -228,8 +228,9 @@ static HoodieRollbackStat mergeRollbackStat(HoodieRollbackStat stat1, HoodieRoll
// used to write the new log files. In this case, the commit time for the log file is the compaction requested time.
// But the index (global) might store the baseCommit of the base and not the requested, hence get the
// baseCommit always by listing the file slice
- Map<String, String> fileIdToBaseCommitTimeForLogMap = table.getSliceView().getLatestFileSlices(partitionPath)
- .collect(Collectors.toMap(FileSlice::getFileId, FileSlice::getBaseInstantTime));
+ Map<String, String> fileIdToBaseCommitTimeForLogMap = table.getSliceView().getLatestFileSlicesBeforeOrOn(partitionPath, rollbackInstant.getTimestamp(),
Review comment:
Can we also add a line to the comment above? Helpful to understand while reading the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#issuecomment-978732519
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718",
"triggerID" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eebcd6c249ec16e99685917a72a4acc6e42acef0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#issuecomment-979442932
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718",
"triggerID" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"triggerType" : "PUSH"
}, {
"hash" : "554fd911028098207297f4de171c5d119b587b7c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3764",
"triggerID" : "554fd911028098207297f4de171c5d119b587b7c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eebcd6c249ec16e99685917a72a4acc6e42acef0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3718)
* 554fd911028098207297f4de171c5d119b587b7c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=3764)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #4110:
URL: https://github.com/apache/hudi/pull/4110
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4110: [HUDI-2841] Fixing lazy rollback for MOR with list based strategy
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4110:
URL: https://github.com/apache/hudi/pull/4110#issuecomment-978497479
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "eebcd6c249ec16e99685917a72a4acc6e42acef0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* eebcd6c249ec16e99685917a72a4acc6e42acef0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org