You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2025/03/31 22:33:54 UTC
[PR] [HUDI-7507] Fixing rollbacks for concurrent rollback planning [hudi]
nsivabalan opened a new pull request, #13064:
URL: https://github.com/apache/hudi/pull/13064
### Change Logs
- If there are two concurrent rollback planning by 2 concurrent writers, each could write two rollback plans to timeline. We are fixing that in this patch. Also, following up on https://github.com/apache/hudi/pull/11580, we are adding locks to rollback planning as well.
This is targetted against 0.x branch.
### Impact
Robust rollback planning even in the event of concurrent writers/planners.
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none"._
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2928705501
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
}, {
"hash" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"triggerType" : "PUSH"
}, {
"hash" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663",
"triggerID" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"triggerType" : "PUSH"
}, {
"hash" : "61a7e2e44c45554c94d79c392d93aa25803f582f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5414",
"triggerID" : "61a7e2e44c45554c94d79c392d93aa25803f582f",
"triggerType" : "PUSH"
}, {
"hash" : "31a39b972a022d918e8b41fcf5d478df4fa52ec9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "31a39b972a022d918e8b41fcf5d478df4fa52ec9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 911c9253d09b4bc1e3db2c491eb9742add2dea50 UNKNOWN
* 61a7e2e44c45554c94d79c392d93aa25803f582f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5414)
* 31a39b972a022d918e8b41fcf5d478df4fa52ec9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing rollbacks for concurrent rollback planning [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2767676323
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554)
* f20131a54eda473b388b9ed3ae446257b0a3e2ab Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2029711258
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -666,7 +670,7 @@ private void rollbackInflightInstant(HoodieInstant inflightInstant,
-> entry.getRollbackInstant().getTimestamp())
.orElseGet(HoodieActiveTimeline::createNewInstantTime);
scheduleRollback(context, commitTime, inflightInstant, false, config.shouldRollbackUsingMarkers(),
- false);
+ false, false);
Review Comment:
at least, when the check failes, just abort the cleaning instead of throwing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2067711551
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackPlanActionExecutor.java:
##########
@@ -129,6 +132,23 @@ protected Option<HoodieRollbackPlan> requestRollback(String startRollbackTime) {
}
}
+ private boolean canProceedWithRollback(HoodieInstant rollbackInstant) {
+ if (config.getWriteConcurrencyMode().supportsOptimisticConcurrencyControl()) {
+ // check for concurrent rollbacks. i.e if the commit being rolledback is already rolled back, we can bail out.
+ HoodieTableMetaClient reloadedMetaClient = HoodieTableMetaClient.reload(table.getMetaClient());
+ HoodieTimeline reloadedActiveTimeline = reloadedMetaClient.reloadActiveTimeline();
Review Comment:
just `getActiveTimeline` should be fine because the `reloadedMetaClient` is fresh new created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2067709473
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -153,6 +156,7 @@ protected HoodieTable(HoodieWriteConfig config, HoodieEngineContext context, Hoo
this.index = getIndex(config, context);
this.storageLayout = getStorageLayout(config);
this.taskContextSupplier = context.getTaskContextSupplier();
+ this.txnManager = new TransactionManager(config, metaClient.getStorage());
Review Comment:
Not sure we introduce txnManager into hoodie table? It looks like it is only used for rollback, can we move it out?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2029484223
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieTableServiceClient.java:
##########
@@ -746,7 +747,7 @@ public HoodieCleanMetadata clean(String cleanInstantTime, boolean scheduleInline
LOG.info("Cleaner started");
// proceed only if multiple clean schedules are enabled or if there are no pending cleans.
if (scheduleInline) {
- scheduleTableServiceInternal(cleanInstantTime, Option.empty(), TableServiceType.CLEAN);
+ scheduleClean(cleanInstantTime);
Review Comment:
this caller was not taking lock for schedule clean before this patch. I am standardizing all calls to scheduleClean to take locks in this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2779753785
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
}, {
"hash" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8686f3b7ea4448b300adcb47c155ac760a0f5db5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600)
* ebc19648210ba5ea57b7f2b487afc5e37729c724 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662)
* 911c9253d09b4bc1e3db2c491eb9742add2dea50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2779755861
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
}, {
"hash" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ebc19648210ba5ea57b7f2b487afc5e37729c724 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662)
* 911c9253d09b4bc1e3db2c491eb9742add2dea50 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2027985123
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -666,7 +670,7 @@ private void rollbackInflightInstant(HoodieInstant inflightInstant,
-> entry.getRollbackInstant().getTimestamp())
.orElseGet(HoodieActiveTimeline::createNewInstantTime);
scheduleRollback(context, commitTime, inflightInstant, false, config.shouldRollbackUsingMarkers(),
- false);
+ false, false);
Review Comment:
Can you elaborate again why the cleaning service needs a timestamp check, I'm afraid the timestamp check in 0.x-branch would fail the cleaning a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2928968356
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
}, {
"hash" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"triggerType" : "PUSH"
}, {
"hash" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663",
"triggerID" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"triggerType" : "PUSH"
}, {
"hash" : "61a7e2e44c45554c94d79c392d93aa25803f582f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5414",
"triggerID" : "61a7e2e44c45554c94d79c392d93aa25803f582f",
"triggerType" : "PUSH"
}, {
"hash" : "31a39b972a022d918e8b41fcf5d478df4fa52ec9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5926",
"triggerID" : "31a39b972a022d918e8b41fcf5d478df4fa52ec9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 911c9253d09b4bc1e3db2c491eb9742add2dea50 UNKNOWN
* 31a39b972a022d918e8b41fcf5d478df4fa52ec9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5926)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2942754646
<img width="1211" alt="image" src="https://github.com/user-attachments/assets/450eec11-f186-4731-b667-487b3cfb5c9d" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2026103908
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java:
##########
@@ -192,6 +198,19 @@ protected Option<HoodieCleanerPlan> requestClean(String startCleanTime) {
return option;
}
+ private void validateForLatestTimestamp(HoodieInstant cleanInstant) {
+ try {
+ if (!skipLocking) {
+ txnManager.beginTransaction(Option.of(cleanInstant), Option.empty());
+ }
+ table.validateForLatestTimestamp(cleanInstant.getTimestamp());
+ } finally {
+ if (!skipLocking) {
Review Comment:
is the lock only used for timestamp validation? so we still support concurrent cleaning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing rollbacks for concurrent rollback planning [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2774214490
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f20131a54eda473b388b9ed3ae446257b0a3e2ab Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557)
* 2be6cb1af83c1a5879e9e7caee74c96c37c6ac58 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "Tiffany0909 (via GitHub)" <gi...@apache.org>.
Tiffany0909 commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2794374559
> <!--
> Meta data
> {
> "version" : 1,
> "metaDataEntries" : [ {
> "hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
> "status" : "DELETED",
> "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
> "triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
> "triggerType" : "PUSH"
> }, {
> "hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
> "status" : "DELETED",
> "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
> "triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
> "triggerType" : "PUSH"
> }, {
> "hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
> "status" : "DELETED",
> "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
> "triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
> "triggerType" : "PUSH"
> }, {
> "hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
> "status" : "DELETED",
> "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
> "triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
> "triggerType" : "PUSH"
> }, {
> "hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
> "status" : "DELETED",
> "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
> "triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
> "triggerType" : "PUSH"
> }, {
> "hash" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
> "status" : "UNKNOWN",
> "url" : "TBD",
> "triggerID" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
> "triggerType" : "PUSH"
> }, {
> "hash" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
> "status" : "SUCCESS",
> "url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663",
> "triggerID" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
> "triggerType" : "PUSH"
> } ]
> }-->
> ## CI report:
>
> * 911c9253d09b4bc1e3db2c491eb9742add2dea50 UNKNOWN
> * cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663)
>
> <details>
> <summary>Bot commands</summary>
> @hudi-bot supports the following commands:
>
> - `@hudi-bot run azure` re-run the last Azure build
> </details>
Get me in
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2779801958
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
}, {
"hash" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"triggerType" : "PUSH"
}, {
"hash" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663",
"triggerID" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ebc19648210ba5ea57b7f2b487afc5e37729c724 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662)
* 911c9253d09b4bc1e3db2c491eb9742add2dea50 UNKNOWN
* cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2032210123
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -666,7 +670,7 @@ private void rollbackInflightInstant(HoodieInstant inflightInstant,
-> entry.getRollbackInstant().getTimestamp())
.orElseGet(HoodieActiveTimeline::createNewInstantTime);
scheduleRollback(context, commitTime, inflightInstant, false, config.shouldRollbackUsingMarkers(),
- false);
+ false, false);
Review Comment:
> we need to make sure that when clean in scheduled at a given instant time in data table there isn't a compaction plan on MDT with a greater instant time.
it sounds very restrictive and may break the Flink cleaning workflow, we may need to skip it for Flink because Flink does not enable MDT in 0.x branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing rollbacks for concurrent rollback planning [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2767573755
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2026105444
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackPlanActionExecutor.java:
##########
@@ -62,12 +66,15 @@ public BaseRollbackPlanActionExecutor(HoodieEngineContext context,
HoodieInstant instantToRollback,
boolean skipTimelinePublish,
boolean shouldRollbackUsingMarkers,
- boolean isRestore) {
+ boolean isRestore,
+ boolean skipLocking) {
super(context, config, table, instantTime);
this.instantToRollback = instantToRollback;
this.skipTimelinePublish = skipTimelinePublish;
this.shouldRollbackUsingMarkers = shouldRollbackUsingMarkers && !instantToRollback.isCompleted();
this.isRestore = isRestore;
+ this.txnManager = new TransactionManager(config, table.getStorage());
+ this.skipLocking = skipLocking;
Review Comment:
init the txn manager only if the lock is required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2774474350
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8686f3b7ea4448b300adcb47c155ac760a0f5db5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing rollbacks for concurrent rollback planning [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2767658143
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing rollbacks for concurrent rollback planning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2025891965
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackPlanActionExecutor.java:
##########
@@ -129,9 +138,36 @@ protected Option<HoodieRollbackPlan> requestRollback(String startRollbackTime) {
}
}
+ private boolean canProceedWithRollback(HoodieInstant rollbackInstant) {
+ if (config.getWriteConcurrencyMode().supportsOptimisticConcurrencyControl()) {
+ // check for concurrent rollbacks. i.e if the commit being rolledback is already rolled back, we can bail out.
+ HoodieTableMetaClient reloadedMetaClient = HoodieTableMetaClient.reload(table.getMetaClient());
Review Comment:
we could just reload the active timeline and leverage that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2026105012
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java:
##########
@@ -56,14 +57,19 @@ public class CleanPlanActionExecutor<T, I, K, O> extends BaseActionExecutor<T, I
private static final Logger LOG = LoggerFactory.getLogger(CleanPlanActionExecutor.class);
private final Option<Map<String, String>> extraMetadata;
+ private final TransactionManager txnManager;
+ private final boolean skipLocking;
public CleanPlanActionExecutor(HoodieEngineContext context,
HoodieWriteConfig config,
HoodieTable<T, I, K, O> table,
String instantTime,
- Option<Map<String, String>> extraMetadata) {
+ Option<Map<String, String>> extraMetadata,
+ boolean skipLocking) {
super(context, config, table, instantTime);
this.extraMetadata = extraMetadata;
+ this.txnManager = new TransactionManager(config, table.getStorage());
Review Comment:
init the txn manager only if the lock is required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #13064:
URL: https://github.com/apache/hudi/pull/13064
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "kbuci (via GitHub)" <gi...@apache.org>.
kbuci commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2031626900
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -666,7 +670,7 @@ private void rollbackInflightInstant(HoodieInstant inflightInstant,
-> entry.getRollbackInstant().getTimestamp())
.orElseGet(HoodieActiveTimeline::createNewInstantTime);
scheduleRollback(context, commitTime, inflightInstant, false, config.shouldRollbackUsingMarkers(),
- false);
+ false, false);
Review Comment:
Sure. Because of the scenario mentioned in https://issues.apache.org/jira/browse/HUDI-7507 in (S1) , we need to make sure that when clean in scheduled at a given instant time in data table there isn't a compaction plan on MDT with a greater instant time. Using the guard in validate timestamp API (taking a lock + refreshing timelines + ensure there are no ingestion/clustering/compaction instant times on data table with greater timestamp) will avoid this. Alternatively we could use a different validate timestamp API for clean (that just checks for compaction timestamp on MDT) but then we would be no longer using a single validate timestamp API for all operations.
If clean planning were to generate its instant time within a lock, then I think that should lessen the chance of having to abort due to a new later ingestion write starting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2032210123
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -666,7 +670,7 @@ private void rollbackInflightInstant(HoodieInstant inflightInstant,
-> entry.getRollbackInstant().getTimestamp())
.orElseGet(HoodieActiveTimeline::createNewInstantTime);
scheduleRollback(context, commitTime, inflightInstant, false, config.shouldRollbackUsingMarkers(),
- false);
+ false, false);
Review Comment:
> we need to make sure that when clean in scheduled at a given instant time in data table there isn't a compaction plan on MDT with a greater instant time.
it sounds very restrictive and may break the Flink cleaning workflow, we may need to skip it for Flink because Flink does not enable MDT in 0.x branch.
> (S1) If Job 2 is ingestion commit and Job 1 is ingestion commit that also does compaction/log compaction on MDT, then when Job 1 runs before Job 2 and can create a compaction plan for all instant times (up to ( x ) ) that doesn’t include instant time (x-1) . Later Job 2 will create instant time (x-1), but timeline will be in a corrupted state since compaction plan was supposed to include (x-1)
As for S1, how could the MDT compaction plan being generated when there are pending instants on DT timeline with smaller timestmap? Should we allow that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2873249576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
}, {
"hash" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"triggerType" : "PUSH"
}, {
"hash" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663",
"triggerID" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"triggerType" : "PUSH"
}, {
"hash" : "61a7e2e44c45554c94d79c392d93aa25803f582f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5414",
"triggerID" : "61a7e2e44c45554c94d79c392d93aa25803f582f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 911c9253d09b4bc1e3db2c491eb9742add2dea50 UNKNOWN
* 61a7e2e44c45554c94d79c392d93aa25803f582f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5414)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2774372911
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f20131a54eda473b388b9ed3ae446257b0a3e2ab Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557)
* 2be6cb1af83c1a5879e9e7caee74c96c37c6ac58 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595)
* 8686f3b7ea4448b300adcb47c155ac760a0f5db5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing rollbacks for concurrent rollback planning [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2767674813
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554)
* f20131a54eda473b388b9ed3ae446257b0a3e2ab UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2029434376
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/clean/CleanPlanActionExecutor.java:
##########
@@ -192,6 +198,19 @@ protected Option<HoodieCleanerPlan> requestClean(String startCleanTime) {
return option;
}
+ private void validateForLatestTimestamp(HoodieInstant cleanInstant) {
+ try {
+ if (!skipLocking) {
+ txnManager.beginTransaction(Option.of(cleanInstant), Option.empty());
+ }
+ table.validateForLatestTimestamp(cleanInstant.getTimestamp());
+ } finally {
+ if (!skipLocking) {
Review Comment:
timestamp validation is just to ensure the timestamp chosen for this cleaning is higher than all other timestamps generated so far (based in the timeline).
So, does not mean concurrent cleaning.
We could have had a concurrent compaction instant which got added to timeline just around the same time clean instant generation happened, but has higher timestamp compared to this clean instant. Just that the clean planning took non trivial amount of time during which, the compaction plan was added to the timeline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2779800139
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
}, {
"hash" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"triggerType" : "PUSH"
}, {
"hash" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ebc19648210ba5ea57b7f2b487afc5e37729c724 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662)
* 911c9253d09b4bc1e3db2c491eb9742add2dea50 UNKNOWN
* cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2043004359
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -666,7 +670,7 @@ private void rollbackInflightInstant(HoodieInstant inflightInstant,
-> entry.getRollbackInstant().getTimestamp())
.orElseGet(HoodieActiveTimeline::createNewInstantTime);
scheduleRollback(context, commitTime, inflightInstant, false, config.shouldRollbackUsingMarkers(),
- false);
+ false, false);
Review Comment:
Hey @danny0405 : the inner most code block where we actually do the timestamp validation, we do skip if MDT is not enabled. So, its automatically taken care. https://github.com/apache/hudi/blob/920a8af46a4d4182257cd2594ab50b8564258363/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/timeline/TimestampUtils.java#L29
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2122387129
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -607,12 +610,12 @@ public abstract Option<HoodieRestorePlan> scheduleRestore(HoodieEngineContext co
String restoreInstantTimestamp,
String savepointToRestoreTimestamp);
- public void rollbackInflightCompaction(HoodieInstant inflightInstant) {
- rollbackInflightCompaction(inflightInstant, s -> Option.empty());
+ public void rollbackInflightCompaction(HoodieInstant inflightInstant, Option<TransactionManager> txnManagerOpt) {
+ rollbackInflightCompaction(inflightInstant, s -> Option.empty(), txnManagerOpt);
}
- public void rollbackInflightLogCompaction(HoodieInstant inflightInstant) {
- rollbackInflightLogCompaction(inflightInstant, s -> Option.empty());
+ public void rollbackInflightLogCompaction(HoodieInstant inflightInstant, Option<TransactionManager> txnManagerOpt) {
+ rollbackInflightLogCompaction(inflightInstant, s -> Option.empty(), txnManagerOpt);
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2928800735
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
}, {
"hash" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"triggerType" : "PUSH"
}, {
"hash" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663",
"triggerID" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"triggerType" : "PUSH"
}, {
"hash" : "61a7e2e44c45554c94d79c392d93aa25803f582f",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5414",
"triggerID" : "61a7e2e44c45554c94d79c392d93aa25803f582f",
"triggerType" : "PUSH"
}, {
"hash" : "31a39b972a022d918e8b41fcf5d478df4fa52ec9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5926",
"triggerID" : "31a39b972a022d918e8b41fcf5d478df4fa52ec9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 911c9253d09b4bc1e3db2c491eb9742add2dea50 UNKNOWN
* 61a7e2e44c45554c94d79c392d93aa25803f582f Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5414)
* 31a39b972a022d918e8b41fcf5d478df4fa52ec9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5926)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2872951702
hey @danny0405 :
wrt to not introducing TransactionManager in HoodieTable, here are the changes it might introduce.
1. Flink: CompactionPlanOperator instantiate TransactionManager and pass it to CompactionUtil.rollbackCompaction(table). Or instantiate TransactionManager in CompactionUtil.
2. RunCompactionActionExecutor.execute() calls into preCommit which can trigger rollback. And so, this percolates to Hoodie*Table.logCompact, Hoodie*Table.compact
-> which in turn goes into Hoodie*TableServiceClient.logCompact and Hoodie*TableServiceClient.comact().
but this means, we need to change public apis in HoodieTable for compact() and logCompact(), rollback APIs to take in an Option<TransactionManager>. All of our public apis are very tightly designed. This causes divergence.
3. Flink: CompactionCommitSink calls into rollbackCompaction(). We need to introduce TransactionManager in CompactionCommitSink.
4. HoodieFlinkCompactor.compact() could call into rollbackCompaction(). Might have to add TransactionManager to HoodieFlinkCompactor.
5. BaseHoodieTableServiceClient.purgePendingClustering() can call into rollback. BaseHoodieTableServiceClient.cluster() also could call into rollback. We just need to pass in the TransactionManager.
May main concern is the change in public apis like HoodieTable.compact(), HoodieTable.logCompact(), HoodieTable.rollback*() APIs.
Attaching pic on the APIs we have in HoodieTable :
<img width="593" alt="image" src="https://github.com/user-attachments/assets/c374e817-afda-4fd4-b2a5-7dca8728bec7" />
<img width="817" alt="image" src="https://github.com/user-attachments/assets/25f69a48-bedd-48ee-905c-5a8d4bd8383c" />
For Flink specifically, I might need some help as well (whether CompactionCommitSink is the right place to introduce the TransactionManager or not).
Also, I was looking at places where we instantiate TransactionManager, looks like we don't have it clean either.
<img width="1185" alt="image" src="https://github.com/user-attachments/assets/dc4fdcaf-8c95-4b59-9078-cc5909c5b38e" />
We have many usages in ActionExecutors which are expected to be called from Hoodie*Table. So, not sure if its worth cleaning it up in this patch where we are looking to fix rollback scheduling. This could be much larger scope if we wanted to clean up the instantiations of TransactionManager to standardize it and keep it at either WriteClient or TableServiceClient.
Let me know what do you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2774375636
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2be6cb1af83c1a5879e9e7caee74c96c37c6ac58 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595)
* 8686f3b7ea4448b300adcb47c155ac760a0f5db5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2027195711
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -666,7 +670,7 @@ private void rollbackInflightInstant(HoodieInstant inflightInstant,
-> entry.getRollbackInstant().getTimestamp())
.orElseGet(HoodieActiveTimeline::createNewInstantTime);
scheduleRollback(context, commitTime, inflightInstant, false, config.shouldRollbackUsingMarkers(),
- false);
+ false, false);
Review Comment:
for rollbacks, we can do that. but for cleaning, we just need it for timestamp validation. So, it increases the locking granularity. entire planning will be under a lock :(
and esply clean planning could involve non-trivial amount of time, we are trying not to take locks for entire planning phase.
Ideally, if we standardize any planning to be under lock, all this will smoothen out. but locking is also costly. so, we don't wanna take locks just for code maintenance/structuring purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2874682679
> May main concern is the change in public apis like HoodieTable.compact(), HoodieTable.logCompact(), HoodieTable.rollback*() APIs.
The scheduling of clean and rollback should be already separeted from other write APIs I think, maybe just add the fresh new txn manager for these 3 APIs should be good already.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing rollbacks for concurrent rollback planning [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2021950437
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackPlanActionExecutor.java:
##########
@@ -129,9 +138,36 @@ protected Option<HoodieRollbackPlan> requestRollback(String startRollbackTime) {
}
}
+ private boolean canProceedWithRollback(HoodieInstant rollbackInstant) {
+ if (config.getWriteConcurrencyMode().supportsOptimisticConcurrencyControl()) {
+ // check for concurrent rollbacks. i.e if the commit being rolledback is already rolled back, we can bail out.
+ HoodieTableMetaClient reloadedMetaClient = HoodieTableMetaClient.reload(table.getMetaClient());
Review Comment:
do we need to rebuild the meta client?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing rollbacks for concurrent rollback planning [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2767575787
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2043145575
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -666,7 +670,7 @@ private void rollbackInflightInstant(HoodieInstant inflightInstant,
-> entry.getRollbackInstant().getTimestamp())
.orElseGet(HoodieActiveTimeline::createNewInstantTime);
scheduleRollback(context, commitTime, inflightInstant, false, config.shouldRollbackUsingMarkers(),
- false);
+ false, false);
Review Comment:
and I agree that we can return an empty plan and avoid failing the write if we encounter timestamp validation w/ clean scheduling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2932803752
@danny0405 : ready for review based on our sync up yesterday.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2872960914
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
}, {
"hash" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"triggerType" : "PUSH"
}, {
"hash" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663",
"triggerID" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"triggerType" : "PUSH"
}, {
"hash" : "61a7e2e44c45554c94d79c392d93aa25803f582f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "61a7e2e44c45554c94d79c392d93aa25803f582f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 911c9253d09b4bc1e3db2c491eb9742add2dea50 UNKNOWN
* cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663)
* 61a7e2e44c45554c94d79c392d93aa25803f582f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2873060432
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
}, {
"hash" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"triggerType" : "PUSH"
}, {
"hash" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663",
"triggerID" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"triggerType" : "PUSH"
}, {
"hash" : "61a7e2e44c45554c94d79c392d93aa25803f582f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5414",
"triggerID" : "61a7e2e44c45554c94d79c392d93aa25803f582f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 911c9253d09b4bc1e3db2c491eb9742add2dea50 UNKNOWN
* cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663)
* 61a7e2e44c45554c94d79c392d93aa25803f582f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=5414)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2026113209
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -666,7 +670,7 @@ private void rollbackInflightInstant(HoodieInstant inflightInstant,
-> entry.getRollbackInstant().getTimestamp())
.orElseGet(HoodieActiveTimeline::createNewInstantTime);
scheduleRollback(context, commitTime, inflightInstant, false, config.shouldRollbackUsingMarkers(),
- false);
+ false, false);
Review Comment:
Can the `skipLocking` flag be maintained outside of the hoodie table or all kinds of executors? like this:
```java
if (skipLocking) {
scheduleRollback(...)
} else {
txnManager.startTxn ...
scheduleRollback(...)
txnManager.endTxn ...
}
```
I feel like the table and executor should not care about whether the lock should be used, that would make the code cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2779874805
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
}, {
"hash" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "911c9253d09b4bc1e3db2c491eb9742add2dea50",
"triggerType" : "PUSH"
}, {
"hash" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663",
"triggerID" : "cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 911c9253d09b4bc1e3db2c491eb9742add2dea50 UNKNOWN
* cdf9466da7e1b8f8e7031c49fb5976d6b78b5c03 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4663)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2779724201
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8686f3b7ea4448b300adcb47c155ac760a0f5db5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600)
* ebc19648210ba5ea57b7f2b487afc5e37729c724 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4662)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2029451716
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackPlanActionExecutor.java:
##########
@@ -129,9 +138,36 @@ protected Option<HoodieRollbackPlan> requestRollback(String startRollbackTime) {
}
}
+ private boolean canProceedWithRollback(HoodieInstant rollbackInstant) {
+ if (config.getWriteConcurrencyMode().supportsOptimisticConcurrencyControl()) {
+ // check for concurrent rollbacks. i.e if the commit being rolledback is already rolled back, we can bail out.
+ HoodieTableMetaClient reloadedMetaClient = HoodieTableMetaClient.reload(table.getMetaClient());
Review Comment:
I have attempted a fix to make this elegant. you can check it out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2043004359
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -666,7 +670,7 @@ private void rollbackInflightInstant(HoodieInstant inflightInstant,
-> entry.getRollbackInstant().getTimestamp())
.orElseGet(HoodieActiveTimeline::createNewInstantTime);
scheduleRollback(context, commitTime, inflightInstant, false, config.shouldRollbackUsingMarkers(),
- false);
+ false, false);
Review Comment:
Hey @danny0405 : one thing to clarify is.
scheduleClean() has two diff callers prior to this patch, where one of them was taking a lock and the other was not taking a lock.
w/ this patch, we are standardizing it to take locks in both callers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing rollbacks for concurrent rollback planning [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2767767591
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f20131a54eda473b388b9ed3ae446257b0a3e2ab Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2029362936
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -666,7 +670,7 @@ private void rollbackInflightInstant(HoodieInstant inflightInstant,
-> entry.getRollbackInstant().getTimestamp())
.orElseGet(HoodieActiveTimeline::createNewInstantTime);
scheduleRollback(context, commitTime, inflightInstant, false, config.shouldRollbackUsingMarkers(),
- false);
+ false, false);
Review Comment:
hey @kbuci : Can you help clarify w/ the use-case here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2779721897
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
}, {
"hash" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ebc19648210ba5ea57b7f2b487afc5e37729c724",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 8686f3b7ea4448b300adcb47c155ac760a0f5db5 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600)
* ebc19648210ba5ea57b7f2b487afc5e37729c724 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "kbuci (via GitHub)" <gi...@apache.org>.
kbuci commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2034000901
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java:
##########
@@ -666,7 +670,7 @@ private void rollbackInflightInstant(HoodieInstant inflightInstant,
-> entry.getRollbackInstant().getTimestamp())
.orElseGet(HoodieActiveTimeline::createNewInstantTime);
scheduleRollback(context, commitTime, inflightInstant, false, config.shouldRollbackUsingMarkers(),
- false);
+ false, false);
Review Comment:
> it sounds very restrictive and may break the Flink cleaning workflow, we may need to skip it for Flink because Flink does not enable MDT in 0.x branch.
Yes good point for clean scheduling we can avoid doing validateTimestamp check if dataset has no MDT. I would prefer though that we skip if based on wether or not dataset has MDT rather than wether ingestion uses Flink engine. Since I'm not sure if there's a straightforward way for clean schedule call to infer the execution engine used by ingestion
> As for S1, how could the MDT compaction plan being generated when there are pending instants on DT timeline with smaller timestmap? Should we allow that.
Oh so in S1 the MDT compaction plan is able to be scheduled since there is no inflight instant on data table at that point in time (which is correct/expected behavior). But (without the validateTimestamp check) the other concurrent clean schedule call on data table can generate a lower timestamp, which will be the same timestamp used on the MDT write (Since an operation on data table at instant time `i` will write a corresponding deltacommit to MDT at with instant time `i`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing rollbacks for concurrent rollback planning [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #13064:
URL: https://github.com/apache/hudi/pull/13064#discussion_r2025894932
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackPlanActionExecutor.java:
##########
@@ -129,9 +138,36 @@ protected Option<HoodieRollbackPlan> requestRollback(String startRollbackTime) {
}
}
+ private boolean canProceedWithRollback(HoodieInstant rollbackInstant) {
+ if (config.getWriteConcurrencyMode().supportsOptimisticConcurrencyControl()) {
+ // check for concurrent rollbacks. i.e if the commit being rolledback is already rolled back, we can bail out.
+ HoodieTableMetaClient reloadedMetaClient = HoodieTableMetaClient.reload(table.getMetaClient());
Review Comment:
actually the reason is 2 call stacks down.
in L 152 we call
validateForLatestTimestampWithoutReload() w/ reloaded meta client as argument.
eventually this calls into
TimestampUtils.validateForLatestTimestamp
within this method, we might need to reload the timeline in most cases. Only in case of this rollback flow, we do not need to reload the timeline.
but to reload the timeline, we need an instant of "HoodieTableMetaClient" and so we could do metaClient.reloadActiveTimeline().
In other words, we can't just take in an activeTimeline as an argument to TimestampUtils.validateForLatestTimestamp
I did think about adding two Optional arguments to this method. but felt it may not be elegant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing rollbacks for concurrent rollback planning [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2774212075
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f20131a54eda473b388b9ed3ae446257b0a3e2ab Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557)
* 2be6cb1af83c1a5879e9e7caee74c96c37c6ac58 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [HUDI-7507] Fixing Rollbacks and Cleaning to acquire locks as needed [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #13064:
URL: https://github.com/apache/hudi/pull/13064#issuecomment-2774381777
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4554",
"triggerID" : "e23cbe1bd99735cc69f0f988a0d40b6ed0548a0b",
"triggerType" : "PUSH"
}, {
"hash" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4557",
"triggerID" : "f20131a54eda473b388b9ed3ae446257b0a3e2ab",
"triggerType" : "PUSH"
}, {
"hash" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595",
"triggerID" : "2be6cb1af83c1a5879e9e7caee74c96c37c6ac58",
"triggerType" : "PUSH"
}, {
"hash" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600",
"triggerID" : "8686f3b7ea4448b300adcb47c155ac760a0f5db5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2be6cb1af83c1a5879e9e7caee74c96c37c6ac58 Azure: [CANCELED](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4595)
* 8686f3b7ea4448b300adcb47c155ac760a0f5db5 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=4600)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org