You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "prashantwason (via GitHub)" <gi...@apache.org> on 2023/05/29 10:41:50 UTC
[GitHub] [hudi] prashantwason opened a new pull request, #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
prashantwason opened a new pull request, #8837:
URL: https://github.com/apache/hudi/pull/8837
[HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks
### Change Logs
New rollback logic described in details in the issue.
### Impact
Fixes issue related to rollback of various indexes in MDT.
### Risk level (write none, low medium or high below)
None
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1577414313
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 910547b944f11039c2fa4b173c54fd5a1abcf6c2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469)
* 678a19f937e2803f1d397e061b29d5693d2d8926 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1577616195
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 678a19f937e2803f1d397e061b29d5693d2d8926 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1223851326
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
Review Comment:
Why not use `lastInstant` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1213473799
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
Review Comment:
Yes, the BaseHoodieWriteClient also has this check. It is duplicated as we allow update() methods on the MDT to be called from outside the write client path. Its safer this way I suppose though duplicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1613992170
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50a92342798b808ebe521d82b99e4622eeb77ce8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207)
* c401984679350ad245c1b60d4f889b8a18715169 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1613909899
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50a92342798b808ebe521d82b99e4622eeb77ce8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1632622262
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 74451cd8ee26909fac98e88850ba352c8c415ef5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493)
* 37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1623536364
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* debb29aa63e665eed1137973fbe94eee395dc286 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1254985856
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -653,37 +608,57 @@ private static void processRestoreMetadata(HoodieActiveTimeline metadataTableTim
*/
public static Map<MetadataPartitionType, HoodieData<HoodieRecord>> convertMetadataToRecords(
HoodieEngineContext engineContext, HoodieActiveTimeline metadataTableTimeline,
- HoodieRollbackMetadata rollbackMetadata, MetadataRecordsGenerationParams recordsGenerationParams,
- String instantTime, Option<String> lastSyncTs, boolean wasSynced) {
+ HoodieTableMetaClient dataTableMetaClient,
+ HoodieRollbackMetadata rollbackMetadata, String instantTime) {
final Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
- Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
- Map<String, Map<String, Long>> partitionToAppendedFiles = new HashMap<>();
- List<HoodieRecord> filesPartitionRecords =
- convertMetadataToRollbackRecords(metadataTableTimeline, rollbackMetadata, partitionToDeletedFiles, partitionToAppendedFiles, instantTime, lastSyncTs, wasSynced);
+ List<HoodieRecord> filesPartitionRecords = convertMetadataToRollbackRecords(rollbackMetadata, instantTime);
+
+ List<HoodieRecord> reAddedRecords = getHoodieRecordsForLogFilesFromRollbackPlan(dataTableMetaClient, instantTime);
+ filesPartitionRecords.addAll(reAddedRecords);
final HoodieData<HoodieRecord> rollbackRecordsRDD = engineContext.parallelize(filesPartitionRecords, 1);
+
partitionToRecordsMap.put(MetadataPartitionType.FILES, rollbackRecordsRDD);
return partitionToRecordsMap;
}
+ private static List<HoodieRecord> getHoodieRecordsForLogFilesFromRollbackPlan(HoodieTableMetaClient dataTableMetaClient, String instantTime) {
+ /*List<HoodieInstant> instants = dataTableMetaClient.reloadActiveTimeline().filterRequestedRollbackTimeline()
+ .filter(instant -> instant.getTimestamp().equals(instantTime) && instant.isRequested()).getInstants();*/
+
+ HoodieInstant rollbackInstant = new HoodieInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.ROLLBACK_ACTION, instantTime);
+
+ // HoodieInstant rollbackInstant = instants.get(0);
+ HoodieInstant requested = HoodieTimeline.getRollbackRequestedInstant(rollbackInstant);
+ try {
+ HoodieRollbackPlan rollbackPlan = TimelineMetadataUtils.deserializeAvroMetadata(
+ dataTableMetaClient.getActiveTimeline().readRollbackInfoAsBytes(requested).get(), HoodieRollbackPlan.class);
+
+ Map<String, Map<String, Long>> partitionToLogFilesMap = new HashMap<>();
+
+ rollbackPlan.getRollbackRequests().forEach(rollbackRequest -> {
+ partitionToLogFilesMap.computeIfAbsent(rollbackRequest.getPartitionPath(), s -> new HashMap<>());
+ // fetch only log files that are expected to be RB'd in DT as part of this rollback. these log files will not be deleted, but rendered
+ // invalid once rollback is complete.
+ partitionToLogFilesMap.get(rollbackRequest.getPartitionPath()).putAll(rollbackRequest.getLogBlocksToBeDeleted());
+ });
Review Comment:
it refers to full log file name.
Sample rollback.requested
```
avrocat /tmp/hudi_trips_mor/.hoodie/20230705160035431.rollback.requested
{"instantToRollback": {"HoodieInstantInfo": {"commitTime": "20230705160008665", "action": "deltacommit"}}, "RollbackRequests": {"array": [{"partitionPath": "americas/brazil/sao_paulo", "fileId": {"string": "0c717479-0609-4dd5-8ea4-f2edcf17986c-0"}, "latestBaseInstant": {"string": "20230705155904980"}, "filesToBeDeleted": [], "logBlocksToBeDeleted": {"map": {"file:/tmp/hudi_trips_mor/americas/brazil/sao_paulo/.0c717479-0609-4dd5-8ea4-f2edcf17986c-0_20230705155904980.log.2_0-81-113": -1}}}, {"partitionPath": "asia/india/chennai", "fileId": {"string": "27df1310-75f0-4353-9834-b0736fd3f6f3-0"}, "latestBaseInstant": {"string": "20230705155904980"}, "filesToBeDeleted": [], "logBlocksToBeDeleted": {"map": {"file:/tmp/hudi_trips_mor/asia/india/chennai/.27df1310-75f0-4353-9834-b0736fd3f6f3-0_20230705155904980.log.2_2-81-115": -1}}}, {"partitionPath": "americas/united_states/san_francisco", "fileId": {"string": "ff8f1c4b-c2e8-4f7d-b7ad-7ecb385a5f1d-0"}, "latestBaseInstant": {"string": "2023
0705155904980"}, "filesToBeDeleted": [], "logBlocksToBeDeleted": {"map": {"file:/tmp/hudi_trips_mor/americas/united_states/san_francisco/.ff8f1c4b-c2e8-4f7d-b7ad-7ecb385a5f1d-0_20230705155904980.log.2_1-81-114": -1}}}]}, "version": {"int": 1}}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1247512623
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -914,26 +982,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (initialized && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records);
- closeInternal();
+ if (deltacommitsSinceCompaction.containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
Review Comment:
Or further more, we can check whether the `rollbackLogFiles` in `HoodieRollbackPartitionMetadata` is empty, if it is not, we can write another commit to MDT.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -914,26 +982,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (initialized && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records);
- closeInternal();
+ if (deltacommitsSinceCompaction.containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
Review Comment:
Or furthermore, we can check whether the `rollbackLogFiles` in `HoodieRollbackPartitionMetadata` is empty, if it is not, we can write another commit to MDT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1620722497
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e6568126aab0b098ccaac59e137e902d7a1070c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1620087569
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a5df4eae6ca374734ebe19a57bb694208a0320d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254)
* e6568126aab0b098ccaac59e137e902d7a1070c3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1634327495
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e82dedc198ec3ed2159e346dd37a3b10d493520 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543)
* de31bf2ee6551ac5212dc30d7adcba3e2873f0d5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1641681433
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677)
* 93c8dc9a2683c82d7abfd33cc26c01eee78722a1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1643017084
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
}, {
"hash" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18698",
"triggerID" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"triggerType" : "PUSH"
}, {
"hash" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18699",
"triggerID" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9b6e75df6b03096ab3e461da8c6b269644e87f10 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18699)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1266720620
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -910,26 +914,25 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
dataMetaClient.reloadActiveTimeline();
- // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
- // restore was performed. If this is not present, then the restore was performed to the earliest commit. This
- // could happen in case of bootstrap followed by rollback e.g. TestBootstrap#testFullBootstrapWithRegexModeWithUpdatesMOR.
- Option<HoodieInstant> restoreInstant = dataMetaClient.getActiveTimeline().getWriteTimeline().lastInstant();
- final String restoreToInstantTime;
- if (restoreInstant.isPresent()) {
- restoreToInstantTime = restoreInstant.get().getTimestamp();
- } else {
- restoreToInstantTime = metadataMetaClient.getActiveTimeline().filterCompletedInstants().firstInstant().get().getTimestamp();
+ // Fetch the commit to restore to (savepointed commit time)
+ HoodieInstant restoreInstant = new HoodieInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.RESTORE_ACTION, instantTime);
+ HoodieInstant requested = HoodieTimeline.getRestoreRequestedInstant(restoreInstant);
Review Comment:
This does not seem any different from the restoreInstant.
Can possibly remove this and the introduced function getRestoreRequestedInstant
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -910,26 +914,25 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
dataMetaClient.reloadActiveTimeline();
- // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
- // restore was performed. If this is not present, then the restore was performed to the earliest commit. This
- // could happen in case of bootstrap followed by rollback e.g. TestBootstrap#testFullBootstrapWithRegexModeWithUpdatesMOR.
- Option<HoodieInstant> restoreInstant = dataMetaClient.getActiveTimeline().getWriteTimeline().lastInstant();
- final String restoreToInstantTime;
- if (restoreInstant.isPresent()) {
- restoreToInstantTime = restoreInstant.get().getTimestamp();
- } else {
- restoreToInstantTime = metadataMetaClient.getActiveTimeline().filterCompletedInstants().firstInstant().get().getTimestamp();
+ // Fetch the commit to restore to (savepointed commit time)
+ HoodieInstant restoreInstant = new HoodieInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.RESTORE_ACTION, instantTime);
+ HoodieInstant requested = HoodieTimeline.getRestoreRequestedInstant(restoreInstant);
+ HoodieRestorePlan restorePlan = null;
+ try {
+ restorePlan = TimelineMetadataUtils.deserializeAvroMetadata(
+ dataMetaClient.getActiveTimeline().readRestoreInfoAsBytes(requested).get(), HoodieRestorePlan.class);
+ } catch (IOException e) {
+ throw new HoodieIOException("Deserialization of rollback plan failed ", e);
Review Comment:
Good to have the restoreInstant in this log so can debug which plan is missing/corrupted.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -910,26 +914,25 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
dataMetaClient.reloadActiveTimeline();
- // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
- // restore was performed. If this is not present, then the restore was performed to the earliest commit. This
- // could happen in case of bootstrap followed by rollback e.g. TestBootstrap#testFullBootstrapWithRegexModeWithUpdatesMOR.
- Option<HoodieInstant> restoreInstant = dataMetaClient.getActiveTimeline().getWriteTimeline().lastInstant();
- final String restoreToInstantTime;
- if (restoreInstant.isPresent()) {
- restoreToInstantTime = restoreInstant.get().getTimestamp();
- } else {
- restoreToInstantTime = metadataMetaClient.getActiveTimeline().filterCompletedInstants().firstInstant().get().getTimestamp();
+ // Fetch the commit to restore to (savepointed commit time)
+ HoodieInstant restoreInstant = new HoodieInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.RESTORE_ACTION, instantTime);
+ HoodieInstant requested = HoodieTimeline.getRestoreRequestedInstant(restoreInstant);
+ HoodieRestorePlan restorePlan = null;
+ try {
+ restorePlan = TimelineMetadataUtils.deserializeAvroMetadata(
+ dataMetaClient.getActiveTimeline().readRestoreInfoAsBytes(requested).get(), HoodieRestorePlan.class);
+ } catch (IOException e) {
+ throw new HoodieIOException("Deserialization of rollback plan failed ", e);
}
-
- // We cannot restore to before the oldest compaction on MDT as we don't have the base files before that time.
- Option<HoodieInstant> oldestCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().firstInstant();
- if (oldestCompaction.isPresent()) {
- if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(restoreToInstantTime, oldestCompaction.get().getTimestamp())) {
- String msg = String.format("Cannot restore MDT to %s because it is before the oldest compaction at %s", restoreToInstantTime,
- oldestCompaction.get().getTimestamp()) + ". Please delete MDT and restore again";
- LOG.error(msg);
- throw new HoodieMetadataException(msg);
- }
+ final String restoreToInstantTime = restorePlan.getSavepointToRestoreTimestamp();
+
+ // fetch the earliest commit to retain and ensure the base file prior to the time to restore is present
+ List<HoodieFileGroup> filesGroups = metadata.getMetadataFileSystemView().getAllFileGroups(MetadataPartitionType.FILES.getPartitionPath()).collect(Collectors.toList());
+ boolean canRestore = filesGroups.get(0).getAllFileSlices().map(fileSlice -> fileSlice.getBaseInstantTime()).anyMatch(
+ instantTime1 -> HoodieTimeline.compareTimestamps(instantTime1, LESSER_THAN_OR_EQUALS, restoreToInstantTime));
+ if (!canRestore) {
+ throw new HoodieMetadataException("Can't restore since there is no base file in MDT lesser than the commit to restore to. "
Review Comment:
Maybe add the restoreToInstantTime to this log for help in debugging.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -910,26 +914,25 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
dataMetaClient.reloadActiveTimeline();
- // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
- // restore was performed. If this is not present, then the restore was performed to the earliest commit. This
- // could happen in case of bootstrap followed by rollback e.g. TestBootstrap#testFullBootstrapWithRegexModeWithUpdatesMOR.
- Option<HoodieInstant> restoreInstant = dataMetaClient.getActiveTimeline().getWriteTimeline().lastInstant();
- final String restoreToInstantTime;
- if (restoreInstant.isPresent()) {
- restoreToInstantTime = restoreInstant.get().getTimestamp();
- } else {
- restoreToInstantTime = metadataMetaClient.getActiveTimeline().filterCompletedInstants().firstInstant().get().getTimestamp();
+ // Fetch the commit to restore to (savepointed commit time)
+ HoodieInstant restoreInstant = new HoodieInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.RESTORE_ACTION, instantTime);
+ HoodieInstant requested = HoodieTimeline.getRestoreRequestedInstant(restoreInstant);
+ HoodieRestorePlan restorePlan = null;
+ try {
+ restorePlan = TimelineMetadataUtils.deserializeAvroMetadata(
+ dataMetaClient.getActiveTimeline().readRestoreInfoAsBytes(requested).get(), HoodieRestorePlan.class);
+ } catch (IOException e) {
+ throw new HoodieIOException("Deserialization of rollback plan failed ", e);
}
-
- // We cannot restore to before the oldest compaction on MDT as we don't have the base files before that time.
- Option<HoodieInstant> oldestCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().firstInstant();
- if (oldestCompaction.isPresent()) {
- if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(restoreToInstantTime, oldestCompaction.get().getTimestamp())) {
- String msg = String.format("Cannot restore MDT to %s because it is before the oldest compaction at %s", restoreToInstantTime,
- oldestCompaction.get().getTimestamp()) + ". Please delete MDT and restore again";
- LOG.error(msg);
- throw new HoodieMetadataException(msg);
- }
+ final String restoreToInstantTime = restorePlan.getSavepointToRestoreTimestamp();
+
+ // fetch the earliest commit to retain and ensure the base file prior to the time to restore is present
+ List<HoodieFileGroup> filesGroups = metadata.getMetadataFileSystemView().getAllFileGroups(MetadataPartitionType.FILES.getPartitionPath()).collect(Collectors.toList());
+ boolean canRestore = filesGroups.get(0).getAllFileSlices().map(fileSlice -> fileSlice.getBaseInstantTime()).anyMatch(
Review Comment:
Can we implement this check without get(0)?
get(0) assumes that files partition has a single fileGroup. If we ever move to more than 1 partitions for file group, it would be hard to remember to change this too and this would be a bug. So best to not use get(0) here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #8837:
URL: https://github.com/apache/hudi/pull/8837
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1245802921
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
+ if (lastCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(restoreToInstantTime, lastCompaction.get().getTimestamp())) {
+ String msg = String.format("Cannot restore MDT to %s because it is older than latest compaction at %s", restoreToInstantTime,
+ lastCompaction.get().getTimestamp()) + ". Please delete MDT and restore again";
+ LOG.error(msg);
+ throw new HoodieMetadataException(msg);
+ }
+ }
+
+ // Restore requires the existing pipelines to be shutdown. So we can safely scan the dataset to find the current
+ // list of files in the filesystem.
+ List<DirectoryInfo> dirInfoList = listAllPartitions(dataMetaClient);
+ Map<String, DirectoryInfo> dirInfoMap = dirInfoList.stream().collect(Collectors.toMap(DirectoryInfo::getRelativePath, Function.identity()));
+ dirInfoList.clear();
+
+ LOG.info("Restoring MDT to " + restoreToInstantTime + " at " + instantTime);
+ getWriteClient().restoreToInstant(restoreToInstantTime, false);
+
+ // At this point we have also reverted the cleans which have occurred after the restoreToInstantTime. Hence, a sync
Review Comment:
TestHoodieBackedMetadata#testTableOperationsWithRestore
Clean is performed with timestamp 0000009
After the clean, we restore to 0000006 which is older than clean timestamp.
So the above test should test this scenario.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1245808678
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -669,32 +669,51 @@ public void restoreToSavepoint() {
* @param savepointTime Savepoint time to rollback to
*/
public void restoreToSavepoint(String savepointTime) {
- boolean initialMetadataTableIfNecessary = config.isMetadataTableEnabled();
- if (initialMetadataTableIfNecessary) {
+ boolean initializeMetadataTableIfNecessary = config.isMetadataTableEnabled();
+ if (initializeMetadataTableIfNecessary) {
try {
- // Delete metadata table directly when users trigger savepoint rollback if mdt existed and beforeTimelineStarts
+ // Delete metadata table directly when users trigger savepoint rollback if mdt existed and if the savePointTime is beforeTimelineStarts
+ // or before the oldest compaction on MDT.
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
String metadataTableBasePathStr = HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath());
HoodieTableMetaClient mdtClient = HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(metadataTableBasePathStr).build();
- // Same as HoodieTableMetadataUtil#processRollbackMetadata
+ Option<HoodieInstant> latestMdtCompaction = mdtClient.getCommitTimeline().filterCompletedInstants().lastInstant();
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1578067101
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 678a19f937e2803f1d397e061b29d5693d2d8926 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613)
* cd4da7877844008950cba2caa5be10b1a81f2795 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1248215039
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java:
##########
@@ -1871,7 +1865,11 @@ private void testTableOperationsImpl(HoodieSparkEngineContext engineContext, Hoo
validateMetadata(client);
// Restore
- client.restoreToInstant("20210101000600000", writeConfig.isMetadataTableEnabled());
+ if (metaClient.getTableType() == COPY_ON_WRITE) {
+ assertThrows(HoodieRestoreException.class, () -> client.restoreToInstant("20210101000600000", writeConfig.isMetadataTableEnabled()));
Review Comment:
@prashantwason : hey, can you help clarify, why do we expect this to fail just for COW table and not MOR ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1245811085
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -669,32 +669,51 @@ public void restoreToSavepoint() {
* @param savepointTime Savepoint time to rollback to
*/
public void restoreToSavepoint(String savepointTime) {
- boolean initialMetadataTableIfNecessary = config.isMetadataTableEnabled();
- if (initialMetadataTableIfNecessary) {
+ boolean initializeMetadataTableIfNecessary = config.isMetadataTableEnabled();
+ if (initializeMetadataTableIfNecessary) {
try {
- // Delete metadata table directly when users trigger savepoint rollback if mdt existed and beforeTimelineStarts
+ // Delete metadata table directly when users trigger savepoint rollback if mdt existed and if the savePointTime is beforeTimelineStarts
+ // or before the oldest compaction on MDT.
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
String metadataTableBasePathStr = HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath());
HoodieTableMetaClient mdtClient = HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(metadataTableBasePathStr).build();
- // Same as HoodieTableMetadataUtil#processRollbackMetadata
+ Option<HoodieInstant> latestMdtCompaction = mdtClient.getCommitTimeline().filterCompletedInstants().lastInstant();
+ boolean deleteMDT = false;
+ if (latestMdtCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(savepointTime, latestMdtCompaction.get().getTimestamp())) {
+ LOG.warn(String.format("Deleting MDT during restore to %s as the savepoint is older than oldest compaction %s on MDT",
+ savepointTime, latestMdtCompaction.get().getTimestamp()));
+ deleteMDT = true;
+ }
+ }
+
HoodieInstant syncedInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, savepointTime);
// The instant required to sync rollback to MDT has been archived and the mdt syncing will be failed
// So that we need to delete the whole MDT here.
if (mdtClient.getCommitsTimeline().isBeforeTimelineStarts(syncedInstant.getTimestamp())) {
+ LOG.warn(String.format("Deleting MDT during restore to %s as the savepoint is older than the MDT timeline %s",
+ savepointTime, mdtClient.getCommitsTimeline().firstInstant().get().getTimestamp()));
+ deleteMDT = true;
+ }
+
+ if (deleteMDT) {
+ // TODO: this should use the correct API to delete from HoodieTableMetadataUtil so hoodie.properties is also updated
+ // To be fixed after HUDI-6200
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1616809728
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a5df4eae6ca374734ebe19a57bb694208a0320d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1643162312
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
}, {
"hash" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18698",
"triggerID" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"triggerType" : "PUSH"
}, {
"hash" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18699",
"triggerID" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"triggerType" : "PUSH"
}, {
"hash" : "d121977ee5e8e7785acf8a0ef82563b971b5ff27",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d121977ee5e8e7785acf8a0ef82563b971b5ff27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9b6e75df6b03096ab3e461da8c6b269644e87f10 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18699)
* d121977ee5e8e7785acf8a0ef82563b971b5ff27 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1631503794
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 74451cd8ee26909fac98e88850ba352c8c415ef5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1266694440
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -1019,14 +1016,23 @@ public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime)
}
}
+ // lets apply a delta commit with DT's rb instant containing following records:
+ // a. any log files as part of RB commit metadata that was added
+ // b. log files added by the commit in DT being rolled back. By rolled back, we mean, a rollback block will be added and does not mean it will be deleted.
+ // both above list should only be added to FILES partition.
+
+ String rollbackInstantTime = createRollbackTimestamp(instantTime);
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
+ dataMetaClient, rollbackMetadata, rollbackInstantTime));
+
if (deltacommitsSinceCompaction.containsInstant(deltaCommitInstant)) {
LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
- if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ if (!getWriteClient().rollback(commitInstantTime, rollbackInstantTime)) {
throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
}
} else {
LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
- commitInstantTime, instantTime));
+ commitInstantTime, rollbackInstantTime));
Review Comment:
I think this should still be instantTime for the log message to make sense.
Ignoring rollback of instant %s at %s. The last %s is supposed to be the instantTime at which the rollback happened.
rollbackInstantTime is the "created" instantTime for MDT and not the instantTime on the DT.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -653,37 +608,57 @@ private static void processRestoreMetadata(HoodieActiveTimeline metadataTableTim
*/
public static Map<MetadataPartitionType, HoodieData<HoodieRecord>> convertMetadataToRecords(
HoodieEngineContext engineContext, HoodieActiveTimeline metadataTableTimeline,
- HoodieRollbackMetadata rollbackMetadata, MetadataRecordsGenerationParams recordsGenerationParams,
- String instantTime, Option<String> lastSyncTs, boolean wasSynced) {
+ HoodieTableMetaClient dataTableMetaClient,
+ HoodieRollbackMetadata rollbackMetadata, String instantTime) {
final Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
- Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
- Map<String, Map<String, Long>> partitionToAppendedFiles = new HashMap<>();
- List<HoodieRecord> filesPartitionRecords =
- convertMetadataToRollbackRecords(metadataTableTimeline, rollbackMetadata, partitionToDeletedFiles, partitionToAppendedFiles, instantTime, lastSyncTs, wasSynced);
+ List<HoodieRecord> filesPartitionRecords = convertMetadataToRollbackRecords(rollbackMetadata, instantTime);
+
+ List<HoodieRecord> reAddedRecords = getHoodieRecordsForLogFilesFromRollbackPlan(dataTableMetaClient, instantTime);
+ filesPartitionRecords.addAll(reAddedRecords);
final HoodieData<HoodieRecord> rollbackRecordsRDD = engineContext.parallelize(filesPartitionRecords, 1);
+
partitionToRecordsMap.put(MetadataPartitionType.FILES, rollbackRecordsRDD);
return partitionToRecordsMap;
}
+ private static List<HoodieRecord> getHoodieRecordsForLogFilesFromRollbackPlan(HoodieTableMetaClient dataTableMetaClient, String instantTime) {
+ /*List<HoodieInstant> instants = dataTableMetaClient.reloadActiveTimeline().filterRequestedRollbackTimeline()
+ .filter(instant -> instant.getTimestamp().equals(instantTime) && instant.isRequested()).getInstants();*/
+
+ HoodieInstant rollbackInstant = new HoodieInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.ROLLBACK_ACTION, instantTime);
+
+ // HoodieInstant rollbackInstant = instants.get(0);
+ HoodieInstant requested = HoodieTimeline.getRollbackRequestedInstant(rollbackInstant);
+ try {
+ HoodieRollbackPlan rollbackPlan = TimelineMetadataUtils.deserializeAvroMetadata(
+ dataTableMetaClient.getActiveTimeline().readRollbackInfoAsBytes(requested).get(), HoodieRollbackPlan.class);
+
+ Map<String, Map<String, Long>> partitionToLogFilesMap = new HashMap<>();
+
+ rollbackPlan.getRollbackRequests().forEach(rollbackRequest -> {
+ partitionToLogFilesMap.computeIfAbsent(rollbackRequest.getPartitionPath(), s -> new HashMap<>());
+ // fetch only log files that are expected to be RB'd in DT as part of this rollback. these log files will not be deleted, but rendered
+ // invalid once rollback is complete.
+ partitionToLogFilesMap.get(rollbackRequest.getPartitionPath()).putAll(rollbackRequest.getLogBlocksToBeDeleted());
Review Comment:
can save an additional get() call here by chaining the putAll to the output of computeIfAbsent
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -653,37 +608,57 @@ private static void processRestoreMetadata(HoodieActiveTimeline metadataTableTim
*/
public static Map<MetadataPartitionType, HoodieData<HoodieRecord>> convertMetadataToRecords(
HoodieEngineContext engineContext, HoodieActiveTimeline metadataTableTimeline,
- HoodieRollbackMetadata rollbackMetadata, MetadataRecordsGenerationParams recordsGenerationParams,
- String instantTime, Option<String> lastSyncTs, boolean wasSynced) {
+ HoodieTableMetaClient dataTableMetaClient,
+ HoodieRollbackMetadata rollbackMetadata, String instantTime) {
final Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
- Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
- Map<String, Map<String, Long>> partitionToAppendedFiles = new HashMap<>();
- List<HoodieRecord> filesPartitionRecords =
- convertMetadataToRollbackRecords(metadataTableTimeline, rollbackMetadata, partitionToDeletedFiles, partitionToAppendedFiles, instantTime, lastSyncTs, wasSynced);
+ List<HoodieRecord> filesPartitionRecords = convertMetadataToRollbackRecords(rollbackMetadata, instantTime);
+
+ List<HoodieRecord> reAddedRecords = getHoodieRecordsForLogFilesFromRollbackPlan(dataTableMetaClient, instantTime);
+ filesPartitionRecords.addAll(reAddedRecords);
final HoodieData<HoodieRecord> rollbackRecordsRDD = engineContext.parallelize(filesPartitionRecords, 1);
+
partitionToRecordsMap.put(MetadataPartitionType.FILES, rollbackRecordsRDD);
return partitionToRecordsMap;
Review Comment:
Collections.singletonMap?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1266731173
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -742,8 +750,10 @@ private static List<HoodieRecord> convertFilesToFilesPartitionRecords(Map<String
partitionToAppendedFiles.forEach((partitionName, appendedFileMap) -> {
final String partition = getPartitionIdentifier(partitionName);
fileChangeCount[1] += appendedFileMap.size();
+ Map<String, Long> appendedFiles = new HashMap<>();
+ appendedFileMap.forEach((k,v) -> appendedFiles.put(k, 1L));
Review Comment:
This is setting the file size to 1. Why is this required?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1641371072
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f9aa542076faa188839bc55b43dd7f22ec32b62 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580)
* fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1243081122
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records, false);
- closeInternal();
+ if (deltaCommitsInfo.get().getKey().containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
+ rollbackMetadata, getRecordsGenerationParams(), instantTime,
+ metadata.getSyncedInstantTime(), true), false);
Review Comment:
just wanted to double confirm. in the list of valid instants we populate while reading MDT using Log Record Reader, we do include rollback instants from DT right? How this might pan out, if a async compaction from DT is rolled back multiple times and then finally it gets committed?
```
public static Set<String> getValidInstantTimestamps(HoodieTableMetaClient dataMetaClient,
HoodieTableMetaClient metadataMetaClient) {
// Only those log files which have a corresponding completed instant on the dataset should be read
// This is because the metadata table is updated before the dataset instants are committed.
HoodieActiveTimeline datasetTimeline = dataMetaClient.getActiveTimeline();
Set<String> validInstantTimestamps = datasetTimeline.filterCompletedInstants().getInstantsAsStream()
.map(HoodieInstant::getTimestamp).collect(Collectors.toSet());
// We should also add completed indexing delta commits in the metadata table, as they do not
// have corresponding completed instant in the data table
validInstantTimestamps.addAll(
metadataMetaClient.getActiveTimeline()
.filter(instant -> instant.isCompleted()
&& (isIndexingCommit(instant.getTimestamp()) || isLogCompactionInstant(instant)))
.getInstantsAsStream()
.map(HoodieInstant::getTimestamp)
.collect(Collectors.toList()));
// For any rollbacks and restores, we cannot neglect the instants that they are rolling back.
// The rollback instant should be more recent than the start of the timeline for it to have rolled back any
// instant which we have a log block for.
final String earliestInstantTime = validInstantTimestamps.isEmpty() ? SOLO_COMMIT_TIMESTAMP : Collections.min(validInstantTimestamps);
datasetTimeline.getRollbackAndRestoreTimeline().filterCompletedInstants().getInstantsAsStream()
.filter(instant -> HoodieTimeline.compareTimestamps(instant.getTimestamp(), HoodieTimeline.GREATER_THAN, earliestInstantTime))
.forEach(instant -> {
validInstantTimestamps.addAll(getRollbackedCommits(instant, datasetTimeline));
});
// SOLO_COMMIT_TIMESTAMP is used during bootstrap so it is a valid timestamp
validInstantTimestamps.add(createIndexInitTimestamp(SOLO_COMMIT_TIMESTAMP, PARTITION_INITIALIZATION_TIME_SUFFIX));
return validInstantTimestamps;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1218524723
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
+ if (lastCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(restoreToInstantTime, lastCompaction.get().getTimestamp())) {
+ String msg = String.format("Cannot restore MDT to %s because it is older than latest compaction at %s", restoreToInstantTime,
+ lastCompaction.get().getTimestamp()) + ". Please delete MDT and restore again";
+ LOG.error(msg);
+ throw new HoodieMetadataException(msg);
+ }
+ }
+
+ // Restore requires the existing pipelines to be shutdown. So we can safely scan the dataset to find the current
+ // list of files in the filesystem.
+ List<DirectoryInfo> dirInfoList = listAllPartitions(dataMetaClient);
+ Map<String, DirectoryInfo> dirInfoMap = dirInfoList.stream().collect(Collectors.toMap(DirectoryInfo::getRelativePath, Function.identity()));
+ dirInfoList.clear();
+
+ LOG.info("Restoring MDT to " + restoreToInstantTime + " at " + instantTime);
+ getWriteClient().restoreToInstant(restoreToInstantTime, false);
+
+ // At this point we have also reverted the cleans which have occurred after the restoreToInstantTime. Hence, a sync
Review Comment:
TestHoodieBackedMetadata#testTableOperationsWithRestore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1631268632
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e9fb2f74b5333e6f8f60de11ea6adce497252648 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436)
* 74451cd8ee26909fac98e88850ba352c8c415ef5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1259240712
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -624,16 +624,11 @@ public static Map<MetadataPartitionType, HoodieData<HoodieRecord>> convertMetada
}
private static List<HoodieRecord> getHoodieRecordsForLogFilesFromRollbackPlan(HoodieTableMetaClient dataTableMetaClient, String instantTime) {
- /*List<HoodieInstant> instants = dataTableMetaClient.reloadActiveTimeline().filterRequestedRollbackTimeline()
- .filter(instant -> instant.getTimestamp().equals(instantTime) && instant.isRequested()).getInstants();*/
-
HoodieInstant rollbackInstant = new HoodieInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.ROLLBACK_ACTION, instantTime);
-
- // HoodieInstant rollbackInstant = instants.get(0);
HoodieInstant requested = HoodieTimeline.getRollbackRequestedInstant(rollbackInstant);
try {
HoodieRollbackPlan rollbackPlan = TimelineMetadataUtils.deserializeAvroMetadata(
- dataTableMetaClient.getActiveTimeline().readRollbackInfoAsBytes(requested).get(), HoodieRollbackPlan.class);
+ dataTableMetaClient.reloadActiveTimeline().readRollbackInfoAsBytes(requested).get(), HoodieRollbackPlan.class);
Review Comment:
Why is there a need to reload active timeline again? The `requested` instant is part of the rollback instant passed by the caller and data table metaclient is also passed by the caller. I think there should not be a need to reload.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1643175610
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
}, {
"hash" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18698",
"triggerID" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"triggerType" : "PUSH"
}, {
"hash" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18699",
"triggerID" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"triggerType" : "PUSH"
}, {
"hash" : "d121977ee5e8e7785acf8a0ef82563b971b5ff27",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18711",
"triggerID" : "d121977ee5e8e7785acf8a0ef82563b971b5ff27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9b6e75df6b03096ab3e461da8c6b269644e87f10 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18699)
* d121977ee5e8e7785acf8a0ef82563b971b5ff27 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18711)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1616746845
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3e22656f66687bb920ec82e6764bf083985df09c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241)
* 7a5df4eae6ca374734ebe19a57bb694208a0320d UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1242961967
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records, false);
- closeInternal();
+ if (deltaCommitsInfo.get().getKey().containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
+ rollbackMetadata, getRecordsGenerationParams(), instantTime,
+ metadata.getSyncedInstantTime(), true), false);
Review Comment:
I get it.
for MOR data table, rollback will add a new log file in DT. And so we need this to track adding the new file.
But can we optimize this so that this gets triggered only for MOR table or only when there are files to be added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1243200575
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
Review Comment:
> Its safer this way I suppose though duplicated
The code is hard to maintain, at least we abstract some utilities for these duplicate checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1243195507
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -669,32 +669,51 @@ public void restoreToSavepoint() {
* @param savepointTime Savepoint time to rollback to
*/
public void restoreToSavepoint(String savepointTime) {
- boolean initialMetadataTableIfNecessary = config.isMetadataTableEnabled();
- if (initialMetadataTableIfNecessary) {
+ boolean initializeMetadataTableIfNecessary = config.isMetadataTableEnabled();
+ if (initializeMetadataTableIfNecessary) {
try {
- // Delete metadata table directly when users trigger savepoint rollback if mdt existed and beforeTimelineStarts
+ // Delete metadata table directly when users trigger savepoint rollback if mdt existed and if the savePointTime is beforeTimelineStarts
+ // or before the oldest compaction on MDT.
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
String metadataTableBasePathStr = HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath());
HoodieTableMetaClient mdtClient = HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(metadataTableBasePathStr).build();
- // Same as HoodieTableMetadataUtil#processRollbackMetadata
+ Option<HoodieInstant> latestMdtCompaction = mdtClient.getCommitTimeline().filterCompletedInstants().lastInstant();
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1628349807
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e9fb2f74b5333e6f8f60de11ea6adce497252648 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1254630576
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -653,37 +608,57 @@ private static void processRestoreMetadata(HoodieActiveTimeline metadataTableTim
*/
public static Map<MetadataPartitionType, HoodieData<HoodieRecord>> convertMetadataToRecords(
HoodieEngineContext engineContext, HoodieActiveTimeline metadataTableTimeline,
- HoodieRollbackMetadata rollbackMetadata, MetadataRecordsGenerationParams recordsGenerationParams,
- String instantTime, Option<String> lastSyncTs, boolean wasSynced) {
+ HoodieTableMetaClient dataTableMetaClient,
+ HoodieRollbackMetadata rollbackMetadata, String instantTime) {
final Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
- Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
- Map<String, Map<String, Long>> partitionToAppendedFiles = new HashMap<>();
- List<HoodieRecord> filesPartitionRecords =
- convertMetadataToRollbackRecords(metadataTableTimeline, rollbackMetadata, partitionToDeletedFiles, partitionToAppendedFiles, instantTime, lastSyncTs, wasSynced);
+ List<HoodieRecord> filesPartitionRecords = convertMetadataToRollbackRecords(rollbackMetadata, instantTime);
+
+ List<HoodieRecord> reAddedRecords = getHoodieRecordsForLogFilesFromRollbackPlan(dataTableMetaClient, instantTime);
+ filesPartitionRecords.addAll(reAddedRecords);
final HoodieData<HoodieRecord> rollbackRecordsRDD = engineContext.parallelize(filesPartitionRecords, 1);
+
partitionToRecordsMap.put(MetadataPartitionType.FILES, rollbackRecordsRDD);
return partitionToRecordsMap;
}
+ private static List<HoodieRecord> getHoodieRecordsForLogFilesFromRollbackPlan(HoodieTableMetaClient dataTableMetaClient, String instantTime) {
+ /*List<HoodieInstant> instants = dataTableMetaClient.reloadActiveTimeline().filterRequestedRollbackTimeline()
+ .filter(instant -> instant.getTimestamp().equals(instantTime) && instant.isRequested()).getInstants();*/
+
+ HoodieInstant rollbackInstant = new HoodieInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.ROLLBACK_ACTION, instantTime);
+
+ // HoodieInstant rollbackInstant = instants.get(0);
+ HoodieInstant requested = HoodieTimeline.getRollbackRequestedInstant(rollbackInstant);
+ try {
+ HoodieRollbackPlan rollbackPlan = TimelineMetadataUtils.deserializeAvroMetadata(
+ dataTableMetaClient.getActiveTimeline().readRollbackInfoAsBytes(requested).get(), HoodieRollbackPlan.class);
+
+ Map<String, Map<String, Long>> partitionToLogFilesMap = new HashMap<>();
+
+ rollbackPlan.getRollbackRequests().forEach(rollbackRequest -> {
+ partitionToLogFilesMap.computeIfAbsent(rollbackRequest.getPartitionPath(), s -> new HashMap<>());
+ // fetch only log files that are expected to be RB'd in DT as part of this rollback. these log files will not be deleted, but rendered
+ // invalid once rollback is complete.
+ partitionToLogFilesMap.get(rollbackRequest.getPartitionPath()).putAll(rollbackRequest.getLogBlocksToBeDeleted());
+ });
Review Comment:
Can you confirm the key of `rollbackRequest.getLogBlocksToBeDeleted()` is file group id or is it the log filename?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -991,10 +992,6 @@ public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime)
// Find the deltacommits since the last compaction
Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
- if (!deltaCommitsInfo.isPresent() || deltaCommitsInfo.get().getKey().empty()) {
Review Comment:
why remove this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1245993652
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -900,10 +903,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()));
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .lastInstant().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> oldestCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().firstInstant();
+ if (oldestCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(restoreToInstantTime, oldestCompaction.get().getTimestamp())) {
+ String msg = String.format("Cannot restore MDT to %s because it is before the oldest compaction at %s", restoreToInstantTime,
+ oldestCompaction.get().getTimestamp()) + ". Please delete MDT and restore again";
+ LOG.error(msg);
+ throw new HoodieMetadataException(msg);
+ }
+ }
+
+ // Restore requires the existing pipelines to be shutdown. So we can safely scan the dataset to find the current
+ // list of files in the filesystem.
+ List<DirectoryInfo> dirInfoList = listAllPartitionsFromFilesystem(instantTime);
+ Map<String, DirectoryInfo> dirInfoMap = dirInfoList.stream().collect(Collectors.toMap(DirectoryInfo::getRelativePath, Function.identity()));
+ dirInfoList.clear();
+
+ LOG.info("Restoring MDT to " + restoreToInstantTime + " at " + instantTime);
+ getWriteClient().restoreToInstant(restoreToInstantTime, false);
+
+ // At this point we have also reverted the cleans which have occurred after the restoreToInstantTime. Hence, a sync
+ // is required to bring back those cleans.
+ try {
+ initMetadataReader();
+ HoodieCleanMetadata cleanMetadata = new HoodieCleanMetadata();
+ Map<String, HoodieCleanPartitionMetadata> partitionMetadata = new HashMap<>();
+ for (String partition : metadata.fetchAllPartitionPaths()) {
+ FileStatus[] metadataFiles = metadata.getAllFilesInPartition(new Path(dataWriteConfig.getBasePath(), partition));
+ if (!dirInfoMap.containsKey(partition)) {
+ // Entire partition has been deleted
+ List<String> filePaths = Arrays.stream(metadataFiles).map(f -> f.getPath().getName()).collect(Collectors.toList());
+ HoodieCleanPartitionMetadata cleanPartitionMetadata = new HoodieCleanPartitionMetadata(partition, "", filePaths, filePaths,
+ Collections.emptyList(), true);
+ partitionMetadata.put(partition, cleanPartitionMetadata);
+ } else {
+ // Some files cleaned in the partition
+ Map<String, Long> fsFiles = dirInfoMap.get(partition).getFileNameToSizeMap();
+ List<String> filesDeleted = Arrays.stream(metadataFiles).map(f -> f.getPath().getName())
+ .filter(n -> !fsFiles.containsKey(n)).collect(Collectors.toList());
+ if (!filesDeleted.isEmpty()) {
+ LOG.info("Found deleted files in partition " + partition + ": " + filesDeleted);
+ HoodieCleanPartitionMetadata cleanPartitionMetadata = new HoodieCleanPartitionMetadata(partition, "", filesDeleted, filesDeleted,
+ Collections.EMPTY_LIST, false);
+ partitionMetadata.put(partition, cleanPartitionMetadata);
Review Comment:
Collections.emptyList()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1642898366
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
}, {
"hash" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18698",
"triggerID" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"triggerType" : "PUSH"
}, {
"hash" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 80f3d5256ccfc42838f274d47357f750b5f967bd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688)
* a1242a526d022ee3346d593f74421c2dd3d4fcc2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18698)
* 9b6e75df6b03096ab3e461da8c6b269644e87f10 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1641668588
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677)
* 93c8dc9a2683c82d7abfd33cc26c01eee78722a1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1612261195
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9751b6399ebf6b629f3940d612bdfe2e2005a25f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1615048617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c401984679350ad245c1b60d4f889b8a18715169 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220)
* 3e22656f66687bb920ec82e6764bf083985df09c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1612235907
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3db0e8c5493cf87bac0845ece1373fff8b90dfc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168)
* 9751b6399ebf6b629f3940d612bdfe2e2005a25f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1623257217
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e6568126aab0b098ccaac59e137e902d7a1070c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309)
* debb29aa63e665eed1137973fbe94eee395dc286 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1628105901
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* debb29aa63e665eed1137973fbe94eee395dc286 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374)
* e9fb2f74b5333e6f8f60de11ea6adce497252648 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1632996425
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525)
* 89a1c11b7cdf452c37c190ee11934c7546c34c25 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1633323333
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89a1c11b7cdf452c37c190ee11934c7546c34c25 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529)
* 5f98b8a27f2948e7f3a0f8673606b793b90729db UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1633520963
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f98b8a27f2948e7f3a0f8673606b793b90729db Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537)
* 0e82dedc198ec3ed2159e346dd37a3b10d493520 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1633516216
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f98b8a27f2948e7f3a0f8673606b793b90729db Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1635249631
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de31bf2ee6551ac5212dc30d7adcba3e2873f0d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555)
* 2f9aa542076faa188839bc55b43dd7f22ec32b62 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1212500637
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
Review Comment:
shouldn't this logic go into restore method in BaseHoodieWriteClient where we trigger the restore for data table. so, this is double/repeated validation right?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -669,32 +669,51 @@ public void restoreToSavepoint() {
* @param savepointTime Savepoint time to rollback to
*/
public void restoreToSavepoint(String savepointTime) {
- boolean initialMetadataTableIfNecessary = config.isMetadataTableEnabled();
- if (initialMetadataTableIfNecessary) {
+ boolean initializeMetadataTableIfNecessary = config.isMetadataTableEnabled();
+ if (initializeMetadataTableIfNecessary) {
try {
- // Delete metadata table directly when users trigger savepoint rollback if mdt existed and beforeTimelineStarts
+ // Delete metadata table directly when users trigger savepoint rollback if mdt existed and if the savePointTime is beforeTimelineStarts
+ // or before the oldest compaction on MDT.
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
String metadataTableBasePathStr = HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath());
HoodieTableMetaClient mdtClient = HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(metadataTableBasePathStr).build();
- // Same as HoodieTableMetadataUtil#processRollbackMetadata
+ Option<HoodieInstant> lastCompaction = mdtClient.getCommitTimeline().filterCompletedInstants().lastInstant();
Review Comment:
rename to "latestMdtCompaction"
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
+ if (lastCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(restoreToInstantTime, lastCompaction.get().getTimestamp())) {
+ String msg = String.format("Cannot restore MDT to %s because it is older than latest compaction at %s", restoreToInstantTime,
+ lastCompaction.get().getTimestamp()) + ". Please delete MDT and restore again";
+ LOG.error(msg);
+ throw new HoodieMetadataException(msg);
+ }
+ }
+
+ // Restore requires the existing pipelines to be shutdown. So we can safely scan the dataset to find the current
+ // list of files in the filesystem.
+ List<DirectoryInfo> dirInfoList = listAllPartitions(dataMetaClient);
+ Map<String, DirectoryInfo> dirInfoMap = dirInfoList.stream().collect(Collectors.toMap(DirectoryInfo::getRelativePath, Function.identity()));
+ dirInfoList.clear();
+
+ LOG.info("Restoring MDT to " + restoreToInstantTime + " at " + instantTime);
+ getWriteClient().restoreToInstant(restoreToInstantTime, false);
+
+ // At this point we have also reverted the cleans which have occurred after the restoreToInstantTime. Hence, a sync
Review Comment:
do we have tests covering this scenario?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
Review Comment:
do we have tests for all these scenarios ?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
Review Comment:
why is this check required? can you help me understand ?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
+ if (lastCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(restoreToInstantTime, lastCompaction.get().getTimestamp())) {
+ String msg = String.format("Cannot restore MDT to %s because it is older than latest compaction at %s", restoreToInstantTime,
+ lastCompaction.get().getTimestamp()) + ". Please delete MDT and restore again";
+ LOG.error(msg);
+ throw new HoodieMetadataException(msg);
+ }
+ }
+
+ // Restore requires the existing pipelines to be shutdown. So we can safely scan the dataset to find the current
+ // list of files in the filesystem.
+ List<DirectoryInfo> dirInfoList = listAllPartitions(dataMetaClient);
+ Map<String, DirectoryInfo> dirInfoMap = dirInfoList.stream().collect(Collectors.toMap(DirectoryInfo::getRelativePath, Function.identity()));
+ dirInfoList.clear();
+
+ LOG.info("Restoring MDT to " + restoreToInstantTime + " at " + instantTime);
+ getWriteClient().restoreToInstant(restoreToInstantTime, false);
+
+ // At this point we have also reverted the cleans which have occurred after the restoreToInstantTime. Hence, a sync
+ // is required to bring back those cleans.
+ try {
+ initTableMetadata();
+ HoodieCleanMetadata cleanMetadata = new HoodieCleanMetadata();
+ Map<String, HoodieCleanPartitionMetadata> partitionMetadata = new HashMap<>();
+ for (String partition : metadata.fetchAllPartitionPaths()) {
+ FileStatus[] metadataFiles = metadata.getAllFilesInPartition(new Path(dataWriteConfig.getBasePath(), partition));
+ if (!dirInfoMap.containsKey(partition)) {
+ // Entire partition has been deleted
+ List<String> filePaths = Arrays.stream(metadataFiles).map(f -> f.getPath().getName()).collect(Collectors.toList());
+ HoodieCleanPartitionMetadata cleanPartitionMetadata = new HoodieCleanPartitionMetadata(partition, "", filePaths, filePaths,
+ Collections.emptyList(), true);
+ partitionMetadata.put(partition, cleanPartitionMetadata);
+ } else {
+ // Some files cleaned in the partition
+ Map<String, Long> fsFiles = dirInfoMap.get(partition).getFileNameToSizeMap();
+ List<String> filesDeleted = Arrays.stream(metadataFiles).map(f -> f.getPath().getName())
+ .filter(n -> !fsFiles.containsKey(n)).collect(Collectors.toList());
+ if (!filesDeleted.isEmpty()) {
+ LOG.info("Found deleted files in partition " + partition + ": " + filesDeleted);
+ HoodieCleanPartitionMetadata cleanPartitionMetadata = new HoodieCleanPartitionMetadata(partition, "", filesDeleted, filesDeleted,
+ Collections.EMPTY_LIST, false);
+ partitionMetadata.put(partition, cleanPartitionMetadata);
+ }
+ }
+ }
+ cleanMetadata.setPartitionMetadata(partitionMetadata);
+
+ // Even if we don't have any deleted files to sync, we still create an empty commit so that we can track the restore has completed.
+ // We cannot create a deltaCommit at instantTime now because a future block (rollback) has already been written to the logFiles.
+ // We need to choose a timestamp which would be a validInstantTime for MDT. This is either a commit timestamp completed on the dataset
+ // or a timestamp with suffix which we use for MDT clean, compaction etc.
+ // String syncCommitTime = HoodieTableMetadataUtil.createIndexInitTimestamp(HoodieActiveTimeline.createNewInstantTime());
+ // TODO: Using METADATA_INDEXER_TIME_SUFFIX for now, but this should have its own suffix. To be fixed after HUDI-6200
+ String syncCommitTime = HoodieActiveTimeline.createNewInstantTime() + METADATA_INDEXER_TIME_SUFFIX;
Review Comment:
@codope : is there any dependencies we have around the indexing commit suffix? do you see any issues here. or can u confirm if we are good
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records, false);
- closeInternal();
+ if (deltaCommitsInfo.get().getKey().containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
Review Comment:
NTS(note to self). need to review this closer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1612198023
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3db0e8c5493cf87bac0845ece1373fff8b90dfc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1612192625
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd4da7877844008950cba2caa5be10b1a81f2795 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626)
* d3db0e8c5493cf87bac0845ece1373fff8b90dfc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1578055679
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 678a19f937e2803f1d397e061b29d5693d2d8926 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613)
* cd4da7877844008950cba2caa5be10b1a81f2795 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1616765848
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3e22656f66687bb920ec82e6764bf083985df09c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241)
* 7a5df4eae6ca374734ebe19a57bb694208a0320d Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1254985620
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -991,10 +992,6 @@ public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime)
// Find the deltacommits since the last compaction
Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
- if (!deltaCommitsInfo.isPresent() || deltaCommitsInfo.get().getKey().empty()) {
Review Comment:
here irrespective of whether the commit to rollback is already present in MDT or not, we might have to go ahead and add the delta commit. thats why I had to fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1634343073
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e82dedc198ec3ed2159e346dd37a3b10d493520 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543)
* de31bf2ee6551ac5212dc30d7adcba3e2873f0d5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1628092201
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* debb29aa63e665eed1137973fbe94eee395dc286 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374)
* e9fb2f74b5333e6f8f60de11ea6adce497252648 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1632606284
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 74451cd8ee26909fac98e88850ba352c8c415ef5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493)
* 37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1633525872
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5f98b8a27f2948e7f3a0f8673606b793b90729db Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537)
* 0e82dedc198ec3ed2159e346dd37a3b10d493520 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1634683273
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de31bf2ee6551ac5212dc30d7adcba3e2873f0d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1641916638
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93c8dc9a2683c82d7abfd33cc26c01eee78722a1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1642860753
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
}, {
"hash" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 80f3d5256ccfc42838f274d47357f750b5f967bd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688)
* a1242a526d022ee3346d593f74421c2dd3d4fcc2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1635243682
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* de31bf2ee6551ac5212dc30d7adcba3e2873f0d5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555)
* 2f9aa542076faa188839bc55b43dd7f22ec32b62 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1632900338
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1223857627
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records, false);
- closeInternal();
+ if (deltaCommitsInfo.get().getKey().containsInstant(deltaCommitInstant)) {
Review Comment:
Use `deltacommitsSinceCompaction` should be fine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1642866577
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
}, {
"hash" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18698",
"triggerID" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 80f3d5256ccfc42838f274d47357f750b5f967bd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688)
* a1242a526d022ee3346d593f74421c2dd3d4fcc2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18698)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1642481437
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 80f3d5256ccfc42838f274d47357f750b5f967bd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1223859381
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records, false);
- closeInternal();
+ if (deltaCommitsInfo.get().getKey().containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
+ rollbackMetadata, getRecordsGenerationParams(), instantTime,
+ metadata.getSyncedInstantTime(), true), false);
Review Comment:
Not sure why we perform the rollback again ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1213476749
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
Review Comment:
CompactionUtils.getDeltaCommitsSinceLatestCompaction returns a Pair. The value in that Pair can be either a DeltaCommit instant (if no compactions happened) or a Commit action (if a compaction was found).
We only want to check for the compaction here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1243211951
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
+ if (lastCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(restoreToInstantTime, lastCompaction.get().getTimestamp())) {
+ String msg = String.format("Cannot restore MDT to %s because it is older than latest compaction at %s", restoreToInstantTime,
+ lastCompaction.get().getTimestamp()) + ". Please delete MDT and restore again";
+ LOG.error(msg);
+ throw new HoodieMetadataException(msg);
+ }
+ }
+
+ // Restore requires the existing pipelines to be shutdown. So we can safely scan the dataset to find the current
+ // list of files in the filesystem.
+ List<DirectoryInfo> dirInfoList = listAllPartitions(dataMetaClient);
+ Map<String, DirectoryInfo> dirInfoMap = dirInfoList.stream().collect(Collectors.toMap(DirectoryInfo::getRelativePath, Function.identity()));
+ dirInfoList.clear();
+
+ LOG.info("Restoring MDT to " + restoreToInstantTime + " at " + instantTime);
+ getWriteClient().restoreToInstant(restoreToInstantTime, false);
+
+ // At this point we have also reverted the cleans which have occurred after the restoreToInstantTime. Hence, a sync
Review Comment:
Ignore me, this is a restore operation itself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1245804286
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records, false);
- closeInternal();
+ if (deltaCommitsInfo.get().getKey().containsInstant(deltaCommitInstant)) {
Review Comment:
Changed. good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1615240130
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3e22656f66687bb920ec82e6764bf083985df09c Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1613571050
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9751b6399ebf6b629f3940d612bdfe2e2005a25f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172)
* 50a92342798b808ebe521d82b99e4622eeb77ce8 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1614181963
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c401984679350ad245c1b60d4f889b8a18715169 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1613593548
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9751b6399ebf6b629f3940d612bdfe2e2005a25f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172)
* 50a92342798b808ebe521d82b99e4622eeb77ce8 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1268098329
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -742,8 +750,10 @@ private static List<HoodieRecord> convertFilesToFilesPartitionRecords(Map<String
partitionToAppendedFiles.forEach((partitionName, appendedFileMap) -> {
final String partition = getPartitionIdentifier(partitionName);
fileChangeCount[1] += appendedFileMap.size();
+ Map<String, Long> appendedFiles = new HashMap<>();
+ appendedFileMap.forEach((k,v) -> appendedFiles.put(k, 1L));
Review Comment:
this code has been refactored and is not valid anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1642903294
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
}, {
"hash" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18698",
"triggerID" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"triggerType" : "PUSH"
}, {
"hash" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18699",
"triggerID" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a1242a526d022ee3346d593f74421c2dd3d4fcc2 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18698)
* 9b6e75df6b03096ab3e461da8c6b269644e87f10 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18699)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1578527944
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* cd4da7877844008950cba2caa5be10b1a81f2795 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1243220802
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records, false);
- closeInternal();
+ if (deltaCommitsInfo.get().getKey().containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
+ rollbackMetadata, getRecordsGenerationParams(), instantTime,
+ metadata.getSyncedInstantTime(), true), false);
Review Comment:
Yeah, the MDT log reader can assure the correctness(ignore all the commits that are not included/valid on the DT) , so this rollback might be unnecessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1251833218
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java:
##########
@@ -1871,7 +1865,11 @@ private void testTableOperationsImpl(HoodieSparkEngineContext engineContext, Hoo
validateMetadata(client);
// Restore
- client.restoreToInstant("20210101000600000", writeConfig.isMetadataTableEnabled());
+ if (metaClient.getTableType() == COPY_ON_WRITE) {
+ assertThrows(HoodieRestoreException.class, () -> client.restoreToInstant("20210101000600000", writeConfig.isMetadataTableEnabled()));
Review Comment:
found an issue with restore metadata update. i've fixed it and removed this conditional block from the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1239143776
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -669,32 +669,51 @@ public void restoreToSavepoint() {
* @param savepointTime Savepoint time to rollback to
*/
public void restoreToSavepoint(String savepointTime) {
- boolean initialMetadataTableIfNecessary = config.isMetadataTableEnabled();
- if (initialMetadataTableIfNecessary) {
+ boolean initializeMetadataTableIfNecessary = config.isMetadataTableEnabled();
+ if (initializeMetadataTableIfNecessary) {
try {
- // Delete metadata table directly when users trigger savepoint rollback if mdt existed and beforeTimelineStarts
+ // Delete metadata table directly when users trigger savepoint rollback if mdt existed and if the savePointTime is beforeTimelineStarts
+ // or before the oldest compaction on MDT.
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
String metadataTableBasePathStr = HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath());
HoodieTableMetaClient mdtClient = HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(metadataTableBasePathStr).build();
- // Same as HoodieTableMetadataUtil#processRollbackMetadata
+ Option<HoodieInstant> latestMdtCompaction = mdtClient.getCommitTimeline().filterCompletedInstants().lastInstant();
+ boolean deleteMDT = false;
+ if (latestMdtCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(savepointTime, latestMdtCompaction.get().getTimestamp())) {
+ LOG.warn(String.format("Deleting MDT during restore to %s as the savepoint is older than oldest compaction %s on MDT",
+ savepointTime, latestMdtCompaction.get().getTimestamp()));
+ deleteMDT = true;
+ }
+ }
+
HoodieInstant syncedInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, savepointTime);
// The instant required to sync rollback to MDT has been archived and the mdt syncing will be failed
// So that we need to delete the whole MDT here.
if (mdtClient.getCommitsTimeline().isBeforeTimelineStarts(syncedInstant.getTimestamp())) {
+ LOG.warn(String.format("Deleting MDT during restore to %s as the savepoint is older than the MDT timeline %s",
Review Comment:
we can do a minor optimization here.
```
if (!deleteMDT && ...)
```
bcoz, if previous block already decided to delete MDT, we don't need to further check other conditions
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
Review Comment:
is it oldest Compaction or latest Compaction. java docs does not align w/ code.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
Review Comment:
+1
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -669,32 +669,51 @@ public void restoreToSavepoint() {
* @param savepointTime Savepoint time to rollback to
*/
public void restoreToSavepoint(String savepointTime) {
- boolean initialMetadataTableIfNecessary = config.isMetadataTableEnabled();
- if (initialMetadataTableIfNecessary) {
+ boolean initializeMetadataTableIfNecessary = config.isMetadataTableEnabled();
+ if (initializeMetadataTableIfNecessary) {
try {
- // Delete metadata table directly when users trigger savepoint rollback if mdt existed and beforeTimelineStarts
+ // Delete metadata table directly when users trigger savepoint rollback if mdt existed and if the savePointTime is beforeTimelineStarts
+ // or before the oldest compaction on MDT.
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
String metadataTableBasePathStr = HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath());
HoodieTableMetaClient mdtClient = HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(metadataTableBasePathStr).build();
- // Same as HoodieTableMetadataUtil#processRollbackMetadata
+ Option<HoodieInstant> latestMdtCompaction = mdtClient.getCommitTimeline().filterCompletedInstants().lastInstant();
Review Comment:
shouldn't this be earliest/first instant instead of lastInstant? as per the comments givein in L675 ish
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -669,32 +669,51 @@ public void restoreToSavepoint() {
* @param savepointTime Savepoint time to rollback to
*/
public void restoreToSavepoint(String savepointTime) {
- boolean initialMetadataTableIfNecessary = config.isMetadataTableEnabled();
- if (initialMetadataTableIfNecessary) {
+ boolean initializeMetadataTableIfNecessary = config.isMetadataTableEnabled();
+ if (initializeMetadataTableIfNecessary) {
try {
- // Delete metadata table directly when users trigger savepoint rollback if mdt existed and beforeTimelineStarts
+ // Delete metadata table directly when users trigger savepoint rollback if mdt existed and if the savePointTime is beforeTimelineStarts
+ // or before the oldest compaction on MDT.
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
String metadataTableBasePathStr = HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath());
HoodieTableMetaClient mdtClient = HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(metadataTableBasePathStr).build();
- // Same as HoodieTableMetadataUtil#processRollbackMetadata
+ Option<HoodieInstant> latestMdtCompaction = mdtClient.getCommitTimeline().filterCompletedInstants().lastInstant();
+ boolean deleteMDT = false;
+ if (latestMdtCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(savepointTime, latestMdtCompaction.get().getTimestamp())) {
+ LOG.warn(String.format("Deleting MDT during restore to %s as the savepoint is older than oldest compaction %s on MDT",
+ savepointTime, latestMdtCompaction.get().getTimestamp()));
+ deleteMDT = true;
+ }
+ }
+
HoodieInstant syncedInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, savepointTime);
// The instant required to sync rollback to MDT has been archived and the mdt syncing will be failed
// So that we need to delete the whole MDT here.
if (mdtClient.getCommitsTimeline().isBeforeTimelineStarts(syncedInstant.getTimestamp())) {
+ LOG.warn(String.format("Deleting MDT during restore to %s as the savepoint is older than the MDT timeline %s",
+ savepointTime, mdtClient.getCommitsTimeline().firstInstant().get().getTimestamp()));
+ deleteMDT = true;
+ }
+
+ if (deleteMDT) {
+ // TODO: this should use the correct API to delete from HoodieTableMetadataUtil so hoodie.properties is also updated
+ // To be fixed after HUDI-6200
Review Comment:
can you take care of this now since we have landed the other patch
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
+ if (lastCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(restoreToInstantTime, lastCompaction.get().getTimestamp())) {
+ String msg = String.format("Cannot restore MDT to %s because it is older than latest compaction at %s", restoreToInstantTime,
+ lastCompaction.get().getTimestamp()) + ". Please delete MDT and restore again";
+ LOG.error(msg);
+ throw new HoodieMetadataException(msg);
+ }
+ }
+
+ // Restore requires the existing pipelines to be shutdown. So we can safely scan the dataset to find the current
+ // list of files in the filesystem.
+ List<DirectoryInfo> dirInfoList = listAllPartitions(dataMetaClient);
+ Map<String, DirectoryInfo> dirInfoMap = dirInfoList.stream().collect(Collectors.toMap(DirectoryInfo::getRelativePath, Function.identity()));
+ dirInfoList.clear();
+
+ LOG.info("Restoring MDT to " + restoreToInstantTime + " at " + instantTime);
+ getWriteClient().restoreToInstant(restoreToInstantTime, false);
+
+ // At this point we have also reverted the cleans which have occurred after the restoreToInstantTime. Hence, a sync
+ // is required to bring back those cleans.
+ try {
+ initTableMetadata();
+ HoodieCleanMetadata cleanMetadata = new HoodieCleanMetadata();
+ Map<String, HoodieCleanPartitionMetadata> partitionMetadata = new HashMap<>();
+ for (String partition : metadata.fetchAllPartitionPaths()) {
+ FileStatus[] metadataFiles = metadata.getAllFilesInPartition(new Path(dataWriteConfig.getBasePath(), partition));
+ if (!dirInfoMap.containsKey(partition)) {
+ // Entire partition has been deleted
+ List<String> filePaths = Arrays.stream(metadataFiles).map(f -> f.getPath().getName()).collect(Collectors.toList());
+ HoodieCleanPartitionMetadata cleanPartitionMetadata = new HoodieCleanPartitionMetadata(partition, "", filePaths, filePaths,
+ Collections.emptyList(), true);
+ partitionMetadata.put(partition, cleanPartitionMetadata);
+ } else {
+ // Some files cleaned in the partition
+ Map<String, Long> fsFiles = dirInfoMap.get(partition).getFileNameToSizeMap();
+ List<String> filesDeleted = Arrays.stream(metadataFiles).map(f -> f.getPath().getName())
+ .filter(n -> !fsFiles.containsKey(n)).collect(Collectors.toList());
+ if (!filesDeleted.isEmpty()) {
+ LOG.info("Found deleted files in partition " + partition + ": " + filesDeleted);
+ HoodieCleanPartitionMetadata cleanPartitionMetadata = new HoodieCleanPartitionMetadata(partition, "", filesDeleted, filesDeleted,
+ Collections.EMPTY_LIST, false);
+ partitionMetadata.put(partition, cleanPartitionMetadata);
+ }
+ }
+ }
+ cleanMetadata.setPartitionMetadata(partitionMetadata);
+
+ // Even if we don't have any deleted files to sync, we still create an empty commit so that we can track the restore has completed.
+ // We cannot create a deltaCommit at instantTime now because a future block (rollback) has already been written to the logFiles.
+ // We need to choose a timestamp which would be a validInstantTime for MDT. This is either a commit timestamp completed on the dataset
+ // or a timestamp with suffix which we use for MDT clean, compaction etc.
+ // String syncCommitTime = HoodieTableMetadataUtil.createIndexInitTimestamp(HoodieActiveTimeline.createNewInstantTime());
+ // TODO: Using METADATA_INDEXER_TIME_SUFFIX for now, but this should have its own suffix. To be fixed after HUDI-6200
+ String syncCommitTime = HoodieActiveTimeline.createNewInstantTime() + METADATA_INDEXER_TIME_SUFFIX;
Review Comment:
@codope : gentle ping
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -625,6 +625,9 @@ private static void processRestoreMetadata(HoodieActiveTimeline metadataTableTim
/**
* Convert rollback action metadata to metadata table records.
+ * <p>
+ * We only need to handle FILES partition here as HUDI rollbacks on MOR table may end up adding a new log file. All other partitions
+ * are handled by actual rollback of the deltacommit which added records to those partitions.
Review Comment:
why we do this. Triggering rollback for the MDT will actually add rollback blocks to all partitions in MDT right. So, no additional handling should be required in my understanding.
or am I missing something.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records, false);
- closeInternal();
+ if (deltaCommitsInfo.get().getKey().containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
+ rollbackMetadata, getRecordsGenerationParams(), instantTime,
+ metadata.getSyncedInstantTime(), true), false);
Review Comment:
yeah. not sure I understand why we do this. rollback in L952 only adds rollback to MDT right. so no new files will be added to data table. So, what exactly are we trying to do here.
Also, w/ this change, we are adding some new instant times to MDT timeline which may not be present in DT timeline. so, did we ensure we account for these in validTimestamps with LogRecordReader?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
+ if (lastCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(restoreToInstantTime, lastCompaction.get().getTimestamp())) {
+ String msg = String.format("Cannot restore MDT to %s because it is older than latest compaction at %s", restoreToInstantTime,
+ lastCompaction.get().getTimestamp()) + ". Please delete MDT and restore again";
+ LOG.error(msg);
+ throw new HoodieMetadataException(msg);
+ }
+ }
+
+ // Restore requires the existing pipelines to be shutdown. So we can safely scan the dataset to find the current
+ // list of files in the filesystem.
+ List<DirectoryInfo> dirInfoList = listAllPartitions(dataMetaClient);
+ Map<String, DirectoryInfo> dirInfoMap = dirInfoList.stream().collect(Collectors.toMap(DirectoryInfo::getRelativePath, Function.identity()));
+ dirInfoList.clear();
+
+ LOG.info("Restoring MDT to " + restoreToInstantTime + " at " + instantTime);
+ getWriteClient().restoreToInstant(restoreToInstantTime, false);
+
+ // At this point we have also reverted the cleans which have occurred after the restoreToInstantTime. Hence, a sync
Review Comment:
I mean, specifically where we have cleans in the timeline which is being restored. I did happen to check the test testTableOperationsWithRestore, but I don't think it tests this case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1633681187
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e82dedc198ec3ed2159e346dd37a3b10d493520 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1633008402
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525)
* 89a1c11b7cdf452c37c190ee11934c7546c34c25 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1567282281
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 910547b944f11039c2fa4b173c54fd5a1abcf6c2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1566956280
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 910547b944f11039c2fa4b173c54fd5a1abcf6c2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1213470892
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -669,32 +669,51 @@ public void restoreToSavepoint() {
* @param savepointTime Savepoint time to rollback to
*/
public void restoreToSavepoint(String savepointTime) {
- boolean initialMetadataTableIfNecessary = config.isMetadataTableEnabled();
- if (initialMetadataTableIfNecessary) {
+ boolean initializeMetadataTableIfNecessary = config.isMetadataTableEnabled();
+ if (initializeMetadataTableIfNecessary) {
try {
- // Delete metadata table directly when users trigger savepoint rollback if mdt existed and beforeTimelineStarts
+ // Delete metadata table directly when users trigger savepoint rollback if mdt existed and if the savePointTime is beforeTimelineStarts
+ // or before the oldest compaction on MDT.
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
String metadataTableBasePathStr = HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath());
HoodieTableMetaClient mdtClient = HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(metadataTableBasePathStr).build();
- // Same as HoodieTableMetadataUtil#processRollbackMetadata
+ Option<HoodieInstant> lastCompaction = mdtClient.getCommitTimeline().filterCompletedInstants().lastInstant();
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1243210528
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
+ if (lastCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(restoreToInstantTime, lastCompaction.get().getTimestamp())) {
+ String msg = String.format("Cannot restore MDT to %s because it is older than latest compaction at %s", restoreToInstantTime,
+ lastCompaction.get().getTimestamp()) + ". Please delete MDT and restore again";
+ LOG.error(msg);
+ throw new HoodieMetadataException(msg);
+ }
+ }
+
+ // Restore requires the existing pipelines to be shutdown. So we can safely scan the dataset to find the current
+ // list of files in the filesystem.
+ List<DirectoryInfo> dirInfoList = listAllPartitions(dataMetaClient);
+ Map<String, DirectoryInfo> dirInfoMap = dirInfoList.stream().collect(Collectors.toMap(DirectoryInfo::getRelativePath, Function.identity()));
+ dirInfoList.clear();
+
+ LOG.info("Restoring MDT to " + restoreToInstantTime + " at " + instantTime);
+ getWriteClient().restoreToInstant(restoreToInstantTime, false);
+
+ // At this point we have also reverted the cleans which have occurred after the restoreToInstantTime. Hence, a sync
Review Comment:
The impli is kind of ugly here, we want to rollback to a specific history time point of the MDT, but we also want to keep the commits for cleaning, is there any possibility we only rollback the commit(we don't want) instead of perform a `restoreToInstant` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1567018996
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 910547b944f11039c2fa4b173c54fd5a1abcf6c2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1641366845
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f9aa542076faa188839bc55b43dd7f22ec32b62 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580)
* fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1577422937
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 910547b944f11039c2fa4b173c54fd5a1abcf6c2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469)
* 678a19f937e2803f1d397e061b29d5693d2d8926 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1213473799
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
Review Comment:
Yes, the BaseHoodieWriteClient also has this check. It is duplicated as we allow update() methods on the MDT to be called from outside. Its safer this way I suppose though duplicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1218525849
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
Review Comment:
TestHoodieBackedMetadata#testTableOperationsWithRestore should cover the real cases.
We should actually not have the !deltaCommitsInfo.isPresent() case because this means MDT is empty (no deltacommits) and we are trying to perform a restore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1643519687
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
}, {
"hash" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18698",
"triggerID" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"triggerType" : "PUSH"
}, {
"hash" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18699",
"triggerID" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"triggerType" : "PUSH"
}, {
"hash" : "d121977ee5e8e7785acf8a0ef82563b971b5ff27",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18711",
"triggerID" : "d121977ee5e8e7785acf8a0ef82563b971b5ff27",
"triggerType" : "PUSH"
}, {
"hash" : "687afd0dcc56a8dd1266486d452656fa3abbf0e4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "687afd0dcc56a8dd1266486d452656fa3abbf0e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d121977ee5e8e7785acf8a0ef82563b971b5ff27 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18711)
* 687afd0dcc56a8dd1266486d452656fa3abbf0e4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1643507345
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
}, {
"hash" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18698",
"triggerID" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"triggerType" : "PUSH"
}, {
"hash" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18699",
"triggerID" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"triggerType" : "PUSH"
}, {
"hash" : "d121977ee5e8e7785acf8a0ef82563b971b5ff27",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18711",
"triggerID" : "d121977ee5e8e7785acf8a0ef82563b971b5ff27",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d121977ee5e8e7785acf8a0ef82563b971b5ff27 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18711)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1245993509
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -914,26 +982,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (initialized && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records);
- closeInternal();
+ if (deltacommitsSinceCompaction.containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
Review Comment:
Can we optimize it a little to confine the scope to MOR table only. Because only MOR table could add new log files for rollback purpose.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1245811595
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -625,6 +625,9 @@ private static void processRestoreMetadata(HoodieActiveTimeline metadataTableTim
/**
* Convert rollback action metadata to metadata table records.
+ * <p>
+ * We only need to handle FILES partition here as HUDI rollbacks on MOR table may end up adding a new log file. All other partitions
+ * are handled by actual rollback of the deltacommit which added records to those partitions.
Review Comment:
This is for the case of MOR table restores/rollbacks which may add additional files (log files).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1248170142
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -973,52 +973,46 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- // The commit which is being rolled back on the dataset
- final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
- // Find the deltacommits since the last compaction
- Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
- CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
- if (!deltaCommitsInfo.isPresent()) {
- LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
- return;
- }
-
- // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
- HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
- HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
-
- // The deltacommit that will be rolled back
- HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
-
- // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
- // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
- if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
- final String compactionInstantTime = compactionInstant.getTimestamp();
- if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
- throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
- + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
- deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
+ if (initialized && metadata != null) {
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent() || deltaCommitsInfo.get().getKey().empty()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
}
- }
- if (deltacommitsSinceCompaction.containsInstant(deltaCommitInstant)) {
- LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
- if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
- throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
+ }
}
- } else {
- LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
- commitInstantTime, instantTime));
- }
- // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
- if (dataMetaClient.getTableType().equals(HoodieTableType.MERGE_ON_READ)) {
Review Comment:
@prashantwason : I jammed w/ Sagar and we feel this is not really required. Lets sync up f2f and get it clarified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1615032413
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c401984679350ad245c1b60d4f889b8a18715169 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220)
* 3e22656f66687bb920ec82e6764bf083985df09c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1245807148
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records, false);
- closeInternal();
+ if (deltaCommitsInfo.get().getKey().containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
+ rollbackMetadata, getRecordsGenerationParams(), instantTime,
+ metadata.getSyncedInstantTime(), true), false);
Review Comment:
In the processAndCommit we are using the "instantTime" which is from the actual restore on the dataset.
Dataset timeline: instantTime.restore
Metadata timeline: instantTime.restore (this is restoring MDT to instantTime) instantTime.deltacommit (this contains the MOR log files added, if any)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1245801386
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
+
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
+ Option<HoodieInstant> lastCompaction = metadataMetaClient.getCommitTimeline().filterCompletedInstants().lastInstant();
Review Comment:
Renamed to oldestCompaction and fixed the retrieval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1245803629
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -837,10 +840,75 @@ public void update(HoodieCleanMetadata cleanMetadata, String instantTime) {
*/
@Override
public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
- processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext,
- metadataMetaClient.getActiveTimeline(), restoreMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime()), false);
- closeInternal();
+ dataMetaClient.reloadActiveTimeline();
+
+ // Since the restore has completed on the dataset, the latest write timeline instant is the one to which the
+ // restore was performed. This should be always present.
+ final String restoreToInstantTime = dataMetaClient.getActiveTimeline().getWriteTimeline()
+ .getReverseOrderedInstants().findFirst().get().getTimestamp();
Review Comment:
Fixed. I am still used to 0.10 APIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1631279954
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e9fb2f74b5333e6f8f60de11ea6adce497252648 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436)
* 74451cd8ee26909fac98e88850ba352c8c415ef5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1245809621
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java:
##########
@@ -669,32 +669,51 @@ public void restoreToSavepoint() {
* @param savepointTime Savepoint time to rollback to
*/
public void restoreToSavepoint(String savepointTime) {
- boolean initialMetadataTableIfNecessary = config.isMetadataTableEnabled();
- if (initialMetadataTableIfNecessary) {
+ boolean initializeMetadataTableIfNecessary = config.isMetadataTableEnabled();
+ if (initializeMetadataTableIfNecessary) {
try {
- // Delete metadata table directly when users trigger savepoint rollback if mdt existed and beforeTimelineStarts
+ // Delete metadata table directly when users trigger savepoint rollback if mdt existed and if the savePointTime is beforeTimelineStarts
+ // or before the oldest compaction on MDT.
+ // We cannot restore to before the oldest compaction on MDT as we don't have the basefiles before that time.
String metadataTableBasePathStr = HoodieTableMetadata.getMetadataTableBasePath(config.getBasePath());
HoodieTableMetaClient mdtClient = HoodieTableMetaClient.builder().setConf(hadoopConf).setBasePath(metadataTableBasePathStr).build();
- // Same as HoodieTableMetadataUtil#processRollbackMetadata
+ Option<HoodieInstant> latestMdtCompaction = mdtClient.getCommitTimeline().filterCompletedInstants().lastInstant();
+ boolean deleteMDT = false;
+ if (latestMdtCompaction.isPresent()) {
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(savepointTime, latestMdtCompaction.get().getTimestamp())) {
+ LOG.warn(String.format("Deleting MDT during restore to %s as the savepoint is older than oldest compaction %s on MDT",
+ savepointTime, latestMdtCompaction.get().getTimestamp()));
+ deleteMDT = true;
+ }
+ }
+
HoodieInstant syncedInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, savepointTime);
// The instant required to sync rollback to MDT has been archived and the mdt syncing will be failed
// So that we need to delete the whole MDT here.
if (mdtClient.getCommitsTimeline().isBeforeTimelineStarts(syncedInstant.getTimestamp())) {
+ LOG.warn(String.format("Deleting MDT during restore to %s as the savepoint is older than the MDT timeline %s",
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1612231341
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d3db0e8c5493cf87bac0845ece1373fff8b90dfc Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168)
* 9751b6399ebf6b629f3940d612bdfe2e2005a25f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1623245122
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e6568126aab0b098ccaac59e137e902d7a1070c3 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309)
* debb29aa63e665eed1137973fbe94eee395dc286 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1252500090
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records, false);
- closeInternal();
+ if (deltaCommitsInfo.get().getKey().containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
+ rollbackMetadata, getRecordsGenerationParams(), instantTime,
+ metadata.getSyncedInstantTime(), true), false);
Review Comment:
Discussed offline, we need to track the inflight log files for cleaning anyway, but we have no good manner to fix that currently, needs think through ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1633328438
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89a1c11b7cdf452c37c190ee11934c7546c34c25 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529)
* 5f98b8a27f2948e7f3a0f8673606b793b90729db Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1635422204
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2f9aa542076faa188839bc55b43dd7f22ec32b62 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1633215123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 89a1c11b7cdf452c37c190ee11934c7546c34c25 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1268098329
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -742,8 +750,10 @@ private static List<HoodieRecord> convertFilesToFilesPartitionRecords(Map<String
partitionToAppendedFiles.forEach((partitionName, appendedFileMap) -> {
final String partition = getPartitionIdentifier(partitionName);
fileChangeCount[1] += appendedFileMap.size();
+ Map<String, Long> appendedFiles = new HashMap<>();
+ appendedFileMap.forEach((k,v) -> appendedFiles.put(k, 1L));
Review Comment:
this code has been refactored and the feedback is not valid anymore.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1642226909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93c8dc9a2683c82d7abfd33cc26c01eee78722a1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683)
* 80f3d5256ccfc42838f274d47357f750b5f967bd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1641508668
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1642209702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 93c8dc9a2683c82d7abfd33cc26c01eee78722a1 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683)
* 80f3d5256ccfc42838f274d47357f750b5f967bd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1248137779
##########
hudi-client/hudi-spark-client/src/test/java/org/apache/hudi/client/functional/TestHoodieBackedMetadata.java:
##########
@@ -1871,7 +1865,11 @@ private void testTableOperationsImpl(HoodieSparkEngineContext engineContext, Hoo
validateMetadata(client);
// Restore
- client.restoreToInstant("20210101000600000", writeConfig.isMetadataTableEnabled());
+ if (metaClient.getTableType() == COPY_ON_WRITE) {
+ assertThrows(HoodieRestoreException.class, () -> client.restoreToInstant("20210101000600000", writeConfig.isMetadataTableEnabled()));
+ } else {
+ client.restoreToInstant("20210101000600000", writeConfig.isMetadataTableEnabled());
+ }
Review Comment:
There should not be a need to check this based on table type. Need to look into why this fails for COW.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1620098616
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a5df4eae6ca374734ebe19a57bb694208a0320d Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254)
* e6568126aab0b098ccaac59e137e902d7a1070c3 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1613987357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 50a92342798b808ebe521d82b99e4622eeb77ce8 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207)
* c401984679350ad245c1b60d4f889b8a18715169 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] prashantwason commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "prashantwason (via GitHub)" <gi...@apache.org>.
prashantwason commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1245896736
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -966,8 +966,8 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
// String syncCommitTime = HoodieTableMetadataUtil.createIndexInitTimestamp(HoodieActiveTimeline.createNewInstantTime());
// TODO: Using METADATA_INDEXER_TIME_SUFFIX for now, but this should have its own suffix. To be fixed after HUDI-6200
String syncCommitTime = HoodieActiveTimeline.createNewInstantTime() + METADATA_INDEXER_TIME_SUFFIX;
Review Comment:
Should use a unique suffix for this.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -984,52 +984,46 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- // The commit which is being rolled back on the dataset
- final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
- // Find the deltacommits since the last compaction
- Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
- CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
- if (!deltaCommitsInfo.isPresent()) {
- LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
- return;
- }
-
- // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
- HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
- HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
-
- // The deltacommit that will be rolled back
- HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
-
- // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
- // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
- if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
- final String compactionInstantTime = compactionInstant.getTimestamp();
- if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
- throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
- + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
- deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
+ if (initialized && metadata != null) {
Review Comment:
Why is this check required? We should not be calling these updateXX APIs if the HoodieBackedTableMetadata is not initialized already.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -984,52 +984,46 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- // The commit which is being rolled back on the dataset
- final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
- // Find the deltacommits since the last compaction
- Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
- CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
- if (!deltaCommitsInfo.isPresent()) {
- LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
- return;
- }
-
- // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
- HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
- HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
-
- // The deltacommit that will be rolled back
- HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
-
- // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
- // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
- if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
- final String compactionInstantTime = compactionInstant.getTimestamp();
- if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
- throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
- + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
- deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
+ if (initialized && metadata != null) {
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent() || deltaCommitsInfo.get().getKey().empty()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
}
- }
- if (deltacommitsSinceCompaction.containsInstant(deltaCommitInstant)) {
- LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
- if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
- throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
Review Comment:
oldest compaction
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -914,26 +982,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (initialized && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records);
- closeInternal();
+ if (deltacommitsSinceCompaction.containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
Review Comment:
Done
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -966,8 +966,8 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
// String syncCommitTime = HoodieTableMetadataUtil.createIndexInitTimestamp(HoodieActiveTimeline.createNewInstantTime());
// TODO: Using METADATA_INDEXER_TIME_SUFFIX for now, but this should have its own suffix. To be fixed after HUDI-6200
String syncCommitTime = HoodieActiveTimeline.createNewInstantTime() + METADATA_INDEXER_TIME_SUFFIX;
Review Comment:
Done
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -984,52 +984,46 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- // The commit which is being rolled back on the dataset
- final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
- // Find the deltacommits since the last compaction
- Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
- CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
- if (!deltaCommitsInfo.isPresent()) {
- LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
- return;
- }
-
- // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
- HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
- HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
-
- // The deltacommit that will be rolled back
- HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
-
- // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
- // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
- if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
- final String compactionInstantTime = compactionInstant.getTimestamp();
- if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
- throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
- + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
- deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
+ if (initialized && metadata != null) {
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent() || deltaCommitsInfo.get().getKey().empty()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
}
- }
- if (deltacommitsSinceCompaction.containsInstant(deltaCommitInstant)) {
- LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
- if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
- throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
Review Comment:
latest=oldest to make it clear
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -851,26 +919,49 @@ public void update(HoodieRestoreMetadata restoreMetadata, String instantTime) {
*/
@Override
public void update(HoodieRollbackMetadata rollbackMetadata, String instantTime) {
- if (enabled && metadata != null) {
- // Is this rollback of an instant that has been synced to the metadata table?
- String rollbackInstant = rollbackMetadata.getCommitsRollback().get(0);
- boolean wasSynced = metadataMetaClient.getActiveTimeline().containsInstant(new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, rollbackInstant));
- if (!wasSynced) {
- // A compaction may have taken place on metadata table which would have included this instant being rolled back.
- // Revisit this logic to relax the compaction fencing : https://issues.apache.org/jira/browse/HUDI-2458
- Option<String> latestCompaction = metadata.getLatestCompactionTime();
- if (latestCompaction.isPresent()) {
- wasSynced = HoodieTimeline.compareTimestamps(rollbackInstant, HoodieTimeline.LESSER_THAN_OR_EQUALS, latestCompaction.get());
- }
+ // The commit which is being rolled back on the dataset
+ final String commitInstantTime = rollbackMetadata.getCommitsRollback().get(0);
+ // Find the deltacommits since the last compaction
+ Option<Pair<HoodieTimeline, HoodieInstant>> deltaCommitsInfo =
+ CompactionUtils.getDeltaCommitsSinceLatestCompaction(metadataMetaClient.getActiveTimeline());
+ if (!deltaCommitsInfo.isPresent()) {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no deltacommits on MDT", commitInstantTime, instantTime));
+ return;
+ }
+
+ // This could be a compaction or deltacommit instant (See CompactionUtils.getDeltaCommitsSinceLatestCompaction)
+ HoodieInstant compactionInstant = deltaCommitsInfo.get().getValue();
+ HoodieTimeline deltacommitsSinceCompaction = deltaCommitsInfo.get().getKey();
+
+ // The deltacommit that will be rolled back
+ HoodieInstant deltaCommitInstant = new HoodieInstant(false, HoodieTimeline.DELTA_COMMIT_ACTION, commitInstantTime);
+
+ // The commit being rolled back should not be older than the latest compaction on the MDT. Compaction on MDT only occurs when all actions
+ // are completed on the dataset. Hence, this case implies a rollback of completed commit which should actually be handled using restore.
+ if (compactionInstant.getAction().equals(HoodieTimeline.COMMIT_ACTION)) {
+ final String compactionInstantTime = compactionInstant.getTimestamp();
+ if (HoodieTimeline.LESSER_THAN_OR_EQUALS.test(commitInstantTime, compactionInstantTime)) {
+ throw new HoodieMetadataException(String.format("Commit being rolled back %s is older than the latest compaction %s. "
+ + "There are %d deltacommits after this compaction: %s", commitInstantTime, compactionInstantTime,
+ deltacommitsSinceCompaction.countInstants(), deltacommitsSinceCompaction.getInstants()));
}
+ }
- Map<MetadataPartitionType, HoodieData<HoodieRecord>> records =
- HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
- rollbackMetadata, getRecordsGenerationParams(), instantTime,
- metadata.getSyncedInstantTime(), wasSynced);
- commit(instantTime, records, false);
- closeInternal();
+ if (deltaCommitsInfo.get().getKey().containsInstant(deltaCommitInstant)) {
+ LOG.info("Rolling back MDT deltacommit " + commitInstantTime);
+ if (!getWriteClient().rollback(commitInstantTime, instantTime)) {
+ throw new HoodieMetadataException("Failed to rollback deltacommit at " + commitInstantTime);
+ }
+ } else {
+ LOG.info(String.format("Ignoring rollback of instant %s at %s since there are no corresponding deltacommits on MDT",
+ commitInstantTime, instantTime));
}
+
+ // Rollback of MOR table may end up adding a new log file. So we need to check for added files and add them to MDT
+ processAndCommit(instantTime, () -> HoodieTableMetadataUtil.convertMetadataToRecords(engineContext, metadataMetaClient.getActiveTimeline(),
+ rollbackMetadata, getRecordsGenerationParams(), instantTime,
+ metadata.getSyncedInstantTime(), true), false);
Review Comment:
I have changed this to use a custom suffix for restore and rollback operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a diff in pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #8837:
URL: https://github.com/apache/hudi/pull/8837#discussion_r1268077898
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -653,37 +608,57 @@ private static void processRestoreMetadata(HoodieActiveTimeline metadataTableTim
*/
public static Map<MetadataPartitionType, HoodieData<HoodieRecord>> convertMetadataToRecords(
HoodieEngineContext engineContext, HoodieActiveTimeline metadataTableTimeline,
- HoodieRollbackMetadata rollbackMetadata, MetadataRecordsGenerationParams recordsGenerationParams,
- String instantTime, Option<String> lastSyncTs, boolean wasSynced) {
+ HoodieTableMetaClient dataTableMetaClient,
+ HoodieRollbackMetadata rollbackMetadata, String instantTime) {
final Map<MetadataPartitionType, HoodieData<HoodieRecord>> partitionToRecordsMap = new HashMap<>();
- Map<String, List<String>> partitionToDeletedFiles = new HashMap<>();
- Map<String, Map<String, Long>> partitionToAppendedFiles = new HashMap<>();
- List<HoodieRecord> filesPartitionRecords =
- convertMetadataToRollbackRecords(metadataTableTimeline, rollbackMetadata, partitionToDeletedFiles, partitionToAppendedFiles, instantTime, lastSyncTs, wasSynced);
+ List<HoodieRecord> filesPartitionRecords = convertMetadataToRollbackRecords(rollbackMetadata, instantTime);
+
+ List<HoodieRecord> reAddedRecords = getHoodieRecordsForLogFilesFromRollbackPlan(dataTableMetaClient, instantTime);
+ filesPartitionRecords.addAll(reAddedRecords);
final HoodieData<HoodieRecord> rollbackRecordsRDD = engineContext.parallelize(filesPartitionRecords, 1);
+
partitionToRecordsMap.put(MetadataPartitionType.FILES, rollbackRecordsRDD);
return partitionToRecordsMap;
}
+ private static List<HoodieRecord> getHoodieRecordsForLogFilesFromRollbackPlan(HoodieTableMetaClient dataTableMetaClient, String instantTime) {
+ /*List<HoodieInstant> instants = dataTableMetaClient.reloadActiveTimeline().filterRequestedRollbackTimeline()
+ .filter(instant -> instant.getTimestamp().equals(instantTime) && instant.isRequested()).getInstants();*/
+
+ HoodieInstant rollbackInstant = new HoodieInstant(HoodieInstant.State.REQUESTED, HoodieTimeline.ROLLBACK_ACTION, instantTime);
+
+ // HoodieInstant rollbackInstant = instants.get(0);
+ HoodieInstant requested = HoodieTimeline.getRollbackRequestedInstant(rollbackInstant);
+ try {
+ HoodieRollbackPlan rollbackPlan = TimelineMetadataUtils.deserializeAvroMetadata(
+ dataTableMetaClient.getActiveTimeline().readRollbackInfoAsBytes(requested).get(), HoodieRollbackPlan.class);
+
+ Map<String, Map<String, Long>> partitionToLogFilesMap = new HashMap<>();
+
+ rollbackPlan.getRollbackRequests().forEach(rollbackRequest -> {
+ partitionToLogFilesMap.computeIfAbsent(rollbackRequest.getPartitionPath(), s -> new HashMap<>());
+ // fetch only log files that are expected to be RB'd in DT as part of this rollback. these log files will not be deleted, but rendered
+ // invalid once rollback is complete.
+ partitionToLogFilesMap.get(rollbackRequest.getPartitionPath()).putAll(rollbackRequest.getLogBlocksToBeDeleted());
Review Comment:
there are chance we will have more than 1 entry for a given partition in rollbackPlan.getRollbackRequests(). I guess, the optimization you are suggesting based on the assumption that we will call put to partitionToFilesMap only once per partition.
So, I don't see a real benefit here. please do let me know wdyt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8837: [HUDI-6153] Changed the rollback mechanism for MDT to actual rollbacks rather than appending revert blocks.
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8837:
URL: https://github.com/apache/hudi/pull/8837#issuecomment-1643533844
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17469",
"triggerID" : "910547b944f11039c2fa4b173c54fd5a1abcf6c2",
"triggerType" : "PUSH"
}, {
"hash" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17613",
"triggerID" : "678a19f937e2803f1d397e061b29d5693d2d8926",
"triggerType" : "PUSH"
}, {
"hash" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17626",
"triggerID" : "cd4da7877844008950cba2caa5be10b1a81f2795",
"triggerType" : "PUSH"
}, {
"hash" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18168",
"triggerID" : "d3db0e8c5493cf87bac0845ece1373fff8b90dfc",
"triggerType" : "PUSH"
}, {
"hash" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18172",
"triggerID" : "9751b6399ebf6b629f3940d612bdfe2e2005a25f",
"triggerType" : "PUSH"
}, {
"hash" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18207",
"triggerID" : "50a92342798b808ebe521d82b99e4622eeb77ce8",
"triggerType" : "PUSH"
}, {
"hash" : "c401984679350ad245c1b60d4f889b8a18715169",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18220",
"triggerID" : "c401984679350ad245c1b60d4f889b8a18715169",
"triggerType" : "PUSH"
}, {
"hash" : "3e22656f66687bb920ec82e6764bf083985df09c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18241",
"triggerID" : "3e22656f66687bb920ec82e6764bf083985df09c",
"triggerType" : "PUSH"
}, {
"hash" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18254",
"triggerID" : "7a5df4eae6ca374734ebe19a57bb694208a0320d",
"triggerType" : "PUSH"
}, {
"hash" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18309",
"triggerID" : "e6568126aab0b098ccaac59e137e902d7a1070c3",
"triggerType" : "PUSH"
}, {
"hash" : "debb29aa63e665eed1137973fbe94eee395dc286",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18374",
"triggerID" : "debb29aa63e665eed1137973fbe94eee395dc286",
"triggerType" : "PUSH"
}, {
"hash" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18436",
"triggerID" : "e9fb2f74b5333e6f8f60de11ea6adce497252648",
"triggerType" : "PUSH"
}, {
"hash" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18493",
"triggerID" : "74451cd8ee26909fac98e88850ba352c8c415ef5",
"triggerType" : "PUSH"
}, {
"hash" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18525",
"triggerID" : "37dd6d9ce0a638d4274d66b67bd0799ee7d3daf5",
"triggerType" : "PUSH"
}, {
"hash" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18529",
"triggerID" : "89a1c11b7cdf452c37c190ee11934c7546c34c25",
"triggerType" : "PUSH"
}, {
"hash" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18537",
"triggerID" : "5f98b8a27f2948e7f3a0f8673606b793b90729db",
"triggerType" : "PUSH"
}, {
"hash" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18543",
"triggerID" : "0e82dedc198ec3ed2159e346dd37a3b10d493520",
"triggerType" : "PUSH"
}, {
"hash" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18555",
"triggerID" : "de31bf2ee6551ac5212dc30d7adcba3e2873f0d5",
"triggerType" : "PUSH"
}, {
"hash" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18580",
"triggerID" : "2f9aa542076faa188839bc55b43dd7f22ec32b62",
"triggerType" : "PUSH"
}, {
"hash" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18677",
"triggerID" : "fa5c3f22ad50c6bdf4cf8fa04f51ecfba1cd8905",
"triggerType" : "PUSH"
}, {
"hash" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18683",
"triggerID" : "93c8dc9a2683c82d7abfd33cc26c01eee78722a1",
"triggerType" : "PUSH"
}, {
"hash" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18688",
"triggerID" : "80f3d5256ccfc42838f274d47357f750b5f967bd",
"triggerType" : "PUSH"
}, {
"hash" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18698",
"triggerID" : "a1242a526d022ee3346d593f74421c2dd3d4fcc2",
"triggerType" : "PUSH"
}, {
"hash" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18699",
"triggerID" : "9b6e75df6b03096ab3e461da8c6b269644e87f10",
"triggerType" : "PUSH"
}, {
"hash" : "d121977ee5e8e7785acf8a0ef82563b971b5ff27",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18711",
"triggerID" : "d121977ee5e8e7785acf8a0ef82563b971b5ff27",
"triggerType" : "PUSH"
}, {
"hash" : "687afd0dcc56a8dd1266486d452656fa3abbf0e4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18723",
"triggerID" : "687afd0dcc56a8dd1266486d452656fa3abbf0e4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d121977ee5e8e7785acf8a0ef82563b971b5ff27 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18711)
* 687afd0dcc56a8dd1266486d452656fa3abbf0e4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=18723)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org