You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2023/08/10 20:19:33 UTC
[GitHub] [hudi] yihua opened a new pull request, #9419: [HUDI-6679] Fix initialization of metadata table partitions upon failure
yihua opened a new pull request, #9419:
URL: https://github.com/apache/hudi/pull/9419
### Change Logs
This PR fixes initialization of metadata table partitions upon failure:
- In `BaseHoodieTableServiceClient.rollbackFailedWrites`, the fix avoids bootstrap rollback logic for MDT as MDT is never a bootstrap table and such logic can be accidentally triggered, since the MDT initial commits, e.g., `00000000000000010`, `00000000000000011`, are smaller than `FULL_BOOTSTRAP_INSTANT_TS` (`00000000000002`).
- In `HoodieBackedTableMetadataWriter.initializeIfNeeded`, when async metadata indexing is disabled, if a partition is inflight, it means that the partition is not fully initialized, so the initialization should be triggered again.
This scenario fails before the fix: When both files and record_index partitions are enabled, for the first commit in the data table, the transaction fails when initializing the second partition in the MDT. In this case, the timelines look like below. In this case, when restarting the pipeline, the rollback triggers irrelevant bootstrap rollback logic causing MDT to be corrupted, not properly re-initializing the record_index partition.
DT
```
<commit_time>.commit.requested
<commit_time>.commit.inflight
```
MDT
```
00000000000000010.deltacommit.requested
00000000000000010.deltacommit.inflight
00000000000000010.deltacommit
00000000000000011.deltacommit.requested
00000000000000011.deltacommit.inflight
```
Afterwards, `00000000000000010` is rolled back and bootstrap rollback logic adding restore kicks in, which are unexpected.
```
╔═════╤══════════════════════╤═══════════════════╤═══════════╤════════════════╤════════════════╤════════════════╤═════════════╤═══════════╤════════════════╤════════════════╤════════════════╗
║ No. │ Instant │ Action │ State │ Requested │ Inflight │ Completed │ MT │ MT │ MT │ MT │ MT ║
║ │ │ │ │ Time │ Time │ Time │ Action │ State │ Requested │ Inflight │ Completed ║
║ │ │ │ │ │ │ │ │ │ Time │ Time │ Time ║
╠═════╪══════════════════════╪═══════════════════╪═══════════╪════════════════╪════════════════╪════════════════╪═════════════╪═══════════╪════════════════╪════════════════╪════════════════╣
║ 0 │ 20230807063905364 │ rollback │ COMPLETED │ 08-06 23:39:06 │ 08-06 23:39:07 │ 08-06 23:40:38 │ - │ - │ - │ - │ - ║
║ │ │ Rolls back │ │ │ │ │ │ │ │ │ ║
║ │ │ 20230807063647472 │ │ │ │ │ │ │ │ │ ║
╟─────┼──────────────────────┼───────────────────┼───────────┼────────────────┼────────────────┼────────────────┼─────────────┼───────────┼────────────────┼────────────────┼────────────────╢
║ 1 │ 20230807063905364010 │ - │ - │ - │ - │ - │ deltacommit │ COMPLETED │ 08-06 23:40:49 │ 08-06 23:40:49 │ 08-06 23:40:51 ║
╟─────┼──────────────────────┼───────────────────┼───────────┼────────────────┼────────────────┼────────────────┼─────────────┼───────────┼────────────────┼────────────────┼────────────────╢
║ 2 │ 20230807064006967 │ deltacommit │ REQUESTED │ 08-06 23:40:39 │ - │ - │ - │ - │ - │ - │ - ║
║ │ │ Rolled back by │ │ │ │ │ │ │ │ │ ║
║ │ │ 20230807064227290 │ │ │ │ │ │ │ │ │ ║
╟─────┼──────────────────────┼───────────────────┼───────────┼────────────────┼────────────────┼────────────────┼─────────────┼───────────┼────────────────┼────────────────┼────────────────╢
║ 3 │ 20230807064041714 │ - │ - │ - │ - │ - │ restore │ COMPLETED │ 08-06 23:40:43 │ 08-06 23:40:43 │ 08-06 23:40:48 ║
╟─────┼──────────────────────┼───────────────────┼───────────┼────────────────┼────────────────┼────────────────┼─────────────┼───────────┼────────────────┼────────────────┼────────────────╢
║ 4 │ 20230807064227290 │ rollback │ INFLIGHT │ 08-06 23:42:28 │ 08-06 23:42:29 │ - │ - │ - │ - │ - │ - ║
║ │ │ Rolls back │ │ │ │ │ │ │ │ │ ║
║ │ │ 20230807064006967 │ │ │ │ │ │ │ │ │ ║
╚═════╧══════════════════════╧═══════════════════╧═══════════╧════════════════╧════════════════╧════════════════╧═════════════╧═══════════╧════════════════╧════════════════╧════════════════╝
```
Exception:
```
org.apache.hudi.exception.HoodieRollbackException: Failed to rollback s3a://<base_path>/hoodie_table commits 20230807064006967
at org.apache.hudi.client.BaseHoodieTableServiceClient.rollback(BaseHoodieTableServiceClient.java:918)
at org.apache.hudi.client.BaseHoodieTableServiceClient.rollback(BaseHoodieTableServiceClient.java:865)
at org.apache.hudi.client.BaseHoodieTableServiceClient.rollbackFailedWrites(BaseHoodieTableServiceClient.java:739)
at org.apache.hudi.client.BaseHoodieTableServiceClient.rollbackFailedWrites(BaseHoodieTableServiceClient.java:723)
at org.apache.hudi.client.BaseHoodieTableServiceClient.rollbackFailedWrites(BaseHoodieTableServiceClient.java:718)
at org.apache.hudi.client.BaseHoodieWriteClient.lambda$startCommitWithTime$97cdbdca$1(BaseHoodieWriteClient.java:928)
at org.apache.hudi.common.util.CleanerUtils.rollbackFailedWrites(CleanerUtils.java:222)
at org.apache.hudi.client.BaseHoodieWriteClient.startCommitWithTime(BaseHoodieWriteClient.java:927)
at org.apache.hudi.client.BaseHoodieWriteClient.startCommitWithTime(BaseHoodieWriteClient.java:920)
at org.apache.hudi.utilities.streamer.StreamSync.startCommit(StreamSync.java:890)
at org.apache.hudi.utilities.streamer.StreamSync.writeToSink(StreamSync.java:767)
at org.apache.hudi.utilities.streamer.StreamSync.syncOnce(StreamSync.java:445)
at org.apache.hudi.utilities.streamer.HoodieStreamer$StreamSyncService.lambda$startService$1(HoodieStreamer.java:767)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalArgumentException: FileGroup count for MDT partition files should be >0
at org.apache.hudi.common.util.ValidationUtils.checkArgument(ValidationUtils.java:42)
at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.prepRecords(HoodieBackedTableMetadataWriter.java:1098)
at org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.commitInternal(SparkHoodieBackedTableMetadataWriter.java:135)
at org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.commit(SparkHoodieBackedTableMetadataWriter.java:122)
at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.processAndCommit(HoodieBackedTableMetadataWriter.java:837)
at org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:1013)
at org.apache.hudi.table.action.BaseActionExecutor.lambda$writeTableMetadata$2(BaseActionExecutor.java:77)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
at org.apache.hudi.table.action.BaseActionExecutor.writeTableMetadata(BaseActionExecutor.java:77)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.finishRollback(BaseRollbackActionExecutor.java:264)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.runRollback(BaseRollbackActionExecutor.java:120)
at org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:141)
at org.apache.hudi.table.HoodieSparkMergeOnReadTable.rollback(HoodieSparkMergeOnReadTable.java:218)
at org.apache.hudi.client.BaseHoodieTableServiceClient.rollback(BaseHoodieTableServiceClient.java:901)
... 16 more
```
### Impact
Makes the initialization of metadata table partitions more robust against failures
### Risk level (write none, low medium or high below)
low
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9419: [HUDI-6679] Fix initialization of metadata table partitions upon failure
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9419:
URL: https://github.com/apache/hudi/pull/9419#issuecomment-1673925975
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "060ce5fe9068a6b38382735d7aa60f3cd40c7e16",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19252",
"triggerID" : "060ce5fe9068a6b38382735d7aa60f3cd40c7e16",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 060ce5fe9068a6b38382735d7aa60f3cd40c7e16 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19252)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9419: [HUDI-6679] Fix initialization of metadata table partitions upon failure
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9419:
URL: https://github.com/apache/hudi/pull/9419#issuecomment-1673916580
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "060ce5fe9068a6b38382735d7aa60f3cd40c7e16",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "060ce5fe9068a6b38382735d7aa60f3cd40c7e16",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 060ce5fe9068a6b38382735d7aa60f3cd40c7e16 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope merged pull request #9419: [HUDI-6679] Fix initialization of metadata table partitions upon failure
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope merged PR #9419:
URL: https://github.com/apache/hudi/pull/9419
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9419: [HUDI-6679] Fix initialization of metadata table partitions upon failure
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9419:
URL: https://github.com/apache/hudi/pull/9419#issuecomment-1674072360
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "060ce5fe9068a6b38382735d7aa60f3cd40c7e16",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19252",
"triggerID" : "060ce5fe9068a6b38382735d7aa60f3cd40c7e16",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 060ce5fe9068a6b38382735d7aa60f3cd40c7e16 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19252)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on pull request #9419: [HUDI-6679] Fix initialization of metadata table partitions upon failure
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope commented on PR #9419:
URL: https://github.com/apache/hudi/pull/9419#issuecomment-1674139030
I am landing it to save CI cycles. There are no failures in UT FT other modules. It's just timing out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org