You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/13 10:23:17 UTC
[GitHub] [hudi] wxplovecc opened a new pull request, #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
wxplovecc opened a new pull request, #6939:
URL: https://github.com/apache/hudi/pull/6939
### Change Logs
Currently when rollOver happened in rollback was not create new log file ,thus caused below exception
some test log:
2022-10-13 16:58:54,613 INFO org.apache.hudi.common.table.log.HoodieLogFormatWriter [] - HoodieLogFile{pathStr='viewfs://dcfs/ns-common/car/dws/dws_order_info_by_flinkbatch_history/2022-10-12/.00000002-1251-4f1c-8f75-71ff51071ee3_20221013052439696.log.1_2-4-0', fileLen=0} exists. Appending to existing file
2022-10-13 16:58:54,974 INFO org.apache.hudi.table.action.rollback.BaseRollbackHelper [] - after testrollback writer.LogFile: viewfs://dcfs/ns-common/car/dws/dws_order_info_by_flinkbatch_history/2022-10-12/.00000002-1251-4f1c-8f75-71ff51071ee3_20221013052439696.log.2_1-0-1
### Impact
no
**Risk level: none | low | medium | high**
low
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6939:
URL: https://github.com/apache/hudi/pull/6939#issuecomment-1277396515
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b46f2d69debcc6144dd0c27a990cd4298d65675f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6939:
URL: https://github.com/apache/hudi/pull/6939#issuecomment-1306707167
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12188",
"triggerID" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"triggerType" : "PUSH"
}, {
"hash" : "33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12867",
"triggerID" : "33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b46f2d69debcc6144dd0c27a990cd4298d65675f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12188)
* 33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12867)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6939:
URL: https://github.com/apache/hudi/pull/6939#discussion_r1024712316
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackHelper.java:
##########
@@ -135,7 +136,10 @@ List<Pair<String, HoodieRollbackStat>> maybeDeleteAndCollectStats(HoodieEngineCo
if (doDelete) {
Map<HoodieLogBlock.HeaderMetadataType, String> header = generateHeader(instantToRollback.getTimestamp());
// if update belongs to an existing log file
- writer.appendBlock(new HoodieCommandBlock(header));
+ // use the log file path from AppendResult in case the file handle may roll over
+ filePath = writer.appendBlock(new HoodieCommandBlock(header)).logFile().getPath();
+ } else {
+ filePath = writer.getLogFile().getPath();
Review Comment:
We need to, seems the helper does not have any UT yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6939:
URL: https://github.com/apache/hudi/pull/6939#issuecomment-1308111634
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12188",
"triggerID" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"triggerType" : "PUSH"
}, {
"hash" : "33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12867",
"triggerID" : "33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd",
"triggerType" : "PUSH"
}, {
"hash" : "33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12886",
"triggerID" : "1308096230",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12867) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12886)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6939:
URL: https://github.com/apache/hudi/pull/6939#issuecomment-1277697640
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12188",
"triggerID" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b46f2d69debcc6144dd0c27a990cd4298d65675f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6939:
URL: https://github.com/apache/hudi/pull/6939#issuecomment-1277406691
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12188",
"triggerID" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b46f2d69debcc6144dd0c27a990cd4298d65675f Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12188)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6939:
URL: https://github.com/apache/hudi/pull/6939#discussion_r1024558833
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/rollback/BaseRollbackHelper.java:
##########
@@ -135,7 +136,10 @@ List<Pair<String, HoodieRollbackStat>> maybeDeleteAndCollectStats(HoodieEngineCo
if (doDelete) {
Map<HoodieLogBlock.HeaderMetadataType, String> header = generateHeader(instantToRollback.getTimestamp());
// if update belongs to an existing log file
- writer.appendBlock(new HoodieCommandBlock(header));
+ // use the log file path from AppendResult in case the file handle may roll over
+ filePath = writer.appendBlock(new HoodieCommandBlock(header)).logFile().getPath();
+ } else {
+ filePath = writer.getLogFile().getPath();
Review Comment:
Could we add a unit test for the logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #6939:
URL: https://github.com/apache/hudi/pull/6939#issuecomment-1306588951
Nice catch ~, i have reviewed and applied a patch:
[5025.patch.zip](https://github.com/apache/hudi/files/9957426/5025.patch.zip)
Can you rebase with the latest master and force-push again ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6939:
URL: https://github.com/apache/hudi/pull/6939#issuecomment-1306703223
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12188",
"triggerID" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"triggerType" : "PUSH"
}, {
"hash" : "33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b46f2d69debcc6144dd0c27a990cd4298d65675f Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12188)
* 33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6939:
URL: https://github.com/apache/hudi/pull/6939#issuecomment-1306980300
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12188",
"triggerID" : "b46f2d69debcc6144dd0c27a990cd4298d65675f",
"triggerType" : "PUSH"
}, {
"hash" : "33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12867",
"triggerID" : "33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 33e1c0d4e837080818ddd7faf5517c7f0c2ca3dd Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12867)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
danny0405 commented on PR #6939:
URL: https://github.com/apache/hudi/pull/6939#issuecomment-1308096230
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6939:
URL: https://github.com/apache/hudi/pull/6939#discussion_r1016121108
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/HoodieLogFormatWriter.java:
##########
@@ -220,6 +220,7 @@ private void rolloverIfNeeded() throws IOException {
LOG.info("CurrentSize " + getCurrentSize() + " has reached threshold " + sizeThreshold
+ ". Rolling over to the next version");
rollOver();
+ createNewFile();
}
Review Comment:
We better not generate a new physical file handle when rollover, the good strategy is: only generate a new file handle when we want to write real data set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #6939: [HUDI-5025] Rollback failed with log file not found when rollOver in rollback process
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #6939:
URL: https://github.com/apache/hudi/pull/6939
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org