You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/12/08 09:44:59 UTC
[GitHub] [hudi] hbgstc123 opened a new pull request, #7408: [HUDI-5350]fix oom cause compaction event lost problem.
hbgstc123 opened a new pull request, #7408:
URL: https://github.com/apache/hudi/pull/7408
### Change Logs
code from `org.apache.hudi.sink.compact.CompactOperator`:
` executor.execute(
() -> doCompaction(instantTime, compactionOperation, collector, reloadWriteConfig()),
(errMsg, t) -> collector.collect(new CompactionCommitEvent(instantTime, compactionOperation.getFileId(), taskID)),
"Execute compaction for instant %s from task %d", instantTime, taskID);
`
In flink inline async compaction, if OOM error happen during execution of `doCompaction(...)`, exception hook will not be executed, and no CompactionCommitEvent is sent to `CompactionCommitSink`. Result in compaction instant stuck in "inflight" state, never succeed or rollback, in fact the compaction is failed.
I change the execute sequence: run ExceptionHook before re-throw FatalErrorOrOOM.
I tested and it can sent `CompactionCommitEvent` to compaction sink to rollback the failed compaction when OOM happen during compaction.
### Impact
no
### Risk level (write none, low medium or high below)
low
### Documentation Update
no
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7408: [HUDI-5350]fix oom cause compaction event lost problem.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7408:
URL: https://github.com/apache/hudi/pull/7408#issuecomment-1343799419
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "53bde66487d957954bb83a2db2d7820e47dc0c25",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13550",
"triggerID" : "53bde66487d957954bb83a2db2d7820e47dc0c25",
"triggerType" : "PUSH"
}, {
"hash" : "913d5701f4208b7499c6a537a20e8bccff7bef8a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13568",
"triggerID" : "913d5701f4208b7499c6a537a20e8bccff7bef8a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53bde66487d957954bb83a2db2d7820e47dc0c25 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13550)
* 913d5701f4208b7499c6a537a20e8bccff7bef8a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13568)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hbgstc123 commented on a diff in pull request #7408: [HUDI-5350]fix oom cause compaction event lost problem.
Posted by GitBox <gi...@apache.org>.
hbgstc123 commented on code in PR #7408:
URL: https://github.com/apache/hudi/pull/7408#discussion_r1044035426
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/NonThrownExecutor.java:
##########
@@ -136,15 +136,15 @@ private <E extends Throwable> Runnable wrapAction(
}
private void handleException(Throwable t, ExceptionHook hook, Supplier<String> actionString) {
+ final String errMsg = String.format("Executor executes action [%s] error", actionString.get());
+ if (hook != null) {
+ hook.apply(errMsg, t);
+ }
// if we have a JVM critical error, promote it immediately, there is a good
// chance the
// logging or job failing will not succeed any more
ExceptionUtils.rethrowIfFatalErrorOrOOM(t);
- final String errMsg = String.format("Executor executes action [%s] error", actionString.get());
logger.error(errMsg, t);
- if (hook != null) {
Review Comment:
thanks for review, moved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7408: [HUDI-5350]fix oom cause compaction event lost problem.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7408:
URL: https://github.com/apache/hudi/pull/7408#issuecomment-1342500158
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "53bde66487d957954bb83a2db2d7820e47dc0c25",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "53bde66487d957954bb83a2db2d7820e47dc0c25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53bde66487d957954bb83a2db2d7820e47dc0c25 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7408: [HUDI-5350]fix oom cause compaction event lost problem.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7408:
URL: https://github.com/apache/hudi/pull/7408#issuecomment-1343794440
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "53bde66487d957954bb83a2db2d7820e47dc0c25",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13550",
"triggerID" : "53bde66487d957954bb83a2db2d7820e47dc0c25",
"triggerType" : "PUSH"
}, {
"hash" : "913d5701f4208b7499c6a537a20e8bccff7bef8a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "913d5701f4208b7499c6a537a20e8bccff7bef8a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53bde66487d957954bb83a2db2d7820e47dc0c25 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13550)
* 913d5701f4208b7499c6a537a20e8bccff7bef8a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7408: [HUDI-5350]fix oom cause compaction event lost problem.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7408:
URL: https://github.com/apache/hudi/pull/7408#issuecomment-1342510017
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "53bde66487d957954bb83a2db2d7820e47dc0c25",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13550",
"triggerID" : "53bde66487d957954bb83a2db2d7820e47dc0c25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53bde66487d957954bb83a2db2d7820e47dc0c25 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13550)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7408: [HUDI-5350]fix oom cause compaction event lost problem.
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #7408:
URL: https://github.com/apache/hudi/pull/7408#issuecomment-1342947319
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "53bde66487d957954bb83a2db2d7820e47dc0c25",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13550",
"triggerID" : "53bde66487d957954bb83a2db2d7820e47dc0c25",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 53bde66487d957954bb83a2db2d7820e47dc0c25 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=13550)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #7408: [HUDI-5350]fix oom cause compaction event lost problem.
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #7408:
URL: https://github.com/apache/hudi/pull/7408#discussion_r1043978608
##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/utils/NonThrownExecutor.java:
##########
@@ -136,15 +136,15 @@ private <E extends Throwable> Runnable wrapAction(
}
private void handleException(Throwable t, ExceptionHook hook, Supplier<String> actionString) {
+ final String errMsg = String.format("Executor executes action [%s] error", actionString.get());
+ if (hook != null) {
+ hook.apply(errMsg, t);
+ }
// if we have a JVM critical error, promote it immediately, there is a good
// chance the
// logging or job failing will not succeed any more
ExceptionUtils.rethrowIfFatalErrorOrOOM(t);
- final String errMsg = String.format("Executor executes action [%s] error", actionString.get());
logger.error(errMsg, t);
- if (hook != null) {
Review Comment:
Can also move the log though ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #7408: [HUDI-5350] Fix oom cause compaction event lost problem
Posted by GitBox <gi...@apache.org>.
danny0405 merged PR #7408:
URL: https://github.com/apache/hudi/pull/7408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org