You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "cshuo (via GitHub)" <gi...@apache.org> on 2026/09/29 04:02:55 UTC
[PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
cshuo opened a new pull request, #20133:
URL: https://github.com/apache/hudi/pull/20133
### Describe the issue this Pull Request addresses
Closes #20132.
When a native data or delete file reaches its size limit, writing the next record triggers a rollover flush inside `writeRecord()`. With `hoodie.write.ignore.failed=true`, a flush failure is currently treated as a failure of just that record, and the loop can continue using a writer whose batch was not finalized successfully.
Flush failures affect the buffered batch and must propagate regardless of the record-level ignore setting.
### Summary and Changelog
- Introduce `HoodieLogFlushException` extending `HoodieException` to distinguish flush failures from generic append errors.
- Wrap native flush and append-result finalization failures with this exception while preserving the original cause.
- Make `HoodieAppendHandle` close the handle and propagate flush failures even when failed record writes may be ignored.
- Add regression cases for data/delete rollover, I/O exceptions, runtime exceptions, errors, suppressed cleanup failures, and stopping before the next record. Verify ordinary I/O and generic append errors still honor the ignore setting.
Validation: all 20 tests in `TestHoodieAppendHandle` and `TestHoodieNativeLogAppendHandle` pass, with no failures, errors, or skips. Checkstyle passes.
```bash
mvn -o -pl hudi-client/hudi-client-common -am \
-Dtest=TestHoodieAppendHandle,TestHoodieNativeLogAppendHandle \
-Dsurefire.failIfNoSpecifiedTests=false \
-DskipITs -DskipSparkTests -DskipScalaTests test
```
### Impact
Native log flush failures abort the write handle instead of being recorded as isolated record failures. Ordinary record failures retain their existing behavior. No new configuration is introduced; the dedicated exception type extends `HoodieException` directly.
### Risk Level
Low. The behavior change is limited to native log flush failure propagation. Tests exercise both data and delete rollover and verify that generic append errors remain ignorable.
### Documentation Update
The new exception's Javadoc documents that flush failures affect the batch and must abort the handle even when individual record failures may be ignored.
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20133:
URL: https://github.com/apache/hudi/pull/20133#issuecomment-5884852635
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b660bde916b756b7fe420a22628c897a659bebc7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343",
"triggerID" : "b660bde916b756b7fe420a22628c897a659bebc7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b660bde916b756b7fe420a22628c897a659bebc7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20133:
URL: https://github.com/apache/hudi/pull/20133#issuecomment-5885354784
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b660bde916b756b7fe420a22628c897a659bebc7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343",
"triggerID" : "b660bde916b756b7fe420a22628c897a659bebc7",
"triggerType" : "PUSH"
}, {
"hash" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17350",
"triggerID" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b660bde916b756b7fe420a22628c897a659bebc7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343)
* 2e48b24850b8a71372afbdb9d44ec7c3d8a064d9 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17350)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20133:
URL: https://github.com/apache/hudi/pull/20133#issuecomment-5892267507
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b660bde916b756b7fe420a22628c897a659bebc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343",
"triggerID" : "b660bde916b756b7fe420a22628c897a659bebc7",
"triggerType" : "PUSH"
}, {
"hash" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17350",
"triggerID" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"triggerType" : "PUSH"
}, {
"hash" : "ec7cb26760441f2665d5c8a9941a74c99c1e6658",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17351",
"triggerID" : "ec7cb26760441f2665d5c8a9941a74c99c1e6658",
"triggerType" : "PUSH"
}, {
"hash" : "d0550228d47003c658ce355a0ca5b765440e449c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17356",
"triggerID" : "d0550228d47003c658ce355a0ca5b765440e449c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d0550228d47003c658ce355a0ca5b765440e449c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17356)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20133:
URL: https://github.com/apache/hudi/pull/20133#issuecomment-5885255413
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b660bde916b756b7fe420a22628c897a659bebc7",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343",
"triggerID" : "b660bde916b756b7fe420a22628c897a659bebc7",
"triggerType" : "PUSH"
}, {
"hash" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b660bde916b756b7fe420a22628c897a659bebc7 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343)
* 2e48b24850b8a71372afbdb9d44ec7c3d8a064d9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #20133:
URL: https://github.com/apache/hudi/pull/20133#discussion_r4129550698
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieNativeLogAppendHandle.java:
##########
@@ -162,8 +163,8 @@ protected void flushAppend() {
writer.flushAppend(getUpdatedHeader(header));
processAppendResults(writer.getLastAppendResults());
}
- } catch (IOException e) {
- throw new HoodieAppendException("Failed while flushing records to native log for fileId " + fileId, e);
+ } catch (Throwable e) {
Review Comment:
🤖 Catching `Throwable` here also wraps JVM-fatal errors like `OutOfMemoryError` or `StackOverflowError` in a `RuntimeException`. The engines then see an ordinary task failure instead of a fatal error. Would `catch (Exception e)` be enough, or could `VirtualMachineError` be rethrown unwrapped? `writeRecord`/`doAppend` would still close the writer through their own handling.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #20133:
URL: https://github.com/apache/hudi/pull/20133#discussion_r4140204087
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieNativeLogAppendHandle.java:
##########
@@ -162,7 +162,7 @@ protected void flushAppend() {
writer.flushAppend(getUpdatedHeader(header));
processAppendResults(writer.getLastAppendResults());
}
- } catch (IOException e) {
+ } catch (Exception e) {
Review Comment:
This is fixed by #20118, which has already merged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #20133:
URL: https://github.com/apache/hudi/pull/20133#discussion_r4131867790
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/exception/HoodieLogFlushException.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.exception;
+
+/**
+ * Failure while finalizing a log append. Since it affects the entire buffered batch, this failure must abort
+ * the write handle even when individual record write failures may be ignored.
+ */
+public class HoodieLogFlushException extends HoodieException {
+
+ public HoodieLogFlushException(String message, Throwable cause) {
Review Comment:
we can continue to use HoodieAppendException, it seems this exception is introduced for general exception of append IO from its doc: "Exception thrown for any higher level errors when <code>HoodieClient</code> is doing a delta commit."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20133:
URL: https://github.com/apache/hudi/pull/20133#issuecomment-5886565428
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b660bde916b756b7fe420a22628c897a659bebc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343",
"triggerID" : "b660bde916b756b7fe420a22628c897a659bebc7",
"triggerType" : "PUSH"
}, {
"hash" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17350",
"triggerID" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"triggerType" : "PUSH"
}, {
"hash" : "ec7cb26760441f2665d5c8a9941a74c99c1e6658",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17351",
"triggerID" : "ec7cb26760441f2665d5c8a9941a74c99c1e6658",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec7cb26760441f2665d5c8a9941a74c99c1e6658 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17351)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20133:
URL: https://github.com/apache/hudi/pull/20133#issuecomment-5885975767
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b660bde916b756b7fe420a22628c897a659bebc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343",
"triggerID" : "b660bde916b756b7fe420a22628c897a659bebc7",
"triggerType" : "PUSH"
}, {
"hash" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17350",
"triggerID" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"triggerType" : "PUSH"
}, {
"hash" : "ec7cb26760441f2665d5c8a9941a74c99c1e6658",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "ec7cb26760441f2665d5c8a9941a74c99c1e6658",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e48b24850b8a71372afbdb9d44ec7c3d8a064d9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17350)
* ec7cb26760441f2665d5c8a9941a74c99c1e6658 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #20133:
URL: https://github.com/apache/hudi/pull/20133
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20133:
URL: https://github.com/apache/hudi/pull/20133#issuecomment-5883570366
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20133?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `7 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.41%. Comparing base ([`39d05f5`](https://app.codecov.io/gh/apache/hudi/commit/39d05f53713ff07c9f4bebc8384b109bd54f23cb?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b660bde`](https://app.codecov.io/gh/apache/hudi/commit/b660bde916b756b7fe420a22628c897a659bebc7?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20133?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://app.codecov.io/gh/apache/hudi/pull/20133?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2FHoodieAppendHandle.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | 0.00% | [3 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20133?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...apache/hudi/exception/HoodieLogFlushException.java](https://app.codecov.io/gh/apache/hudi/pull/20133?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fexception%2FHoodieLogFlushException.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2V4Y2VwdGlvbi9Ib29kaWVMb2dGbHVzaEV4Y2VwdGlvbi5qYXZh) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20133?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/io/HoodieNativeLogAppendHandle.java](https://app.codecov.io/gh/apache/hudi/pull/20133?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2FHoodieNativeLogAppendHandle.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU5hdGl2ZUxvZ0FwcGVuZEhhbmRsZS5qYXZh) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20133?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20133 +/- ##
============================================
- Coverage 80.41% 80.41% -0.01%
+ Complexity 34897 34896 -1
============================================
Files 2546 2547 +1
Lines 142929 142933 +4
Branches 17383 17385 +2
============================================
- Hits 114938 114934 -4
- Misses 20076 20084 +8
Partials 7915 7915
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.97% <ø> (ø)` | |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.45% <0.00%> (-0.03%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.74% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.85% <ø> (ø)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.17% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.05% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.99% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.02% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.50% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.06% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20133/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20133/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20133/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.98% <0.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20133?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...apache/hudi/exception/HoodieLogFlushException.java](https://app.codecov.io/gh/apache/hudi/pull/20133?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fexception%2FHoodieLogFlushException.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2V4Y2VwdGlvbi9Ib29kaWVMb2dGbHVzaEV4Y2VwdGlvbi5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/hudi/io/HoodieNativeLogAppendHandle.java](https://app.codecov.io/gh/apache/hudi/pull/20133?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2FHoodieNativeLogAppendHandle.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZU5hdGl2ZUxvZ0FwcGVuZEhhbmRsZS5qYXZh) | `94.17% <0.00%> (-1.95%)` | :arrow_down: |
| [...in/java/org/apache/hudi/io/HoodieAppendHandle.java](https://app.codecov.io/gh/apache/hudi/pull/20133?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fio%2FHoodieAppendHandle.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2lvL0hvb2RpZUFwcGVuZEhhbmRsZS5qYXZh) | `80.59% <0.00%> (-0.61%)` | :arrow_down: |
... and [1 file with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20133/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20133:
URL: https://github.com/apache/hudi/pull/20133#issuecomment-5883653818
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b660bde916b756b7fe420a22628c897a659bebc7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b660bde916b756b7fe420a22628c897a659bebc7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b660bde916b756b7fe420a22628c897a659bebc7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #20133:
URL: https://github.com/apache/hudi/pull/20133#discussion_r4133103905
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/exception/HoodieLogFlushException.java:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.exception;
+
+/**
+ * Failure while finalizing a log append. Since it affects the entire buffered batch, this failure must abort
+ * the write handle even when individual record write failures may be ignored.
+ */
+public class HoodieLogFlushException extends HoodieException {
+
+ public HoodieLogFlushException(String message, Throwable cause) {
Review Comment:
Updated to reuse HoodieAppendException and removed the new exception class. All 19 targeted tests pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #20133:
URL: https://github.com/apache/hudi/pull/20133#discussion_r4135371785
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##########
@@ -401,7 +402,10 @@ protected boolean writeRecord(HoodieRecord<T> hoodieRecord) {
return true;
} catch (Exception e) {
log.error("Error writing record {}", hoodieRecord, e);
- if (!config.getIgnoreWriteFailed() || ExceptionUtil.isCausedBy(e, HoodieEarlyConflictDetectionException.class)) {
+ // A failed log flush affects the entire buffered batch, not just the current record.
+ if (!config.getIgnoreWriteFailed()
+ || ExceptionUtil.isCausedBy(e, HoodieAppendException.class)
Review Comment:
🤖 Only rollover flush failures are non-ignorable here. Can `appendRecord`/`appendDeleteRecord` also break the whole batch? E.g. the native parquet writer fails while flushing a full row group, or `ensureDataFileWriter` fails after `dataLogFile` is set. With ignore-failed on, the loop keeps going on that writer, and `testRecordWriteFailureCanStillBeIgnored` locks that in.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieNativeLogAppendHandle.java:
##########
@@ -162,7 +162,7 @@ protected void flushAppend() {
writer.flushAppend(getUpdatedHeader(header));
processAppendResults(writer.getLastAppendResults());
}
- } catch (IOException e) {
+ } catch (Exception e) {
Review Comment:
🤖 If `dataFileWriter.close()` throws inside `flushAppend`→`closeFileWriters`, `dataFileWriter` never gets nulled and `deleteFileWriter` never gets closed. `closeLogWriterQuietly` then calls `writer.close()`, which closes the data writer a second time and can throw again before the delete writer is closed. Could that leak the delete file's output stream?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20133:
URL: https://github.com/apache/hudi/pull/20133#issuecomment-5883720110
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b660bde916b756b7fe420a22628c897a659bebc7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343",
"triggerID" : "b660bde916b756b7fe420a22628c897a659bebc7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b660bde916b756b7fe420a22628c897a659bebc7 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20133:
URL: https://github.com/apache/hudi/pull/20133#issuecomment-5885730026
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b660bde916b756b7fe420a22628c897a659bebc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343",
"triggerID" : "b660bde916b756b7fe420a22628c897a659bebc7",
"triggerType" : "PUSH"
}, {
"hash" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17350",
"triggerID" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 2e48b24850b8a71372afbdb9d44ec7c3d8a064d9 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17350)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20133:
URL: https://github.com/apache/hudi/pull/20133#issuecomment-5890193349
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b660bde916b756b7fe420a22628c897a659bebc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343",
"triggerID" : "b660bde916b756b7fe420a22628c897a659bebc7",
"triggerType" : "PUSH"
}, {
"hash" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17350",
"triggerID" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"triggerType" : "PUSH"
}, {
"hash" : "ec7cb26760441f2665d5c8a9941a74c99c1e6658",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17351",
"triggerID" : "ec7cb26760441f2665d5c8a9941a74c99c1e6658",
"triggerType" : "PUSH"
}, {
"hash" : "d0550228d47003c658ce355a0ca5b765440e449c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d0550228d47003c658ce355a0ca5b765440e449c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec7cb26760441f2665d5c8a9941a74c99c1e6658 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17351)
* d0550228d47003c658ce355a0ca5b765440e449c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on code in PR #20133:
URL: https://github.com/apache/hudi/pull/20133#discussion_r4140215743
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/io/HoodieAppendHandle.java:
##########
@@ -401,7 +402,10 @@ protected boolean writeRecord(HoodieRecord<T> hoodieRecord) {
return true;
} catch (Exception e) {
log.error("Error writing record {}", hoodieRecord, e);
- if (!config.getIgnoreWriteFailed() || ExceptionUtil.isCausedBy(e, HoodieEarlyConflictDetectionException.class)) {
+ // A failed log flush affects the entire buffered batch, not just the current record.
+ if (!config.getIgnoreWriteFailed()
+ || ExceptionUtil.isCausedBy(e, HoodieAppendException.class)
Review Comment:
An append for native writer can trigger an internal row-group flush, so an IOException does not necessarily affect only the current record, and a subsequent successful write would not establish that the buffered batch is intact. However, appendRecord also includes record conversion, so making all its failures fatal would conflate record-level errors with writer failures. This is a pre-existing gap that needs a separate follow-up to establish that boundary and revisit the mock-based test. This PR remains scoped to explicit flushAppend failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(common): propagate native log flush failures despite ignore failed writes [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20133:
URL: https://github.com/apache/hudi/pull/20133#issuecomment-5890307032
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b660bde916b756b7fe420a22628c897a659bebc7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17343",
"triggerID" : "b660bde916b756b7fe420a22628c897a659bebc7",
"triggerType" : "PUSH"
}, {
"hash" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17350",
"triggerID" : "2e48b24850b8a71372afbdb9d44ec7c3d8a064d9",
"triggerType" : "PUSH"
}, {
"hash" : "ec7cb26760441f2665d5c8a9941a74c99c1e6658",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17351",
"triggerID" : "ec7cb26760441f2665d5c8a9941a74c99c1e6658",
"triggerType" : "PUSH"
}, {
"hash" : "d0550228d47003c658ce355a0ca5b765440e449c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17356",
"triggerID" : "d0550228d47003c658ce355a0ca5b765440e449c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* ec7cb26760441f2665d5c8a9941a74c99c1e6658 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17351)
* d0550228d47003c658ce355a0ca5b765440e449c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17356)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org