You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "wombatu-kun (via GitHub)" <gi...@apache.org> on 2026/06/17 05:50:51 UTC
[PR] test(flink): retry short CollectSink reads to de-flake stream-read ITs [hudi]
wombatu-kun opened a new pull request, #19030:
URL: https://github.com/apache/hudi/pull/19030
### Describe the issue this Pull Request addresses
`ITTestHoodieDataSource.testStreamReadFromSpecifiedCommitWithChangelog` is intermittently failing in CI with a streaming-read row-count mismatch, e.g. expecting the 11 merged rows but receiving 9, with the entire `par2` partition (`id3`, `id4`) missing. It surfaced on an unrelated PR's `test-flink-1 (flink2.1, 1.11.4, 1.15.2)` job: https://github.com/apache/hudi/actions/runs/27664812470/job/81816514280
These streaming reads are collected with `CollectTableSink` and terminated by a forced `SuccessException` once the expected row count is reached, with up to a 30s await. `fetchResultWithExpectedNum` tolerates a known benign teardown race (an `IOException("Stream is closed")`, or a `ParquetColumnarRowSplitReader#readNextRowGroup` NPE) that can fire when the source's `SplitFetcher` closes its stream during the job's cascading shutdown. That tolerance (added in #19019) assumed the race only fires after the sink has already collected its expected rows. Under CI load it can instead fire before the read completes, so the job ends with fewer rows than expected; the tolerated exception is then swallowed and the test asserts on the incomplete result, producing the confusing row-count mismatch.
### Summary and Changelog
`fetchResultWithExpectedNum` no longer assumes that a tolerated terminal failure implies a complete read. The submit-and-collect path is wrapped in a new `submitAndFetchWithRetry`, which re-reads (up to `MAX_STREAM_READ_ATTEMPTS = 3`) when the collected row count is below the expectation. Re-reading the already committed table is idempotent (the collect sink clears its global result on each submit), so a transient teardown race no longer leaks into the assertion. The swallowed non-`SuccessException` cause is now logged so an incomplete read is diagnosable. Behavior is unchanged on the happy path: a complete first read returns immediately with no retry.
The change is confined to the test harness in `ITTestHoodieDataSource`; no production code is touched.
### Impact
De-flakes `ITTestHoodieDataSource` streaming-read tests (and any test going through `execSelectSqlWithExpectedNum`). No public API, on-disk format, or production behavior change.
### Risk Level
low. Test-only change. The retry is a no-op on the happy path and only re-reads an idempotent, already-committed table when a read comes back short. Validated locally on JDK 11 / flink1.20: 60 consecutive runs of the failing parameterization pass, and a fault-injection run that truncates the first attempt (reproducing the exact 9-of-11 CI symptom) confirms the retry recovers the full result; the full test method (6 parameterizations) passes with 0 Checkstyle violations.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(flink): retry short CollectSink reads to de-flake stream-read ITs [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19030:
URL: https://github.com/apache/hudi/pull/19030#issuecomment-4728054553
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7040c81a31fa989a70863e0db0b534c738c39b61",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14777",
"triggerID" : "7040c81a31fa989a70863e0db0b534c738c39b61",
"triggerType" : "PUSH"
}, {
"hash" : "b7f334f1d8bb8e447fc67c807d37fc7ef54cfc47",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14782",
"triggerID" : "b7f334f1d8bb8e447fc67c807d37fc7ef54cfc47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b7f334f1d8bb8e447fc67c807d37fc7ef54cfc47 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14782)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(flink): retry short CollectSink reads to de-flake stream-read ITs [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19030:
URL: https://github.com/apache/hudi/pull/19030#issuecomment-4727348739
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7040c81a31fa989a70863e0db0b534c738c39b61",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14777",
"triggerID" : "7040c81a31fa989a70863e0db0b534c738c39b61",
"triggerType" : "PUSH"
}, {
"hash" : "b7f334f1d8bb8e447fc67c807d37fc7ef54cfc47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14782",
"triggerID" : "b7f334f1d8bb8e447fc67c807d37fc7ef54cfc47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7040c81a31fa989a70863e0db0b534c738c39b61 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14777)
* b7f334f1d8bb8e447fc67c807d37fc7ef54cfc47 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14782)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(flink): retry short CollectSink reads to de-flake stream-read ITs [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19030:
URL: https://github.com/apache/hudi/pull/19030#issuecomment-4726991293
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7040c81a31fa989a70863e0db0b534c738c39b61",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14777",
"triggerID" : "7040c81a31fa989a70863e0db0b534c738c39b61",
"triggerType" : "PUSH"
}, {
"hash" : "b7f334f1d8bb8e447fc67c807d37fc7ef54cfc47",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14782",
"triggerID" : "b7f334f1d8bb8e447fc67c807d37fc7ef54cfc47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7040c81a31fa989a70863e0db0b534c738c39b61 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14777)
* b7f334f1d8bb8e447fc67c807d37fc7ef54cfc47 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14782)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(flink): retry short CollectSink reads to de-flake stream-read ITs [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #19030:
URL: https://github.com/apache/hudi/pull/19030#discussion_r3426031054
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java:
##########
@@ -4012,4 +4048,28 @@ private static boolean containsReadNextRowGroupFrame(Throwable t) {
}
return false;
}
+
+ /**
+ * Whether {@code e} (or any of its causes) is the normal {@link CollectSinkTableFactory.SuccessException}
+ * terminator (the happy path), as opposed to one of the tolerated teardown-race symptoms.
+ */
+ private static boolean isSuccessException(Throwable e) {
+ for (Throwable cur = e; cur != null; cur = cur.getCause()) {
+ if (cur instanceof CollectSinkTableFactory.SuccessException) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Short description of {@code e}'s root cause, for logging which tolerated terminal failure fired.
+ */
+ private static String describeTerminalCause(Throwable e) {
+ Throwable root = e;
+ while (root.getCause() != null) {
+ root = root.getCause();
+ }
+ return root.getClass().getName() + ": " + root.getMessage();
Review Comment:
Done b7f334f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(flink): retry short CollectSink reads to de-flake stream-read ITs [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19030:
URL: https://github.com/apache/hudi/pull/19030#issuecomment-4726748847
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7040c81a31fa989a70863e0db0b534c738c39b61",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14777",
"triggerID" : "7040c81a31fa989a70863e0db0b534c738c39b61",
"triggerType" : "PUSH"
}, {
"hash" : "b7f334f1d8bb8e447fc67c807d37fc7ef54cfc47",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b7f334f1d8bb8e447fc67c807d37fc7ef54cfc47",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7040c81a31fa989a70863e0db0b534c738c39b61 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14777)
* b7f334f1d8bb8e447fc67c807d37fc7ef54cfc47 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(flink): retry short CollectSink reads to de-flake stream-read ITs [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19030:
URL: https://github.com/apache/hudi/pull/19030#discussion_r3425900414
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java:
##########
@@ -126,6 +128,15 @@
*/
@ExtendWith(FlinkMiniCluster.class)
public class ITTestHoodieDataSource {
+ private static final Logger LOG = LoggerFactory.getLogger(ITTestHoodieDataSource.class);
+
+ // A streaming read collected via CollectTableSink is terminated by a forced SuccessException once it
+ // reaches its expected row count. A benign teardown race (see isAcceptableTerminalFailure) can instead
+ // close the source stream mid-read and terminate the job before all rows are emitted, leaving an
+ // incomplete result. Re-reading from the same (already committed) table is idempotent, so retry a few
+ // times before giving up. See fetchResultWithExpectedNum.
Review Comment:
🤖 nit: the comment ends with `See fetchResultWithExpectedNum`, but the retry loop now lives in `submitAndFetchWithRetry` — could you update the reference so a future reader lands in the right place?
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java:
##########
@@ -4012,4 +4048,28 @@ private static boolean containsReadNextRowGroupFrame(Throwable t) {
}
return false;
}
+
+ /**
+ * Whether {@code e} (or any of its causes) is the normal {@link CollectSinkTableFactory.SuccessException}
+ * terminator (the happy path), as opposed to one of the tolerated teardown-race symptoms.
+ */
+ private static boolean isSuccessException(Throwable e) {
+ for (Throwable cur = e; cur != null; cur = cur.getCause()) {
+ if (cur instanceof CollectSinkTableFactory.SuccessException) {
+ return true;
+ }
+ }
+ return false;
+ }
+
+ /**
+ * Short description of {@code e}'s root cause, for logging which tolerated terminal failure fired.
+ */
+ private static String describeTerminalCause(Throwable e) {
+ Throwable root = e;
+ while (root.getCause() != null) {
+ root = root.getCause();
+ }
+ return root.getClass().getName() + ": " + root.getMessage();
Review Comment:
🤖 nit: `getClass().getName()` produces the fully-qualified class name, which is a bit noisy in a log line — `getSimpleName()` would keep the output tighter and just as diagnosable here.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(flink): retry short CollectSink reads to de-flake stream-read ITs [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #19030:
URL: https://github.com/apache/hudi/pull/19030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(flink): retry short CollectSink reads to de-flake stream-read ITs [hudi]
Posted by "wombatu-kun (via GitHub)" <gi...@apache.org>.
wombatu-kun commented on code in PR #19030:
URL: https://github.com/apache/hudi/pull/19030#discussion_r3426029124
##########
hudi-flink-datasource/hudi-flink/src/test/java/org/apache/hudi/table/ITTestHoodieDataSource.java:
##########
@@ -126,6 +128,15 @@
*/
@ExtendWith(FlinkMiniCluster.class)
public class ITTestHoodieDataSource {
+ private static final Logger LOG = LoggerFactory.getLogger(ITTestHoodieDataSource.class);
+
+ // A streaming read collected via CollectTableSink is terminated by a forced SuccessException once it
+ // reaches its expected row count. A benign teardown race (see isAcceptableTerminalFailure) can instead
+ // close the source stream mid-read and terminate the job before all rows are emitted, leaving an
+ // incomplete result. Re-reading from the same (already committed) table is idempotent, so retry a few
+ // times before giving up. See fetchResultWithExpectedNum.
Review Comment:
Done b7f334f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(flink): retry short CollectSink reads to de-flake stream-read ITs [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19030:
URL: https://github.com/apache/hudi/pull/19030#issuecomment-4726554219
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7040c81a31fa989a70863e0db0b534c738c39b61",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14777",
"triggerID" : "7040c81a31fa989a70863e0db0b534c738c39b61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7040c81a31fa989a70863e0db0b534c738c39b61 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14777)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] test(flink): retry short CollectSink reads to de-flake stream-read ITs [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19030:
URL: https://github.com/apache/hudi/pull/19030#issuecomment-4726525814
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7040c81a31fa989a70863e0db0b534c738c39b61",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7040c81a31fa989a70863e0db0b534c738c39b61",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7040c81a31fa989a70863e0db0b534c738c39b61 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org