You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2026/10/24 07:00:43 UTC
[PR] fix(spark): keep streaming queries running when a write client swap fails an async table service run [hudi]
nsivabalan opened a new pull request, #20431:
URL: https://github.com/apache/hudi/pull/20431
> **WIP** — stacked draft for early alignment.
### Describe the issue this Pull Request addresses
Stacked on #20415 (sink re-creates its write client on a schema change) and #20429 (async table services retry a run that failed because their write client was replaced). **Review only the last commit**; the earlier commits are those PRs.
With #20415, `HoodieStreamingSink` re-creates its write client when the writer schema changes and closes the previous one, as `HoodieStreamer` does. An async compaction or clustering still running on the previous client may then fail. Before #20429 that stopped the async service, and the sink then failed its next micro-batch with `Async Compactor shutdown unexpectedly` — without the cause — ending the query.
Related: #20400.
### Summary and Changelog
- No sink-side change is needed for the retry itself: the sink already switches its async services to the new client before closing the previous one (`retireWriteClient`), so with #20429 a run that fails on the closed client is retried by its service with the new client and the query keeps running.
- `HoodieStreamingSink`: when an async table service stops with a failure it could not recover from, the micro-batch that fails because of it (`Async Compactor shutdown unexpectedly` / `Async clustering service shutdown unexpectedly`) now carries that failure as its cause.
- Tests (`TestStructuredStreaming`), using a Spark record merger that controls merges inside async compaction tasks:
- `testStructuredStreamingRetriesAsyncCompactionFailedByWriteClientSwap`: an async compaction is held mid-run, a batch writer evolves the table schema so that the next micro-batch re-creates the sink's write client, then the held compaction fails; it is retried with the new client and completes, and the query keeps running.
- `testStructuredStreamingFailsWithCauseWhenAsyncCompactionFails`: an async compaction that keeps failing fails the query, with the compaction failure in the cause chain.
### Impact
Streaming queries survive an async compaction/clustering that fails because a schema change replaced the sink's write client; other async table service failures still fail the query, now with their cause.
### Risk Level
low. One sink-side change (failure cause propagation); the retry behaviour comes from #20429. Verified with `TestStructuredStreaming` (27/27).
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): keep streaming queries running when a write client swap fails an async table service run [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #20431:
URL: https://github.com/apache/hudi/pull/20431#issuecomment-6454939293
Superseded by #20445, which is stacked on #20444 (restores the non-blocking 1.2.0 write client switch) and applies the retryable/non-retryable handling to the streaming sink only. HoodieStreamer stays at its 1.2.0 behaviour; see #20429. Closing this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): keep streaming queries running when a write client swap fails an async table service run [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20431:
URL: https://github.com/apache/hudi/pull/20431#issuecomment-6381335674
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20431?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `67.63006%` with `56 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.08%. Comparing base ([`60caf5f`](https://app.codecov.io/gh/apache/hudi/commit/60caf5f423989ff6d07136407bc8b0c20ee04c65?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`959b2c4`](https://app.codecov.io/gh/apache/hudi/commit/959b2c41f3632013f72960f8b0e24276e29fa48a?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 88 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20431?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...in/scala/org/apache/hudi/HoodieStreamingSink.scala](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FHoodieStreamingSink.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9Ib29kaWVTdHJlYW1pbmdTaW5rLnNjYWxh) | 73.07% | [10 Missing and 11 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...n/scala/org/apache/hudi/HoodieSparkSqlWriter.scala](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FHoodieSparkSqlWriter.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9Ib29kaWVTcGFya1NxbFdyaXRlci5zY2FsYQ==) | 83.82% | [4 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../org/apache/hudi/async/AsyncClusteringService.java](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fasync%2FAsyncClusteringService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ2x1c3RlcmluZ1NlcnZpY2UuamF2YQ==) | 0.00% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/async/AsyncCompactService.java](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fasync%2FAsyncCompactService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | 0.00% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...java/org/apache/hudi/async/HoodieAsyncService.java](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fasync%2FHoodieAsyncService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0hvb2RpZUFzeW5jU2VydmljZS5qYXZh) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20431 +/- ##
============================================
- Coverage 80.49% 80.08% -0.41%
- Complexity 34953 35507 +554
============================================
Files 2549 2575 +26
Lines 143060 146882 +3822
Branches 17392 18584 +1192
============================================
+ Hits 115151 117632 +2481
- Misses 20004 21121 +1117
- Partials 7905 8129 +224
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.88% <ø> (-0.12%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.21% <0.00%> (-0.33%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `86.16% <ø> (+0.19%)` | :arrow_up: |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.85% <78.52%> (-3.00%)` | :arrow_down: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.24% <ø> (+1.06%)` | :arrow_up: |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.80% <ø> (+0.02%)` | :arrow_up: |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `71.10% <ø> (+0.05%)` | :arrow_up: |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `77.31% <ø> (+1.08%)` | :arrow_up: |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.57% <ø> (+0.07%)` | :arrow_up: |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `84.06% <ø> (+0.75%)` | :arrow_up: |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.01% <ø> (+0.01%)` | :arrow_up: |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20431/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `57.90% <ø> (+4.70%)` | :arrow_up: |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20431/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20431/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.94% <0.00%> (-0.04%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20431?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...rg/apache/hudi/common/table/marker/MarkerType.java](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fcommon%2Ftable%2Fmarker%2FMarkerType.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL21hcmtlci9NYXJrZXJUeXBlLmphdmE=) | `100.00% <ø> (ø)` | |
| [...src/main/java/org/apache/hudi/DataSourceUtils.java](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2FDataSourceUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL0RhdGFTb3VyY2VVdGlscy5qYXZh) | `71.77% <100.00%> (+0.46%)` | :arrow_up: |
| [...apache/hudi/utilities/streamer/HoodieStreamer.java](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fstreamer%2FHoodieStreamer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3N0cmVhbWVyL0hvb2RpZVN0cmVhbWVyLmphdmE=) | `75.80% <ø> (+1.10%)` | :arrow_up: |
| [...org/apache/hudi/utilities/streamer/StreamSync.java](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-utilities%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Futilities%2Fstreamer%2FStreamSync.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3N0cmVhbWVyL1N0cmVhbVN5bmMuamF2YQ==) | `77.62% <ø> (-0.11%)` | :arrow_down: |
| [...java/org/apache/hudi/async/HoodieAsyncService.java](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fasync%2FHoodieAsyncService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0hvb2RpZUFzeW5jU2VydmljZS5qYXZh) | `77.61% <0.00%> (-4.93%)` | :arrow_down: |
| [.../org/apache/hudi/async/AsyncClusteringService.java](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fasync%2FAsyncClusteringService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ2x1c3RlcmluZ1NlcnZpY2UuamF2YQ==) | `54.34% <0.00%> (-24.04%)` | :arrow_down: |
| [...ava/org/apache/hudi/async/AsyncCompactService.java](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-client%2Fhudi-client-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fasync%2FAsyncCompactService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1jbGllbnQtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2FzeW5jL0FzeW5jQ29tcGFjdFNlcnZpY2UuamF2YQ==) | `53.19% <0.00%> (-23.13%)` | :arrow_down: |
| [...n/scala/org/apache/hudi/HoodieSparkSqlWriter.scala](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FHoodieSparkSqlWriter.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9Ib29kaWVTcGFya1NxbFdyaXRlci5zY2FsYQ==) | `77.05% <83.82%> (-1.94%)` | :arrow_down: |
| [...in/scala/org/apache/hudi/HoodieStreamingSink.scala](https://app.codecov.io/gh/apache/hudi/pull/20431?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fhudi%2FHoodieStreamingSink.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvaHVkaS9Ib29kaWVTdHJlYW1pbmdTaW5rLnNjYWxh) | `71.30% <73.07%> (+1.60%)` | :arrow_up: |
... and [271 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20431/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): keep streaming queries running when a write client swap fails an async table service run [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed pull request #20431: fix(spark): keep streaming queries running when a write client swap fails an async table service run
URL: https://github.com/apache/hudi/pull/20431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org