You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "wombatu-kun (via GitHub)" <gi...@apache.org> on 2026/10/14 04:41:16 UTC
[PR] fix(flink): make the bucket pruning function serializable for source V2 [hudi]
wombatu-kun opened a new pull request, #20318:
URL: https://github.com/apache/hudi/pull/20318
### Describe the issue this Pull Request addresses
Closes #20317.
With `read.source-v2.enabled=true`, a Flink query that filters a bucket index table on all its bucket index key fields fails while Flink builds the job graph, with `java.io.NotSerializableException: org.apache.hudi.source.prune.PrimaryKeyPruners$$Lambda$...`. `PrimaryKeyPruners.getBucketIdFunc` returns a plain `java.util.function.Function` lambda, and the FLIP-27 `HoodieSource` carries it into the job graph twice: through its reader function supplier, which captures the `HoodieTableSource` and with it the `dataBucketFunc` field, and through the `PartitionBucketIdFunc` in its scan context.
### Summary and Changelog
The bucket id function built for bucket pruning is serializable, so Source V2 queries that filter by the bucket keys can be submitted.
- `PrimaryKeyPruners#getBucketIdFunc` returns Flink's `SerializableFunction`, which extends `java.util.function.Function`, so the returned lambda is serializable while its callers keep using it as a `Function`.
- Tests: `ITTestHoodieDataSource#testBucketPruning` also runs with Source V2, on COPY_ON_WRITE and MERGE_ON_READ. `TestPrimaryKeyPruners#testBucketIdFunctionIsSerializable` checks that the function survives Java serialization and still computes the same bucket ids.
### Impact
Source V2 batch and streaming queries with an equality filter on all the bucket index key fields of a bucket index table run instead of failing at job submission. 1.2.1 fails the same way, and its `PrimaryKeyPruners` is identical to master's. No storage format, configuration or user-facing API change.
### Risk Level
Low. Only the target type of the lambda changes. It captures the same list of key values, so the bucket ids it computes do not change. `org.apache.flink.util.function.SerializableFunction` is `@Public` and available for every supported Flink version: in `flink-core` up to 1.19, and in `flink-core-api`, a dependency of `flink-core`, from 1.20.
Verified:
- The Source V2 cases of `testBucketPruning` fail on master with `NotSerializableException` and pass with the fix, under the `flink1.20` profile and CI's `flink2.2` profile; the legacy cases pass either way. A streaming Source V2 read with the same filter fails and passes the same way.
- The new unit test fails without the fix.
- The other objects the source carries for bucket pruning are serializable, including the calculator of the partition-level bucket index, whose Caffeine cache serializes its settings without its entries.
- It merges cleanly with #20292. It conflicts with my #20314 on the `getBucketIdFunc` signature; whichever lands second resolves it by having both overloads return `SerializableFunction`, which the compiler enforces.
### Documentation Update
None.
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(flink): make the bucket pruning function serializable for source V2 [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #20318:
URL: https://github.com/apache/hudi/pull/20318
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(flink): make the bucket pruning function serializable for source V2 [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20318:
URL: https://github.com/apache/hudi/pull/20318#issuecomment-6161941867
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20318?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 74.08%. Comparing base ([`b86a34b`](https://app.codecov.io/gh/apache/hudi/commit/b86a34b59e015b68796d88de8b18eea38c1739ae?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`398d1d0`](https://app.codecov.io/gh/apache/hudi/commit/398d1d0157176829c64063fdd2d50b839606b7b9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 5 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20318 +/- ##
============================================
- Coverage 74.12% 74.08% -0.05%
- Complexity 32390 32438 +48
============================================
Files 2553 2553
Lines 143524 143776 +252
Branches 17474 17511 +37
============================================
+ Hits 106386 106513 +127
- Misses 29687 29801 +114
- Partials 7451 7462 +11
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `82.23% <ø> (-0.24%)` | :arrow_down: |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `79.46% <ø> (+<0.01%)` | :arrow_up: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `86.05% <ø> (-0.01%)` | :arrow_down: |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `40.91% <ø> (ø)` | |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.20% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.80% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `69.72% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.62% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.01% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.41% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20318/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `54.48% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20318/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20318/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.94% <ø> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20318?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...rg/apache/hudi/source/prune/PrimaryKeyPruners.java](https://app.codecov.io/gh/apache/hudi/pull/20318?src=pr&el=tree&filepath=hudi-flink-datasource%2Fhudi-flink%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fsource%2Fprune%2FPrimaryKeyPruners.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1mbGluay1kYXRhc291cmNlL2h1ZGktZmxpbmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvc291cmNlL3BydW5lL1ByaW1hcnlLZXlQcnVuZXJzLmphdmE=) | `94.11% <ø> (ø)` | |
... and [5 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20318/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org