You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "rahil-c (via GitHub)" <gi...@apache.org> on 2026/10/02 18:24:49 UTC
[PR] fix(spark): keep read_blob's broadcast storage configuration intact under Kryo [hudi]
rahil-c opened a new pull request, #20192:
URL: https://github.com/apache/hudi/pull/20192
### Describe the issue this Pull Request addresses
closes #20191
`read_blob()` on OUT_OF_LINE blobs fails under `KryoSerializer` when Hudi's Kryo registrator is not configured.
The broadcast `HadoopStorageConfiguration` keeps its Hadoop `Configuration` in a `transient` field that only Java serialization restores, so executors get a null configuration.
### Summary and Changelog
`read_blob()` now works under any `spark.serializer`.
- `BatchedBlobReader`: adds `broadcastStorageConf`, which broadcasts the Hadoop configuration wrapped in Spark's `SerializableConfiguration`, and `storageConfFrom`, which rebuilds the `HadoopStorageConfiguration` on the executor. This matches how the rest of Hudi's read path ships configuration (for example `HoodieFileGroupReaderBasedFileFormat`).
- `BatchedBlobReader.readBatched`, `processRDD` and `BatchedBlobReadExec.doExecute` use the new broadcast.
- `TestBatchedBlobReaderKryo`: reads OUT_OF_LINE blobs after a Kryo round trip of the broadcast value with no Hudi registrator. Local mode does not serialize broadcasts, so the test does the round trip itself.
The new test fails on master with a `NullPointerException` from the null configuration (`HadoopStorageConfiguration.writeObject`) and passes with this change.
`TestBatchedBlobReaderKryo`, `TestReadBlobSQL` and `TestBatchedBlobReader` pass (37 tests).
### Impact
None on public APIs. `BatchedBlobReader.processRDD` now takes `Broadcast[SerializableConfiguration]`; it is an internal helper.
### Risk Level
low
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): keep read_blob's broadcast storage configuration intact under Kryo [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20192:
URL: https://github.com/apache/hudi/pull/20192#issuecomment-5959994935
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b9e1320b16c6ab6f58fc40948a6261b0e2937496",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17424",
"triggerID" : "b9e1320b16c6ab6f58fc40948a6261b0e2937496",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9e1320b16c6ab6f58fc40948a6261b0e2937496 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): keep read_blob's broadcast storage configuration intact under Kryo [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20192:
URL: https://github.com/apache/hudi/pull/20192#issuecomment-5959183571
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b9e1320b16c6ab6f58fc40948a6261b0e2937496",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17424",
"triggerID" : "b9e1320b16c6ab6f58fc40948a6261b0e2937496",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9e1320b16c6ab6f58fc40948a6261b0e2937496 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17424)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): keep read_blob's broadcast storage configuration intact under Kryo [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20192:
URL: https://github.com/apache/hudi/pull/20192#issuecomment-5959049042
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b9e1320b16c6ab6f58fc40948a6261b0e2937496",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b9e1320b16c6ab6f58fc40948a6261b0e2937496",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b9e1320b16c6ab6f58fc40948a6261b0e2937496 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): keep read_blob's broadcast storage configuration intact under Kryo [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20192:
URL: https://github.com/apache/hudi/pull/20192#issuecomment-5958935171
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20192?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 80.48%. Comparing base ([`afe9e6b`](https://app.codecov.io/gh/apache/hudi/commit/afe9e6bcedab79ce27e156e22d4da2985a4447f5?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b9e1320`](https://app.codecov.io/gh/apache/hudi/commit/b9e1320b16c6ab6f58fc40948a6261b0e2937496?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20192 +/- ##
============================================
- Coverage 80.48% 80.48% -0.01%
Complexity 34956 34956
============================================
Files 2549 2549
Lines 143063 143059 -4
Branches 17392 17392
============================================
- Hits 115150 115144 -6
- Misses 20009 20010 +1
- Partials 7904 7905 +1
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.99% <ø> (ø)` | |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.53% <ø> (-0.01%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.97% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.85% <ø> (-0.01%)` | :arrow_down: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.16% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.78% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `71.03% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.25% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.50% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.06% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20192/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20192/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.99% <ø> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20192?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ache/spark/sql/hudi/blob/BatchedBlobReadExec.scala](https://app.codecov.io/gh/apache/hudi/pull/20192?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fhudi%2Fblob%2FBatchedBlobReadExec.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2h1ZGkvYmxvYi9CYXRjaGVkQmxvYlJlYWRFeGVjLnNjYWxh) | `93.75% <ø> (-0.37%)` | :arrow_down: |
| [...apache/spark/sql/hudi/blob/BatchedBlobReader.scala](https://app.codecov.io/gh/apache/hudi/pull/20192?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fhudi%2Fblob%2FBatchedBlobReader.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2h1ZGkvYmxvYi9CYXRjaGVkQmxvYlJlYWRlci5zY2FsYQ==) | `82.13% <ø> (-0.19%)` | :arrow_down: |
... and [1 file with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/20192/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): keep read_blob's broadcast storage configuration intact under Kryo [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous closed pull request #20192: fix(spark): keep read_blob's broadcast storage configuration intact under Kryo
URL: https://github.com/apache/hudi/pull/20192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): keep read_blob's broadcast storage configuration intact under Kryo [hudi]
Posted by "voonhous (via GitHub) via commits" <co...@hudi.apache.org>.
voonhous commented on PR #20192:
URL: https://github.com/apache/hudi/pull/20192#issuecomment-6701594829
Temporarily closing this PR while we debug a GitHub settings error on master (#20471); it will be reopened shortly, no action needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org