You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "voonhous (via GitHub)" <gi...@apache.org> on 2026/09/30 12:59:54 UTC
[PR] fix(spark): check data schema in Spark 4.0 guard [hudi]
voonhous opened a new pull request, #20158:
URL: https://github.com/apache/hudi/pull/20158
### Describe the issue this Pull Request addresses
Spark 4.0 with `spark.sql.variant.pushVariantIntoScan=true` is unsupported (#20032), and #20033 added a guard that fails such a read with an error naming the conf. CI builds no Spark 4.0 profile, so neither the guard nor its test had run. On Spark 4.0.2 the test fails: a query that reads no variant column gets past the guard and dies with `HoodieSchemaException: Illegal initial character: 0`.
The guard checks the required schema of the scan. `PushVariantIntoScan` rewrites every variant column of the relation, into a placeholder `struct<0:boolean>` when no expression asks for it, so the rewrite sits in the data schema of every scan of the table. Converting that data schema to a HoodieSchema then trips on the field name `0`.
### Summary and Changelog
- `HoodieFileGroupReaderBasedFileFormat.buildReaderWithPartitionValues` validates the data schema instead of the required schema.
- `SparkAdapter.validateVariantProjectionReadable`: parameter renamed to `schema`, doc updated.
- `TestVariantDataType` "Test Spark 4.0 rejects reads rewritten by pushVariantIntoScan": the query that reads no variant now expects the guard. It expected rows, which it never returned.
<details>
<summary>Spark 4.0.2, conf on, COW and MOR, AVRO and SPARK record types, top-level and nested variant</summary>
| Query | Before | After |
|:--|:--|:--|
| `variant_get(v, ...)`, `cast(v as string)`, `where v is null` | guard message | guard message |
| `select id, ts` | `Illegal initial character: 0` | guard message |
| `select count(*)` | not run | guard message |
Guard message: `Column 'v' was rewritten by spark.sql.variant.pushVariantIntoScan into a variant projection struct, which Hudi does not support on Spark 4.0.2. Set spark.sql.variant.pushVariantIntoScan=false (the Spark 4.0 default) or read the table with Spark 4.1+.`
The test fails before the change and passes with it. `TestBaseSpark4AdapterVariantMethods` passes on 4.0.2 either way.
</details>
### Impact
Spark 4.0 only, and only with the internal conf turned on: reads that already failed now fail with the message that names the conf. The hook stays a no-op on Spark 3.x, 4.1 and 4.2.
### Risk Level
low
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): check data schema in Spark 4.0 guard [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20158:
URL: https://github.com/apache/hudi/pull/20158#issuecomment-5912225382
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c476d23f185945b29e3c7fe457226b6dc8345701",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c476d23f185945b29e3c7fe457226b6dc8345701",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c476d23f185945b29e3c7fe457226b6dc8345701 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): check data schema in Spark 4.0 guard [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20158:
URL: https://github.com/apache/hudi/pull/20158#issuecomment-5914363384
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c476d23f185945b29e3c7fe457226b6dc8345701",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17388",
"triggerID" : "c476d23f185945b29e3c7fe457226b6dc8345701",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c476d23f185945b29e3c7fe457226b6dc8345701 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17388)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): check data schema in Spark 4.0 guard [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #20158:
URL: https://github.com/apache/hudi/pull/20158#issuecomment-5912348448
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "c476d23f185945b29e3c7fe457226b6dc8345701",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17388",
"triggerID" : "c476d23f185945b29e3c7fe457226b6dc8345701",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c476d23f185945b29e3c7fe457226b6dc8345701 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=17388)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): check data schema in Spark 4.0 guard [hudi]
Posted by "voonhous (via GitHub)" <gi...@apache.org>.
voonhous merged PR #20158:
URL: https://github.com/apache/hudi/pull/20158
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix(spark): check data schema in Spark 4.0 guard [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20158:
URL: https://github.com/apache/hudi/pull/20158#issuecomment-5912091093
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/20158?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 80.41%. Comparing base ([`4d3af13`](https://app.codecov.io/gh/apache/hudi/commit/4d3af133b42e488abc37ebf9f81e4d75debbcd8b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`c476d23`](https://app.codecov.io/gh/apache/hudi/commit/c476d23f185945b29e3c7fe457226b6dc8345701?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 1 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #20158 +/- ##
============================================
- Coverage 80.41% 80.41% -0.01%
+ Complexity 34900 34893 -7
============================================
Files 2546 2546
Lines 142931 142929 -2
Branches 17383 17383
============================================
- Hits 114940 114938 -2
Misses 20078 20078
Partials 7913 7913
```
| [Components](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hudi-common](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.96% <ø> (ø)` | |
| [hudi-client](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.48% <ø> (-0.01%)` | :arrow_down: |
| [hudi-flink](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `85.74% <ø> (ø)` | |
| [hudi-spark-datasource](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.85% <ø> (-0.01%)` | :arrow_down: |
| [hudi-utilities](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `78.19% <ø> (ø)` | |
| [hudi-cli](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.05% <ø> (ø)` | |
| [hudi-hadoop](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `70.98% <ø> (ø)` | |
| [hudi-sync](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `76.02% <ø> (ø)` | |
| [hudi-io](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.50% <ø> (ø)` | |
| [hudi-timeline-service](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `83.06% <ø> (ø)` | |
| [hudi-cloud](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `81.00% <ø> (ø)` | |
| [hudi-kafka-connect](https://app.codecov.io/gh/apache/hudi/pull/20158/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.20% <ø> (ø)` | |
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/20158/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/20158/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.99% <ø> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/20158?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...scala/org/apache/spark/sql/hudi/SparkAdapter.scala](https://app.codecov.io/gh/apache/hudi/pull/20158?src=pr&el=tree&filepath=hudi-client%2Fhudi-spark-client%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fhudi%2FSparkAdapter.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1jbGllbnQvaHVkaS1zcGFyay1jbGllbnQvc3JjL21haW4vc2NhbGEvb3JnL2FwYWNoZS9zcGFyay9zcWwvaHVkaS9TcGFya0FkYXB0ZXIuc2NhbGE=) | `63.63% <ø> (-1.59%)` | :arrow_down: |
| [...parquet/HoodieFileGroupReaderBasedFileFormat.scala](https://app.codecov.io/gh/apache/hudi/pull/20158?src=pr&el=tree&filepath=hudi-spark-datasource%2Fhudi-spark-common%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fspark%2Fsql%2Fexecution%2Fdatasources%2Fparquet%2FHoodieFileGroupReaderBasedFileFormat.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zcGFyay1kYXRhc291cmNlL2h1ZGktc3BhcmstY29tbW9uL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUvc3Bhcmsvc3FsL2V4ZWN1dGlvbi9kYXRhc291cmNlcy9wYXJxdWV0L0hvb2RpZUZpbGVHcm91cFJlYWRlckJhc2VkRmlsZUZvcm1hdC5zY2FsYQ==) | `84.30% <ø> (-0.06%)` | :arrow_down: |
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org