You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "baibaichen (via GitHub)" <gi...@apache.org> on 2026/04/11 17:18:31 UTC
[I] [VL] Spark 4.1: Support Parquet struct field compatibility improvements [gluten]
baibaichen opened a new issue, #11914:
URL: https://github.com/apache/gluten/issues/11914
## Backend
VL (Velox)
## Bug description
Two Parquet struct field handling issues in GlutenParquetIOSuite on Spark 4.1:
**Issue 1: SPARK-53535** — Spark 4.1 added `returnNullStructIfAllFieldsMissing` parameter for missing struct field handling. Gluten has not adapted to this change.
**Issue 2: Vectorized reader missing struct fields** — Pre-existing test (SPARK-34863) for vectorized reader complex type handling. SPARK-53535 modified this test, causing it to fail on Spark 4.1 with Gluten.
These are two independent root causes but both only affect Spark 4.1.
Parent issue: #11550
### Impact
| Suite | Exclude | spark40 | spark41 |
|-------|---------|:-------:|:-------:|
| GlutenParquetIOSuite | SPARK-53535 | 🟢 | 🔴 |
| GlutenParquetIOSuite | vectorized reader: missing all struct fields | 🟢 | 🔴 |
Note: GlutenParquetIOSuite also has a SPARK-54220 (NullType) exclude tracked separately under the Tracking issue #11910.
### References
- Apache Spark JIRA: [SPARK-53535](https://issues.apache.org/jira/browse/SPARK-53535)
- Apache Spark JIRA: [SPARK-34863](https://issues.apache.org/jira/browse/SPARK-34863)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Spark 4.1: Support Parquet struct field compatibility improvements [gluten]
Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo closed issue #11914: [VL] Spark 4.1: Support Parquet struct field compatibility improvements
URL: https://github.com/apache/gluten/issues/11914
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Spark 4.1: Support Parquet struct field compatibility improvements [gluten]
Posted by "rui-mo (via GitHub)" <gi...@apache.org>.
rui-mo commented on issue #11914:
URL: https://github.com/apache/gluten/issues/11914#issuecomment-4655713272
The behaviors mentioned in this patch are supposed to be enabled through the changes in [Velox PR #17555](https://github.com/facebookincubator/velox/pull/17555), which Gluten will soon depend on. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org