You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@gluten.apache.org by "baibaichen (via GitHub)" <gi...@apache.org> on 2026/03/03 04:27:22 UTC
[I] [VL] Support type widening in Parquet reader (SPARK-40876) [incubator-gluten]
baibaichen opened a new issue, #11683:
URL: https://github.com/apache/incubator-gluten/issues/11683
**Labels**: enhancement, VELOX
---
## Description
Enable the `GlutenParquetTypeWideningSuite` test suite for Spark 4.0 and 4.1, which validates Parquet type widening support ([SPARK-40876](https://issues.apache.org/jira/browse/SPARK-40876)).
### Background
`GlutenParquetTypeWideningSuite` has **84 tests** covering two types of Parquet type conversions:
1. **Physical→Logical type restoration**: Reading `int32 + INT(8)` as `TINYINT` (safe, writer guarantees value range)
2. **Schema evolution widening**: Reading old `IntegerType` data as `LongType`, `DoubleType`, or `DecimalType` (Spark 4.0 feature)
Currently the suite is disabled with **74 out of 84 tests failing**. The failures fall into four categories:
| Category | Count | Issue | Fix |
|----------|-------|-------|-----|
| A | 13 | Velox doesn't support INT→DOUBLE/REAL/DECIMAL widening | Velox C++ `convertType()` extension |
| B | 29 | Exception type mismatch + no Decimal precision check | Exception translation + C++ precision check |
| C | 31 | Parquet V2 encoding assertions + Decimal conversion limits | Disable native writer + test overrides + Velox C++ |
| D | 1 | parquet-mr only decimal narrowing overflow→null | Exclude (cannot reproduce with native reader) |
### Plan
This will be addressed in **3 PRs**:
1. **PR 1 — Exception translation**: Add `translateException()` to convert Velox type errors to `SchemaColumnConvertNotSupportedException`. Enable the suite with appropriate excludes/overrides for tests that pass without C++ changes.
2. **PR 2 — SPARK-18108 + Revert OAP**: Fix partition column type conflicts. Import upstream Velox [PR #15173](https://github.com/facebookincubator/velox/pull/15173).
3. **PR 3 — Type widening implementation**: Velox C++ changes for INT→DOUBLE/REAL/DECIMAL and Decimal→Decimal widening. Requires upstream Velox PR first, then enable remaining tests.
### Test Results (Target)
| | Spark 4.0 | Spark 4.1 |
|--|-----------|-----------|
| ✅ Passed | 46 | 46 |
| 🟢 Override (passed) | 35 | 35 |
| ❌ Excluded | 3 | 3 |
| **Total** | **84** | **84** |
Sub-issue of #11550.
This issue was written with the assistance of AI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Support type widening in Parquet reader (SPARK-40876) [gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen closed issue #11683: [VL] Support type widening in Parquet reader (SPARK-40876)
URL: https://github.com/apache/gluten/issues/11683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Support type widening in Parquet reader (SPARK-40876) [incubator-gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen closed issue #11683: [VL] Support type widening in Parquet reader (SPARK-40876)
URL: https://github.com/apache/incubator-gluten/issues/11683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org
Re: [I] [VL] Support type widening in Parquet reader (SPARK-40876) [gluten]
Posted by "baibaichen (via GitHub)" <gi...@apache.org>.
baibaichen commented on issue #11683:
URL: https://github.com/apache/gluten/issues/11683#issuecomment-4220665654
fixed in #11860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@gluten.apache.org
For additional commands, e-mail: commits-help@gluten.apache.org