You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "sankalpsthakur (via GitHub)" <gi...@apache.org> on 2026/10/07 10:16:24 UTC
[PR] [SPARK-60010][SQL] Fail vectorized Parquet reads of a non-zero-scale DECIMAL as an integral type [spark]
sankalpsthakur opened a new pull request, #59268:
URL: https://github.com/apache/spark/pull/59268
### What changes were proposed in this pull request?
`ParquetVectorUpdaterFactory.getUpdater` accepted a DECIMAL-annotated INT32 column for a `ByteType`, `ShortType`, `IntegerType` or `LongType` read type (the last through the INT32 -> long widening path), and a DECIMAL-annotated INT64 column for `LongType`. It never checked the annotation, so it returned the stored unscaled value.
This PR adds `canReadAsIntegral(descriptor)` and requires it on those five plain integral branches. It is false only for a `DecimalLogicalTypeAnnotation` with a non-zero scale. A rejected read falls through to the existing `constructConvertNotSupportedException`, the path that already rejects DECIMAL(10, 2) (INT64) read as int. The user sees `FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`. The decimal branches (`canReadAs*Decimal`) are unchanged.
A DECIMAL with scale 0 stores the integral value itself, so it can still be read as an integral type. The row reader returns the same values for those inputs, both on master and with #59080.
This is the vectorized counterpart of #59080 (SPARK-59251). That PR makes the row-based reader (`ParquetRowConverter`) reject incompatible primitive conversions with the same error. This PR does not change the row reader.
### Why are the changes needed?
Reading Parquet data with a user-specified schema silently returns wrong values:
```scala
spark.sql("SELECT CAST(123.45 AS DECIMAL(9, 2)) AS v").write.parquet("/tmp/dec92")
spark.read.schema("v INT").parquet("/tmp/dec92").show()
// before: 12345, the unscaled value
// after: [FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH] ... Data type mismatches when reading
// Parquet column [v]. Expected Spark type int, actual Parquet type INT32.
```
The same read of a DECIMAL(10, 2) column (INT64) already failed with that error, so the outcome depended on the decimal's precision. Reading DECIMAL(10, 2) as bigint also returned 12345.
### Does this PR introduce _any_ user-facing change?
Yes. This applies to the vectorized Parquet reader, which is the default. Two reads of a DECIMAL with a non-zero scale used to return the unscaled value and now fail with `FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`:
- a DECIMAL stored as INT32 (precision <= 9), read as tinyint, smallint, int or bigint
- a DECIMAL stored as INT64, read as bigint
DECIMAL with scale 0 and reads as a decimal type are unaffected.
Notes for reviewers:
- A query that depended on the old (wrong) result now fails. Reviewers may want a migration guide note or a legacy flag. This PR adds neither.
- For these inputs the two readers now disagree. The parquet-mr row reader is used when the vectorized reader is disabled or not applicable. It still returns the unscaled value, on master and with #59080 as it stands. Aligning it is a follow-up, possibly within #59080's scope.
- The same blind spot exists for other non-decimal read types of a DECIMAL-annotated INT32/INT64 column: double, date and ANSI intervals. They are left out to keep this PR to the integral types in the JIRA.
### How was this patch tested?
New tests:
- `ParquetTypeWideningSuite`, `SPARK-60010: unsupported parquet conversion DecimalType(9,2) / DecimalType(18,2) -> ByteType / ShortType / IntegerType / LongType` (8 tests):
- Files are written with dictionary encoding on and off, using the Parquet v1 and v2 writers, and read with the vectorized reader.
- Each test checks `FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH` with column `[a]`, the expected Spark type, and actual type INT32 or INT64.
- `ParquetTypeWideningSuite`, `SPARK-60010: parquet conversion DecimalType(9,0) -> ByteType / ShortType / IntegerType / LongType` and `DecimalType(18,0) -> LongType` (5 tests): both readers still read scale-0 decimals as integral values.
- `ParquetVectorUpdaterSuite`, `SPARK-60010: factory rejects a DECIMAL with a non-zero scale read as an integral type`:
- `getUpdater` throws `SchemaColumnConvertNotSupportedException` for INT32 DECIMAL(9, 2), INT64 DECIMAL(9, 2) and INT64 DECIMAL(18, 2) with all four integral types.
- It still returns an updater for INT32 DECIMAL(9, 0) and INT64 DECIMAL(18, 0).
No existing test or SQL golden file depended on the old behavior. I checked the Parquet-related tests in sql/core and sql/hive and the golden files, so none were updated.
Nothing was built or run locally. Validation is the fork's GitHub Actions run for this branch (the Build check on this PR).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-60010][SQL] Fail vectorized Parquet reads of a non-zero-scale DECIMAL as an integral type [spark]
Posted by "sankalpsthakur (via GitHub)" <gi...@apache.org>.
sankalpsthakur commented on PR #59268:
URL: https://github.com/apache/spark/pull/59268#issuecomment-6150054202
ci green, tests cover the non-zero-scale decimal read path, ptal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org