You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2026/10/07 10:17:00 UTC
[jira] [Updated] (SPARK-60010) Reading a Parquet DECIMAL column with an integer schema returns the raw unscaled integer instead of honoring the decimal scale
[ https://issues.apache.org/jira/browse/SPARK-60010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ASF GitHub Bot updated SPARK-60010:
-----------------------------------
Labels: pull-request-available (was: )
> Reading a Parquet DECIMAL column with an integer schema returns the raw unscaled integer instead of honoring the decimal scale
> ------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60010
> URL: https://issues.apache.org/jira/browse/SPARK-60010
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.1
> Reporter: Brian Wu
> Priority: Major
> Labels: pull-request-available
>
> When a Parquet column physically stored as DECIMAL is read with an integer read-schema, Spark's behavior is unexpected: for low-precision decimals it silently returns the raw unscaled stored integer (ignoring the scale), while for higher-precision decimals it throws a type-mismatch error.
>
> Reproduction:
> {code:java}
> import org.apache.spark.sql.types._
> val intSchema = StructType(Seq(StructField("v", IntegerType)))
> Case A – DECIMAL(9,2), physically stored as INT32:
> spark.sql("SELECT CAST(123.45 AS DECIMAL(9,2)) AS v").write.mode("overwrite").parquet("/tmp/dec92")
> spark.read.schema(intSchema).parquet("/tmp/dec92").show()
> Result - silently returns the unscaled stored value (scale ignored)
> +-----+
> | v|
> +-----+
> |12345|
> +-----+
> Case B – DECIMAL(10,2), physically stored as INT64:
> spark.sql("SELECT CAST(123.45 AS DECIMAL(10,2)) AS v").write.mode("overwrite").parquet("/tmp/dec102")
> spark.read.schema(intSchema).parquet("/tmp/dec102").show()
> Result - throws
> org.apache.spark.SparkException: [FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH] Encountered error while reading file file:///tmp/dec102/part-00000-a59fef00-bb30-4788-a195-2f4c6173117e-c000.snappy.parquet. Data type mismatches when reading Parquet column [v]. Expected Spark type int, actual Parquet type INT64{code}
>
> [https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#numeric-types]
> Parquet standard does explicitly not define behavior here. However, it seems Spark behavior does not make sense in these cases. Suggest Spark should honor the schema type and not interpret the physical value as unscaled.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org