[jira] [Created] (SPARK-59251) Reject incompatible Parquet row reader conversions

["Jiayi Wang (Jira)" <ji...@apache.org>]

Jiayi Wang created SPARK-59251:
----------------------------------

             Summary: Reject incompatible Parquet row reader conversions
                 Key: SPARK-59251
                 URL: https://issues.apache.org/jira/browse/SPARK-59251
             Project: Spark
          Issue Type: Bug
          Components: Bug
    Affects Versions: 4.2.0
            Reporter: Jiayi Wang


### Problem

Spark's row-based Parquet reader and vectorized Parquet reader handle some incompatible requested
schemas differently. The vectorized reader raises `SchemaColumnConvertNotSupportedException`, while
the row-based reader silently interprets the physical value as the requested Catalyst type.

Two examples are:

| Parquet file type | Requested Spark type | Row-based reader | Vectorized reader |
| --- | --- | --- | --- |
| `FIXED_LEN_BYTE_ARRAY(4)` | `STRING` | Returns the raw bytes as UTF-8 | Rejects the conversion |
| `INT32 (DATE)` | `DECIMAL(10, 0)` | Returns the day count as a decimal | Rejects the conversion |

The row reader's `ParquetRowConverter` currently accepts every binary-like primitive as a string
and treats any `INT32` or `INT64` without decimal metadata as an unannotated integer-backed
decimal. The latter ignores semantic logical annotations such as `DATE`.

This is a correctness problem because changing `spark.sql.parquet.enableVectorizedReader` can
change a query from failing cleanly to returning incorrectly interpreted data.

### Expected behavior

Both readers should reject these unsupported conversions with
`FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`.

Supported conversions should remain unchanged, including Parquet `BINARY` to Spark `STRING` and
unannotated or signed-integer `INT32`/`INT64` to a sufficiently compatible Spark decimal type.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org