[I] native_datafusion: BINARY column without DecimalLogicalTypeAnnotation should be rejected when read as DECIMAL [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove opened a new issue, #4351:
URL: https://github.com/apache/datafusion-comet/issues/4351

   ## Description
   
   When a Parquet column is plain BINARY (no `DecimalLogicalTypeAnnotation`), Spark's vectorized reader rejects reading it as a `DecimalType` via `ParquetVectorUpdaterFactory.getUpdater`'s `BINARY` case (lines 199-205): `canReadAsDecimal` and `canReadAsBinaryDecimal` both require the column to have a `DecimalLogicalTypeAnnotation`. Without the annotation, Spark falls through and throws `SchemaColumnConvertNotSupportedException`.
   
   `native_datafusion`'s `schema_adapter.rs` currently allows this conversion silently. The first rejection block (lines 599-619) lists `Decimal128(_, _) | Decimal256(_, _)` in the allowed targets, intending to permit a "binary-encoded decimal", but Arrow already exposes a Parquet BINARY column with `DecimalLogicalTypeAnnotation` as `DataType::Decimal128`, never as `DataType::Binary`. So when `physical_type == Binary` is observed in the adapter, the source is unambiguously a non-decimal BINARY column.
   
   When the cast falls through, DataFusion can't actually cast Binary to Decimal128; the column reaches Arrow's `RecordBatch::try_new` validation and fails with:
   
   ```
   Invalid argument error: column types must match schema types, expected Decimal128(37, 1) but found Binary at column index 0
   ```
   
   surfaced to the JVM as `CometNativeException` instead of the Spark-equivalent `SchemaColumnConvertNotSupportedException`.
   
   ## Reproducer
   
   `SPARK-34212 Parquet should read decimals correctly` in `ParquetQuerySuite` (Spark 4.1.x):
   
   ```scala
   val df = sql(s"SELECT 1 a, 123456 b, ${Int.MaxValue.toLong * 10} c, CAST('1.2' AS BINARY) d")
   df.write.parquet(path.toString)
   
   withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true") {
     Seq("a DECIMAL(3, 2)", "c DECIMAL(18, 1)", "d DECIMAL(37, 1)").foreach { schema =>
       val e = intercept[SparkException] {
         readParquet(schema, path).collect()
       }.getCause
       assert(e.isInstanceOf[SchemaColumnConvertNotSupportedException])
     }
   }
   ```
   
   The `d DECIMAL(37, 1)` iteration fails because `native_datafusion` doesn't reject the BINARY → DECIMAL read.
   
   ## Fix
   
   Drop `DataType::Decimal128(_, _) | DataType::Decimal256(_, _)` from the allowed-targets match at `native/core/src/parquet/schema_adapter.rs:608-609`. Add a regression test in the `parquet_string_read_as_int_errors` style.
   
   ## Related
   
   - #3720 (parent: native_datafusion silent schema-mismatch acceptance)
   - #4297 / #4343 / #4344 (other rejection-class gaps already addressed)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] native_datafusion: BINARY column without DecimalLogicalTypeAnnotation should be rejected when read as DECIMAL [datafusion-comet]

["andygrove (via GitHub)" <gi...@apache.org>]

andygrove closed issue #4351: native_datafusion: BINARY column without DecimalLogicalTypeAnnotation should be rejected when read as DECIMAL
URL: https://github.com/apache/datafusion-comet/issues/4351


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org