You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/01/29 00:42:13 UTC
[I] [native_datafusion] [Spark SQL Tests] Schema incompatibility tests expect exceptions that native_datafusion handles gracefully [datafusion-comet]
andygrove opened a new issue, #3311:
URL: https://github.com/apache/datafusion-comet/issues/3311
## Summary
8 Spark SQL tests fail because they expect a `SparkException` when reading Parquet with mismatched schemas, but `native_datafusion` handles type widening/mismatches gracefully instead of throwing.
## Failing Tests
- `ParquetSchemaSuite`: "schema mismatch failure error message for parquet vectorized reader"
- `ParquetSchemaSuite`: "SPARK-45604: schema mismatch failure error on timestamp_ntz to array<timestamp_ntz>"
- `ParquetQuerySuite`: "SPARK-36182: can't read TimestampLTZ as TimestampNTZ"
- `ParquetQuerySuite`: "SPARK-34212 Parquet should read decimals correctly"
- `ParquetIOSuite`: "SPARK-35640: read binary as timestamp should throw schema incompatible error"
- `ParquetIOSuite`: "SPARK-35640: int as long should throw schema incompatible error"
- `FileBasedDataSourceSuite`: "Spark native readers should respect spark.sql.caseSensitive"
- `ParquetReadSuite`: "row group skipping doesn't overflow when reading into larger type"
## Root Cause
These tests assert that reading with a mismatched schema throws an exception. Native DataFusion silently handles these cases (type widening, schema evolution). This is a test assumptions issue — the tests need guards to skip when `native_datafusion` is active, or expectations need updating.
## Related
Discovered in CI for #3307 (enable native_datafusion in auto scan mode).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [native_datafusion] [Spark SQL Tests] Schema incompatibility tests expect exceptions that native_datafusion handles gracefully [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #3311:
URL: https://github.com/apache/datafusion-comet/issues/3311#issuecomment-4106650400
This issue is now a partial duplicate of #3720
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [native_datafusion] [Spark SQL Tests] Schema incompatibility tests expect exceptions that native_datafusion handles gracefully [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #3311:
URL: https://github.com/apache/datafusion-comet/issues/3311#issuecomment-4119955199
This issue is now replaces with https://github.com/apache/datafusion-comet/issues/3720 and https://github.com/apache/datafusion-comet/issues/3760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [native_datafusion] [Spark SQL Tests] Schema incompatibility tests expect exceptions that native_datafusion handles gracefully [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #3311: [native_datafusion] [Spark SQL Tests] Schema incompatibility tests expect exceptions that native_datafusion handles gracefully
URL: https://github.com/apache/datafusion-comet/issues/3311
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org