You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/03/22 17:39:56 UTC

[I] native_datafusion: case-insensitive mode doesn't detect duplicate/ambiguous Parquet fields [datafusion-comet]

andygrove opened a new issue, #3760:
URL: https://github.com/apache/datafusion-comet/issues/3760

   ## Description
   
   When running Spark SQL tests with `native_datafusion` scan, tests that expect errors for duplicate or ambiguous fields in case-insensitive mode fail because DataFusion's Parquet reader doesn't enforce Spark's case-sensitivity validation rules.
   
   ## Affected Tests
   
   ### `Spark native readers should respect spark.sql.caseSensitive` (FileBasedDataSourceSuite)
   
   Writes a Parquet file with columns `A`, `b`, `B`, then reads with `caseSensitive=false`. Spark expects a `SparkException` when selecting `b` (ambiguous between `b` and `B`), but `native_datafusion` reads without error.
   
   ### `SPARK-25207: exception when duplicate fields in case-insensitive mode` (ParquetFilterSuite, V1 and V2)
   
   Writes Parquet with columns `A`, `B`, `b`, then reads with `caseSensitive=false`. Spark expects a `SparkException` with cause `RuntimeException` containing `Found duplicate field(s) "B": [B, b]`. The native reader either doesn't detect the duplicate, or wraps the error with a different exception type/cause than expected.
   
   ## Context
   
   PR #3687 added a fallback from `native_datafusion` for duplicate fields in case-insensitive mode, avoiding the test failures by falling back to the Spark reader. These tests remain ignored because the native reader itself doesn't implement the validation.
   
   ## Related
   
   - #3687: fall back from native_datafusion for duplicate fields in case-insensitive mode
   - Split from #3311


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] native_datafusion: case-insensitive mode doesn't detect duplicate/ambiguous Parquet fields [datafusion-comet]

Posted by "vaibhawvipul (via GitHub)" <gi...@apache.org>.
vaibhawvipul commented on issue #3760:
URL: https://github.com/apache/datafusion-comet/issues/3760#issuecomment-4141459506

   I would love to pick it up, if no one else is working on it. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] native_datafusion: case-insensitive mode doesn't detect duplicate/ambiguous Parquet fields [datafusion-comet]

Posted by "vaibhawvipul (via GitHub)" <gi...@apache.org>.
vaibhawvipul commented on issue #3760:
URL: https://github.com/apache/datafusion-comet/issues/3760#issuecomment-4141458045

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] native_datafusion: case-insensitive mode doesn't detect duplicate/ambiguous Parquet fields [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #3760: native_datafusion: case-insensitive mode doesn't detect duplicate/ambiguous Parquet fields
URL: https://github.com/apache/datafusion-comet/issues/3760


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org