You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2026/02/12 00:34:55 UTC

[PR] fix: [df52] handle case-insensitive column matching in PhysicalExprAdapter [datafusion-comet]

andygrove opened a new pull request, #3495:
URL: https://github.com/apache/datafusion-comet/pull/3495

   ## Summary
   
   - Fix case-insensitive column matching in the new `SparkPhysicalExprAdapterFactory` on the df52 branch
   - `DefaultPhysicalExprAdapter` uses Arrow's `field_with_name`/`index_of` which are case-sensitive. When a parquet file has lowercase column names (e.g., "a") but the table schema has uppercase (e.g., "A"), the lookup fails and columns are filled with nulls
   - Fix remaps physical schema field names to match logical names (case-insensitively) before passing to the default adapter, then restores original physical names in the rewritten expressions so that downstream `reassign_expr_columns` can find columns in the actual parquet stream schema
   
   ## Test plan
   
   - [ ] Existing test "test V1 parquet native scan -- case insensitive" in `ParquetReadV1Suite` should now pass
   - [ ] Existing Rust tests continue to pass
   - [ ] CI passes
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: [df52] handle case-insensitive column matching in PhysicalExprAdapter [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #3495:
URL: https://github.com/apache/datafusion-comet/pull/3495


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: [df52] handle case-insensitive column matching in PhysicalExprAdapter [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #3495:
URL: https://github.com/apache/datafusion-comet/pull/3495#issuecomment-3891089703

   ## CI Comparison: PR #3495 vs df52 baseline (PR #3470)
   
   | | PR #3470 (df52 baseline) | PR #3495 (with this fix) |
   |---|---|---|
   | **Failing CI jobs** | 34 | 23 |
   
   ### Jobs fixed by this PR (fail in #3470, pass in #3495) — 11 jobs
   
   - `macos-14/Spark 3.4, JDK 11, Scala 2.12 [parquet]`
   - `macos-14/Spark 3.5, JDK 17, Scala 2.13 [parquet]`
   - `macos-14/Spark 4.0, JDK 17, Scala 2.13 [parquet]`
   - `spark-sql-auto-sql_hive-1/spark-3.4.3`
   - `spark-sql-auto-sql_hive-1/spark-3.5.8`
   - `spark-sql-native_datafusion-sql_hive-2/spark-3.5.8`
   - `ubuntu-latest/Spark 3.4, JDK 11, Scala 2.12/auto [parquet]`
   - `ubuntu-latest/Spark 3.5, JDK 17, Scala 2.12/native_iceberg_compat [parquet]`
   - `ubuntu-latest/Spark 3.5.5, JDK 17, Scala 2.13/auto [parquet]`
   - `ubuntu-latest/Spark 3.5.6, JDK 17, Scala 2.13/auto [parquet]`
   - `ubuntu-latest/Spark 4.0, JDK 17/auto [parquet]`
   
   ### Regressions introduced by this PR (pass in #3470, fail in #3495)
   
   None — every failure in #3495 also exists in #3470.
   
   ### Summary
   
   This PR is a strict improvement over the df52 baseline: **11 fewer failing CI jobs with zero regressions**.
   
   ---
   > [!NOTE]
   > This comment was generated with the assistance of AI (Claude Code) and should be verified independently.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org