You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/04/11 18:07:15 UTC

[I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]

alamb opened a new issue, #15689:
URL: https://github.com/apache/datafusion/issues/15689

   ### Describe the bug
   
   As @xudong963  mentions in 
   - https://github.com/xudong963/arrow-datafusion/pull/5#discussion_r2034641672.
   
   And also brought up again in 
   - https://github.com/apache/datafusion/pull/15661
   
   When table_schema is different from file_schema then the current statistics merging code will incorrectly merge statistics
   
   Specifically, it merges column statistics based on their ordinal position (order in the file) 
   
   Currently this isn't a huge problem as the statistics are only used in a limited way for some optimizations, but as we start to rely on statistics for correctness, such as https://github.com/apache/datafusion/issues/6672 it is more important
   
   ### To Reproduce
   
   if we have two files
   * File 1: `(a int32, b int32)`
   * File 2: `(b int32, a int32)`
   
   I think the code on main will combine statistics for columns a in File 1 and column `b` in File 2 together. 
   
   
   
   ### Expected behavior
   
   I expect that only statistics from the same logical column are merged together. 
   
   
   
   
   ### Additional context
   
   After https://github.com/apache/datafusion/pull/15661 is merged, I suggest:
   1. adding some function that knows how to map columns from a file schema --> table schema (filling in any missing columns with `ColumnStatistics::new_unnown`) before combining them
   2. Adding testst
   
   Maybe we can simply reuse  the existing [`SchemaMapper`](https://docs.rs/datafusion/latest/datafusion/datasource/schema_adapter/trait.SchemaMapper.html) / factory 🤔  so we are sure the statistics merging is consistent with runtime


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]

Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2829609951

   Hi @friendlymatthew , friendly ping --how's it going?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]

Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2830174889

   > oh [@xudong963](https://github.com/xudong963) if there's any follow up issues, feel free to ping me. I'm off next week and want to dive deeper into statistics
   
   Sure, I think there will be


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]

Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2830135218

   Hey @xudong963 -- I'm currently switching jobs so time's been a bit constrained at the moment. I've only had time to look over this last weekend, but didn't make much progress.
   
   Feel free to take the issue, I'm eager to read your PR!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]

Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2800240401

   @alamb Thank you for writing the issue in detail
   @friendlymatthew Thank you for taking it
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]

Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2830170142

   oh @xudong963 if there's any follow up issues, feel free to ping me. I'm off next week and want to dive deeper into statistics


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]

Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 closed issue #15689: ListingTable statistics improperly merges statistics when files have different schemas
URL: https://github.com/apache/datafusion/issues/15689


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]

Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2830098107

   Fyi, I'm cooking a PR to fix the issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]

Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2797733418

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org