You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/04/11 18:07:15 UTC
[I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]
alamb opened a new issue, #15689:
URL: https://github.com/apache/datafusion/issues/15689
### Describe the bug
As @xudong963 mentions in
- https://github.com/xudong963/arrow-datafusion/pull/5#discussion_r2034641672.
And also brought up again in
- https://github.com/apache/datafusion/pull/15661
When table_schema is different from file_schema then the current statistics merging code will incorrectly merge statistics
Specifically, it merges column statistics based on their ordinal position (order in the file)
Currently this isn't a huge problem as the statistics are only used in a limited way for some optimizations, but as we start to rely on statistics for correctness, such as https://github.com/apache/datafusion/issues/6672 it is more important
### To Reproduce
if we have two files
* File 1: `(a int32, b int32)`
* File 2: `(b int32, a int32)`
I think the code on main will combine statistics for columns a in File 1 and column `b` in File 2 together.
### Expected behavior
I expect that only statistics from the same logical column are merged together.
### Additional context
After https://github.com/apache/datafusion/pull/15661 is merged, I suggest:
1. adding some function that knows how to map columns from a file schema --> table schema (filling in any missing columns with `ColumnStatistics::new_unnown`) before combining them
2. Adding testst
Maybe we can simply reuse the existing [`SchemaMapper`](https://docs.rs/datafusion/latest/datafusion/datasource/schema_adapter/trait.SchemaMapper.html) / factory 🤔 so we are sure the statistics merging is consistent with runtime
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2829609951
Hi @friendlymatthew , friendly ping --how's it going?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2830174889
> oh [@xudong963](https://github.com/xudong963) if there's any follow up issues, feel free to ping me. I'm off next week and want to dive deeper into statistics
Sure, I think there will be
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2830135218
Hey @xudong963 -- I'm currently switching jobs so time's been a bit constrained at the moment. I've only had time to look over this last weekend, but didn't make much progress.
Feel free to take the issue, I'm eager to read your PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2800240401
@alamb Thank you for writing the issue in detail
@friendlymatthew Thank you for taking it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2830170142
oh @xudong963 if there's any follow up issues, feel free to ping me. I'm off next week and want to dive deeper into statistics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 closed issue #15689: ListingTable statistics improperly merges statistics when files have different schemas
URL: https://github.com/apache/datafusion/issues/15689
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]
Posted by "xudong963 (via GitHub)" <gi...@apache.org>.
xudong963 commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2830098107
Fyi, I'm cooking a PR to fix the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] ListingTable statistics improperly merges statistics when files have different schemas [datafusion]
Posted by "friendlymatthew (via GitHub)" <gi...@apache.org>.
friendlymatthew commented on issue #15689:
URL: https://github.com/apache/datafusion/issues/15689#issuecomment-2797733418
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org