You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "robert3005 (via GitHub)" <gi...@apache.org> on 2025/04/22 10:40:43 UTC
[I] Merging Statistics is slow [datafusion]
robert3005 opened a new issue, #15809:
URL: https://github.com/apache/datafusion/issues/15809
### Is your feature request related to a problem or challenge?
If you have a datasource that reports a sum statistic then merging statistics for a file group is considerably slower when compared to cases without sum statistic. Profiling reveals it's mostly due to `Precision<ScalarValue>::add` doing a lot of extra work.
### Describe the solution you'd like
Ideally we'd transpose the stats aggregation and aggregate them column wise instead of column group wise
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Merging Statistics is slow when sum statistic is present [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #15809: Merging Statistics is slow when sum statistic is present
URL: https://github.com/apache/datafusion/issues/15809
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Merging Statistics is slow when sum statistic is present [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on issue #15809:
URL: https://github.com/apache/datafusion/issues/15809#issuecomment-3407941988
Discussed today in a call: could we refactor the logic to combine statistics by first converting them to a RecordBatch and then evaluating the combination in a vectorized fashion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org