You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "2010YOUY01 (via GitHub)" <gi...@apache.org> on 2025/07/09 07:19:29 UTC
[I] Support memory profiling in benchmarks [datafusion]
2010YOUY01 opened a new issue, #16720:
URL: https://github.com/apache/datafusion/issues/16720
### Is your feature request related to a problem or challenge?
Currently in DataFusion's benchmark: it only measures execution time. It would be helpful to also measure the total memory used.
### Describe the solution you'd like
Approach 1: Implement memory tracker inside operators, update max RSS for each poll, and collect the stats when the query finished.
Comet has implemented something similar https://github.com/apache/datafusion-comet/pull/1702
Approach 2: Profile memory stats externally in the benchmark script
I'm now slightly leaning toward approach 2, as it doesn't add complexity to the core.
cc @ding-young
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support memory profiling in benchmarks [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 closed issue #16720: Support memory profiling in benchmarks
URL: https://github.com/apache/datafusion/issues/16720
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support memory profiling in benchmarks [datafusion]
Posted by "ding-young (via GitHub)" <gi...@apache.org>.
ding-young commented on issue #16720:
URL: https://github.com/apache/datafusion/issues/16720#issuecomment-3051614458
I think it would be super helpful to include max RSS or other memory metrics in the benchmark results. Recently, I've been manually profiling memory usage with tools like heaptrack or by printing out RSS during execution. One challenge, though, is that the `cargo command` itself interferes with capturing precise memory usage, since it includes the overhead from the compilation or runtime setup. So ideally, we should build the binary first and then measure memory usage from the standalone executable. However, that’s not always straightforward because it's not obvious where the final binary is located.
Anyway, I'll try to figure out how to isolate and report memory usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Support memory profiling in benchmarks [datafusion]
Posted by "ding-young (via GitHub)" <gi...@apache.org>.
ding-young commented on issue #16720:
URL: https://github.com/apache/datafusion/issues/16720#issuecomment-3051587282
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org