[GitHub] [arrow-datafusion] boaz-codota commented on issue #899: Track memory usage for each individual operator

[GitBox <gi...@apache.org>]

boaz-codota commented on issue #899:
URL: https://github.com/apache/arrow-datafusion/issues/899#issuecomment-901911003


   This got me interesting, so I started looking into it, and I'm not sure how we aim to tackle it.
   
   My 1st idea was to write a Decorator which implements `ExecutionPlan` and which replace the allocator in some way, but that seems non-feasible due to the fact that it both requires replacing the `GlobalAllocator` (which is way more intrusive than I imagine DataFusion wanting to ever be), and will not work well-given concurrency (can't know what thread allocated that memory). An example for a `GlobalAllocator` Decorator can be found in: https://github.com/cuviper/alloc_geiger
   
   The other approach I found (from the article below) is implementing something similar to Servo's `malloc_size_of` which can be found in: https://github.com/servo/servo/blob/faf3a183f3755a9986ec4379abadf3523bd8b3c0/components/malloc_size_of/lib.rs
   This solution is quite intrusive (from what I can see) and requires manually "registering" any memory allocation to add up to a per `ExecutionPlan` sum.
   
   Not sure where to go from here, would love to hear some feedback.
   
   Some references: https://rust-analyzer.github.io/blog/2020/12/04/measuring-memory-usage-in-rust.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org