[GitHub] [arrow-datafusion] Dandandan commented on issue #7149: Top-K query optimization suboptimal memory usage without custom allocators

["Dandandan (via GitHub)" <gi...@apache.org>]

Dandandan commented on issue #7149:
URL: https://github.com/apache/arrow-datafusion/issues/7149#issuecomment-1658340758

   Hi @gruuya thanks for the report. I don't have a direct answer
   
   Two things:
   
   * It looks like you're testing in debug mode? What happens if you build in release mode?
   * The sorting / top-k optimizations are not designed with a certain allocator in mind, but I think we've seen some interesting things with some allocators not actively freeing memory (e.g. when memory is fragmented). It could also be that different allocators report memory usage differently. I think you did some experiments as well at influxDB IOX with some allocators @alamb @mkmik 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org