You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@arrow.apache.org by "hahn80 (via GitHub)" <gi...@apache.org> on 2026/03/09 02:17:22 UTC
[D] Memory Used for GrouBy or Join [arrow]
GitHub user hahn80 created a discussion: Memory Used for GrouBy or Join
In Apache Arrow Acero C++, group_by builds an internal aggregation map, while hash join builds an internal hash table.
How can we measure or track the memory usage specifically allocated for these two structures during execution?
Thank you!
GitHub link: https://github.com/apache/arrow/discussions/49469
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Memory Used for GrouBy or Join [arrow]
Posted by "hahn80 (via GitHub)" <gi...@apache.org>.
GitHub user hahn80 added a comment to the discussion: Memory Used for GrouBy or Join
Thank you very much for your inside. I will follow your idea to test it. I am working with CPP, so I probably track a bit deeper into what acero offers.
GitHub link: https://github.com/apache/arrow/discussions/49469#discussioncomment-16073064
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Memory Used for GrouBy or Join [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
GitHub user zanmato1984 added a comment to the discussion: Memory Used for GrouBy or Join
It’s only supported in Arrow C++ today. You can pass a dedicated `MemoryPool` via `ExecContext` for the plan and query `bytes_allocated()` (and `max_memory()` for peak). This gives a plan‑level estimate because other operators also allocate from that pool. For hash‑table‑only memory, you’d need to route allocations for `SwissJoin` (and/or hash aggregate) to a separate pool, which currently requires finer‑grained plumbing or instrumentation.
GitHub link: https://github.com/apache/arrow/discussions/49469#discussioncomment-16064051
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org
Re: [D] Memory Used for GrouBy or Join [arrow]
Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
GitHub user zanmato1984 added a comment to the discussion: Memory Used for GrouBy or Join
Great. Let me know if you have any further questions! Thanks.
GitHub link: https://github.com/apache/arrow/discussions/49469#discussioncomment-16089142
----
This is an automatically sent email for user@arrow.apache.org.
To unsubscribe, please send an email to: user-unsubscribe@arrow.apache.org