Re: [D] Add a new backend: Bolt [incubator-gluten]

["wangxinshuo-bolt (via GitHub)" <gi...@apache.org>]

GitHub user wangxinshuo-bolt added a comment to the discussion: Add a new backend: Bolt

> > Bolt can dynamically over-issue memory after reaching the off-heap configuration value based on the actual physical memory usage, thereby reducing the amount of spilled data, improving performance, and reducing the risk of OOM.
> 
> Looking forward to see this open-sourced — especially how it helps avoid OOM. We’ve been struggling with that a lot.

@boneanxs We have implemented all memory management functions in Bolt, so `Bolt can dynamically over-issue memory after reaching the off-heap configuration value based on the actual physical memory usage`. The specific code is here:
https://github.com/bytedance/bolt/blob/c9b1507b696edbae28d5133d6d512a1eac77e446/bolt/common/memory/sparksql/ExecutionMemoryPool.cpp#L347-L419

We will also refactor the memory module later to avoid shortcomings of the existing mechanism. All planning and discussions will be conducted in the bolt repo's issues.

GitHub link: https://github.com/apache/incubator-gluten/discussions/10929#discussioncomment-15738427

----
This is an automatically sent email for dev@gluten.apache.org.
To unsubscribe, please send an email to: dev-unsubscribe@gluten.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@gluten.apache.org
For additional commands, e-mail: dev-help@gluten.apache.org