(datafusion-comet) branch main updated: doc: Add memory tuning section to user guide (#684)

[vi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

viirya pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git


The following commit(s) were added to refs/heads/main by this push:
     new 1f77953e doc: Add memory tuning section to user guide (#684)
1f77953e is described below

commit 1f77953e60ca5cdee972590402d4f05144098249
Author: Liang-Chi Hsieh <vi...@gmail.com>
AuthorDate: Fri Jul 19 20:11:03 2024 -0700

    doc: Add memory tuning section to user guide (#684)
    
    * doc: Add memory tuning section to user guide
    
    * Update docs/source/user-guide/tuning.md
    
    Co-authored-by: Andy Grove <ag...@apache.org>
    
    * More
    
    ---------
    
    Co-authored-by: Andy Grove <ag...@apache.org>
---
 docs/source/user-guide/tuning.md | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/docs/source/user-guide/tuning.md b/docs/source/user-guide/tuning.md
index f46ab9e0..876f498d 100644
--- a/docs/source/user-guide/tuning.md
+++ b/docs/source/user-guide/tuning.md
@@ -21,6 +21,21 @@ under the License.
 
 Comet provides some tuning options to help you get the best performance from your queries.
 
+## Memory Tuning
+
+Comet currently doesn't share the memory allocation from Spark but owns its own memory allocation.
+That's said, Comet requires additional memory to be allocated. Comet provides some memory related configurations to help you tune the memory usage.
+
+By default, the amount of memory is `spark.comet.memory.overhead.factor` * `spark.executor.memory`.
+The default value for `spark.comet.memory.overhead.factor` is 0.2. You can increase the factor to require more
+memory for Comet to use, if you see OOM error. Generally, increasing memory overhead will improve the performance of your queries.
+For example, some operators like `SortMergeJoin` and `HashAggregate` may require more memory to run.
+Once the memory is not enough, the operator will spill to disk, which will slow down the query.
+
+Besides, you can also set the memory explicitly by setting `spark.comet.memoryOverhead` to the desired value.
+Comet will allocate at least `spark.comet.memory.overhead.min` memory.
+
+If both `spark.comet.memoryOverhead` and `spark.comet.memory.overhead.factor` are set, the former will be used.
 
 ## Shuffle
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org