You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by co...@apache.org on 2024/08/19 19:32:07 UTC
(datafusion-comet) branch main updated: doc: Document CometPlugin to start Comet in cluster mode (#836)
This is an automated email from the ASF dual-hosted git repository.
comphead pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git
The following commit(s) were added to refs/heads/main by this push:
new 27ab86be doc: Document CometPlugin to start Comet in cluster mode (#836)
27ab86be is described below
commit 27ab86bee87af76db075508b85c6630c0636607c
Author: Oleks V <co...@users.noreply.github.com>
AuthorDate: Mon Aug 19 12:32:02 2024 -0700
doc: Document CometPlugin to start Comet in cluster mode (#836)
* Documenting CometPlugin
Co-authored-by: Liang-Chi Hsieh <vi...@gmail.com>
---
docs/source/user-guide/installation.md | 3 +++
docs/source/user-guide/tuning.md | 11 +++++++++++
2 files changed, 14 insertions(+)
diff --git a/docs/source/user-guide/installation.md b/docs/source/user-guide/installation.md
index 45f988e2..c6900d6d 100644
--- a/docs/source/user-guide/installation.md
+++ b/docs/source/user-guide/installation.md
@@ -152,3 +152,6 @@ To enable columnar shuffle which supports all partitioning and basic complex typ
```
--conf spark.comet.exec.shuffle.mode=jvm
```
+
+### Memory tuning
+In addition to Apache Spark memory configuration parameters the Comet introduces own parameters to configure memory allocation for native execution. More [Comet Memory Tuning](./tuning.md)
\ No newline at end of file
diff --git a/docs/source/user-guide/tuning.md b/docs/source/user-guide/tuning.md
index 876f498d..9704bade 100644
--- a/docs/source/user-guide/tuning.md
+++ b/docs/source/user-guide/tuning.md
@@ -37,6 +37,17 @@ Comet will allocate at least `spark.comet.memory.overhead.min` memory.
If both `spark.comet.memoryOverhead` and `spark.comet.memory.overhead.factor` are set, the former will be used.
+## Memory Tuning using CometPlugin
+Configuring memory for Spark and Comet might be a tedious task as it requires to tune Spark executor overhead memory and Comet memory overhead configs. Comet provides a Spark plugin `CometPlugin` which can be set up to your Spark application to help memory settings.
+
+For users running the Comet in clusters like Kubernetes or YARN, `CometPlugin` can also make the resource manager respect correctly Comet memory parameters `spark.comet.memory*`.
+it is needed to pass to the starting command line additional Spark configuration parameter `--conf spark.plugins=org.apache.spark.CometPlugin`
+
+The resource managers respects Apache Spark memory configuration before starting the containers.
+
+The `CometPlugin` plugin overrides `spark.executor.memoryOverhead` adding up the Comet memory configuration.
+
+
## Shuffle
Comet provides Comet shuffle features that can be used to improve the performance of your queries.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org