You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by yu...@apache.org on 2024/12/24 23:27:02 UTC
(yunikorn-site) branch master updated: [YUNIKORN-2919] Document spark-operator: Account spark.memory.offHeap.size (#508)
This is an automated email from the ASF dual-hosted git repository.
yuchaoran pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new c496925cba [YUNIKORN-2919] Document spark-operator: Account spark.memory.offHeap.size (#508)
c496925cba is described below
commit c496925cbaf6c9d7f5dfbb935a64f9afe06fd759
Author: Hsien-Cheng(Ryan) Huang <ry...@gmail.com>
AuthorDate: Wed Dec 25 07:26:57 2024 +0800
[YUNIKORN-2919] Document spark-operator: Account spark.memory.offHeap.size (#508)
---
docs/user_guide/workloads/run_spark.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/user_guide/workloads/run_spark.md b/docs/user_guide/workloads/run_spark.md
index fff6e1b2c2..deca293768 100644
--- a/docs/user_guide/workloads/run_spark.md
+++ b/docs/user_guide/workloads/run_spark.md
@@ -30,7 +30,7 @@ under the License.
:::note
Pre-requisites:
- This tutorial assumes YuniKorn is [installed](../../get_started/get_started.md) under the namespace `yunikorn`
-- Use spark-operator version >= 2.0 to enable support for YuniKorn gang scheduling
+- Use spark-operator version >= 2.1.0 to enable support for YuniKorn gang scheduling
:::
### Install YuniKorn
@@ -48,7 +48,7 @@ helm install yunikorn yunikorn/yunikorn --create-namespace --namespace yunikorn
We should install `spark-operator` with `controller.batchScheduler.enable=true` and set `controller.batchScheduler.default=yunikorn` to enable Gang Scheduling. It's optional to set the default scheduler to YuniKorn since you can specify it later on, but it's recommended to do so.
Also, note that the total requested memory for the Spark job is the sum of memory requested for the driver and that for all executors, where each is computed as below:
* Driver requested memory = `spark.driver.memory` + `spark.driver.memoryOverhead`
-* Executor requested memory = `spark.executor.memory` + `spark.executor.memoryOverhead` + `spark.executor.pyspark.memory`
+* Executor requested memory = `spark.executor.memory` + `spark.executor.memoryOverhead` + `spark.executor.pyspark.memory` + `spark.memory.offHeap.size`
```shell script
helm repo add spark-operator https://kubeflow.github.io/spark-operator
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org