You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2019/05/19 04:30:00 UTC
[jira] [Updated] (SPARK-27754) Introduce spark on k8s config for
driver request cores
[ https://issues.apache.org/jira/browse/SPARK-27754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-27754:
----------------------------------
Description:
Spark on k8s supports config for specifying the executor cpu requests
(spark.kubernetes.executor.request.cores) but a similar config is missing
for the driver. Instead, currently spark.driver.cores value is used for integer value.
Although pod spec can have cpu for the fine-grained control like the following, this PR proposes additional configuration spark.kubernetes.driver.request.cores for driver request cores.
{code}
resources:
requests:
memory: "64Mi"
cpu: "250m"
{code}
was:
Spark on k8s supports config for specifying the executor cpu requests
(spark.kubernetes.executor.request.cores) but a similar config is missing
for the driver. Apparently `spark.driver.cores` works but its not evident that this accepts
fractional values (its defined as an Integer config but apparently accepts decimals). To keep in sync
with the executor config a similar driver config can be
introduced (spark.kubernetes.driver.request.cores) for explicitly specifying
the driver CPU requests. If not provided, the value will default to `spark.driver.cores` as before.
> Introduce spark on k8s config for driver request cores
> ------------------------------------------------------
>
> Key: SPARK-27754
> URL: https://issues.apache.org/jira/browse/SPARK-27754
> Project: Spark
> Issue Type: Improvement
> Components: Kubernetes
> Affects Versions: 3.0.0
> Reporter: Arun Mahadevan
> Priority: Minor
>
> Spark on k8s supports config for specifying the executor cpu requests
> (spark.kubernetes.executor.request.cores) but a similar config is missing
> for the driver. Instead, currently spark.driver.cores value is used for integer value.
> Although pod spec can have cpu for the fine-grained control like the following, this PR proposes additional configuration spark.kubernetes.driver.request.cores for driver request cores.
> {code}
> resources:
> requests:
> memory: "64Mi"
> cpu: "250m"
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org