You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "antoine (JIRA)" <ji...@apache.org> on 2018/10/31 10:00:22 UTC
[jira] [Commented] (SPARK-22354) --executor-cores in spark-submit
fails to set "spark.executor.cores" for mesos workers
[ https://issues.apache.org/jira/browse/SPARK-22354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16669855#comment-16669855 ]
antoine commented on SPARK-22354:
---------------------------------
I'm not 100% sure but it seems I have a similar issue.
I have a Mesos test cluster of 2 nodes with 4 cores each.
If I use spark-shell everything is good and I can use 2 3 or 7 cores without issue.
Though when using spark-submit with mesos-dispatcher service on. I cannot manage to get more than one server cores use at the same time by one job.
If I try :
{{bin/spark-submit --class SparkPi --master mesos://mesosdispatcher:7077 --deploy-mode cluster --executor-memory 2G --conf "spark.executor.cores=2" --conf "spark.cores.max=6" http://websrv/sparkpi_2.10-1.0.jar 6000}}
One of my two mesos node will have two workers but using 4 cores max.
I'm not sure if it's related but I have the feeling this is a similar issue.
I'm using Spark 2.3.2 custom build with hadoop 2.9.1 and Mesos 1.7.0
> --executor-cores in spark-submit fails to set "spark.executor.cores" for mesos workers
> --------------------------------------------------------------------------------------
>
> Key: SPARK-22354
> URL: https://issues.apache.org/jira/browse/SPARK-22354
> Project: Spark
> Issue Type: Bug
> Components: Mesos
> Affects Versions: 2.2.0
> Environment: Mesos 1.0.1
> Spark 2.2.0
> Reporter: David McWhorter
> Priority: Major
>
> We are running spark in cluster-mode and limit the amount of CPU and memory per executor so that many executors spin up per mesos worker.
> When we specify --executor-cores 1 in the spark-submit command to the dispatcher, Mesos allocates only one CPU for the workers, but Spark itself thinks that it hasas many CPUs as are available for each worker and so only one spark worker starts per mesos worker. If we explicitly set --conf "spark.executor.cores=1" then this problem goes away and many spark workers spin up for each mesos worker.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org