partitions number with variable number of cores

[Jonathan Esterhazy <je...@groupon.com>]

I use Spark in a cluster shared with other applications. The number of
nodes (and cores) assigned to my job varies depending on how many unrelated
jobs are running in the same cluster.

Is there any way for me to determine at runtime how many cores have been
allocated to my job, so I can select an appropriate partitioning strategy?

I've tried calling of SparkContext.getExecutorMemoryStatus.size, but if I
call this early in the job (which is when I want this info), the executors
haven't attached yet, and I get 0.

Has anyone else found a way to dynamically adjust their partitions to match
unpredictable node allocation?

Re: partitions number with variable number of cores

[Gen <ge...@gmail.com>]

Maybe I am wrong, but how many resource that a spark application can use
depends on the mode of deployment(the type of resource manager), you can
take a look at  https://spark.apache.org/docs/latest/job-scheduling.html
<https://spark.apache.org/docs/latest/job-scheduling.html>  . 

For your case, I think mesos is better which can realize the dynamic sharing
of CPU cores

Best



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/partitions-number-with-variable-number-of-cores-tp15367p15710.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org