Determine number of running executors

[Tobias Pfeiffer <tg...@preferred.jp>]

Hi,

when running on YARN, is there a way for the Spark driver to know how many
executors, cores per executor etc. there are? I want to know this so I can
repartition to a good number.

Thanks
Tobias

Re: Determine number of running executors

[Tobias Pfeiffer <tg...@preferred.jp>]

Hi,

On Sat, Jan 17, 2015 at 3:05 AM, Shuai Zheng <sz...@gmail.com> wrote:
>
> Can you share more information about how do you do that? I also have
> similar question about this.
>

Not very proud about it ;-), but here you go:

// find the number of workers available to us.
val _runCmd = scala.util.Properties.propOrElse("sun.java.command", "")
val numCoresRe = ".*--executor-cores ([0-9]+) --num-executors ([0-9]+).*".r
val totalNumCores = _runCmd match {
  case numCoresRe(coresPerExecutor, numExecutors) =>
    coresPerExecutor.toInt * numExecutors.toInt
  case _ =>
    0
}
if (totalNumCores > 0)
  logger.debug("total number of cores: " + totalNumCores)
else
  logger.warn("could not extract number of cores from run command: " + _runCmd)

Tobias

RE: Determine number of running executors

[Shuai Zheng <sz...@gmail.com>]

Hi Tobias,

 

Can you share more information about how do you do that? I also have similar question about this.

 

Thanks a lot,

 

Regards,

 

Shuai

 

From: Tobias Pfeiffer [mailto:tgp@preferred.jp] 
Sent: Wednesday, November 26, 2014 12:25 AM
To: Sandy Ryza
Cc: Yanbo Liang; user
Subject: Re: Determine number of running executors

 

Hi,

 

Thanks for your help!

 

Sandy, I had a bit of trouble finding the spark.executor.cores property. (It wasn't there although its value should have been 2.)

I ended up throwing regular expressions on scala.util.Properties.propOrElse("sun.java.command", ""), which worked surprisingly well ;-)

 

Thanks

Tobias

 

Re: Determine number of running executors

[Tobias Pfeiffer <tg...@preferred.jp>]

Hi,

Thanks for your help!

Sandy, I had a bit of trouble finding the spark.executor.cores property.
(It wasn't there although its value should have been 2.)
I ended up throwing regular expressions
on scala.util.Properties.propOrElse("sun.java.command", ""), which worked
surprisingly well ;-)

Thanks
Tobias

Re: Determine number of running executors

[Sandy Ryza <sa...@cloudera.com>]

Hi Tobias,

One way to find out the number of executors is through
SparkContext#getExecutorMemoryStatus.  You can find out the number of by
asking the SparkConf for the "spark.executor.cores" property, which, if not
set, means 1 for YARN.

-Sandy


On Fri, Nov 21, 2014 at 1:30 AM, Yanbo Liang <ya...@gmail.com> wrote:

> You can get parameter such as spark.executor.memory, but you can not get
> executor or core numbers.
> Because executor and core are parameters of spark deploy environment not
> spark context.
>
> val conf = new SparkConf().set("spark.executor.memory","2G")
> val sc = new SparkContext(conf)
>
> sc.getConf.get("spark.executor.memory")
> conf.get("spark.executor.memory")
>
> 2014-11-21 15:35 GMT+08:00 Tobias Pfeiffer <tg...@preferred.jp>:
>
>> Hi,
>>
>> when running on YARN, is there a way for the Spark driver to know how
>> many executors, cores per executor etc. there are? I want to know this so I
>> can repartition to a good number.
>>
>> Thanks
>> Tobias
>>
>
>

Re: Determine number of running executors

[Yanbo Liang <ya...@gmail.com>]

You can get parameter such as spark.executor.memory, but you can not get
executor or core numbers.
Because executor and core are parameters of spark deploy environment not
spark context.

val conf = new SparkConf().set("spark.executor.memory","2G")
val sc = new SparkContext(conf)

sc.getConf.get("spark.executor.memory")
conf.get("spark.executor.memory")

2014-11-21 15:35 GMT+08:00 Tobias Pfeiffer <tg...@preferred.jp>:

> Hi,
>
> when running on YARN, is there a way for the Spark driver to know how many
> executors, cores per executor etc. there are? I want to know this so I can
> repartition to a good number.
>
> Thanks
> Tobias
>