You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nicolas Mai <ni...@gmail.com> on 2014/07/24 20:16:58 UTC
Getting the number of slaves
Hi,
Is there a way to get the number of slaves/workers during runtime?
I searched online but didn't find anything :/ The application I'm working
will run on different clusters corresponding to different deployment stages
(beta -> prod). It would be great to get the number of slaves currently in
use, in order set the level of parallelism and RDD partitions, based on that
number.
Thanks!
Nicolas
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Getting-the-number-of-slaves-tp10604.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Getting the number of slaves
Posted by Tathagata Das <ta...@gmail.com>.
To clarify, that is the number of executors requested by the SparkContext
from the cluster manager.
On Tue, Jul 28, 2015 at 5:18 PM, amkcom <am...@gmail.com> wrote:
> try sc.getConf.getInt("spark.executor.instances", 1)
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Getting-the-number-of-slaves-tp10604p24043.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Getting the number of slaves
Posted by amkcom <am...@gmail.com>.
try sc.getConf.getInt("spark.executor.instances", 1)
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Getting-the-number-of-slaves-tp10604p24043.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Getting the number of slaves
Posted by Sung Hwan Chung <co...@cs.stanford.edu>.
Is there a more fixed way of doing this? E.g., if I submit a job with 10
executors, I want to see 10 all the time, and not a fluctuating number
based on currently available executors.
In a tight cluster with lots of jobs running, I can see that this number
goes up slowly and even down (when an executor fails). What I want is the
actual number of executors that was requested.
On Mon, Jul 28, 2014 at 4:06 PM, Zongheng Yang <zo...@gmail.com> wrote:
> Nicholas,
>
> The (somewhat common) situation you ran into probably meant the
> executors were still connecting. A typical solution is to sleep a
> couple seconds before querying that field.
>
> On Mon, Jul 28, 2014 at 3:57 PM, Andrew Or <an...@databricks.com> wrote:
> > Yes, both of these are derived from the same source, and this source
> > includes the driver. In other words, if you submit a job with 10
> executors
> > you will get back 11 for both statuses.
> >
> >
> > 2014-07-28 15:40 GMT-07:00 Sung Hwan Chung <co...@cs.stanford.edu>:
> >
> >> Do getExecutorStorageStatus and getExecutorMemoryStatus both return the
> >> number of executors + the driver?
> >> E.g., if I submit a job with 10 executors, I get 11 for
> >> getExeuctorStorageStatus.length and getExecutorMemoryStatus.size
> >>
> >>
> >> On Thu, Jul 24, 2014 at 4:53 PM, Nicolas Mai <ni...@gmail.com>
> >> wrote:
> >>>
> >>> Thanks, this is what I needed :) I should have searched more...
> >>>
> >>> Something I noticed though: after the SparkContext is initialized, I
> had
> >>> to
> >>> wait for a few seconds until sc.getExecutorStorageStatus.length returns
> >>> the
> >>> correct number of workers in my cluster (otherwise it returns 1, for
> the
> >>> driver)...
> >>>
> >>>
> >>>
> >>> --
> >>> View this message in context:
> >>>
> http://apache-spark-user-list.1001560.n3.nabble.com/Getting-the-number-of-slaves-tp10604p10619.html
> >>> Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
> >>
> >>
> >
>
Re: Getting the number of slaves
Posted by Zongheng Yang <zo...@gmail.com>.
Nicholas,
The (somewhat common) situation you ran into probably meant the
executors were still connecting. A typical solution is to sleep a
couple seconds before querying that field.
On Mon, Jul 28, 2014 at 3:57 PM, Andrew Or <an...@databricks.com> wrote:
> Yes, both of these are derived from the same source, and this source
> includes the driver. In other words, if you submit a job with 10 executors
> you will get back 11 for both statuses.
>
>
> 2014-07-28 15:40 GMT-07:00 Sung Hwan Chung <co...@cs.stanford.edu>:
>
>> Do getExecutorStorageStatus and getExecutorMemoryStatus both return the
>> number of executors + the driver?
>> E.g., if I submit a job with 10 executors, I get 11 for
>> getExeuctorStorageStatus.length and getExecutorMemoryStatus.size
>>
>>
>> On Thu, Jul 24, 2014 at 4:53 PM, Nicolas Mai <ni...@gmail.com>
>> wrote:
>>>
>>> Thanks, this is what I needed :) I should have searched more...
>>>
>>> Something I noticed though: after the SparkContext is initialized, I had
>>> to
>>> wait for a few seconds until sc.getExecutorStorageStatus.length returns
>>> the
>>> correct number of workers in my cluster (otherwise it returns 1, for the
>>> driver)...
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Getting-the-number-of-slaves-tp10604p10619.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>>
>
Re: Getting the number of slaves
Posted by Andrew Or <an...@databricks.com>.
Yes, both of these are derived from the same source, and this source
includes the driver. In other words, if you submit a job with 10 executors
you will get back 11 for both statuses.
2014-07-28 15:40 GMT-07:00 Sung Hwan Chung <co...@cs.stanford.edu>:
> Do getExecutorStorageStatus and getExecutorMemoryStatus both return the
> number of executors + the driver?
> E.g., if I submit a job with 10 executors, I get 11 for
> getExeuctorStorageStatus.length and getExecutorMemoryStatus.size
>
>
> On Thu, Jul 24, 2014 at 4:53 PM, Nicolas Mai <ni...@gmail.com>
> wrote:
>
>> Thanks, this is what I needed :) I should have searched more...
>>
>> Something I noticed though: after the SparkContext is initialized, I had
>> to
>> wait for a few seconds until sc.getExecutorStorageStatus.length returns
>> the
>> correct number of workers in my cluster (otherwise it returns 1, for the
>> driver)...
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Getting-the-number-of-slaves-tp10604p10619.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>
Re: Getting the number of slaves
Posted by Sung Hwan Chung <co...@cs.stanford.edu>.
Do getExecutorStorageStatus and getExecutorMemoryStatus both return the
number of executors + the driver?
E.g., if I submit a job with 10 executors, I get 11 for
getExeuctorStorageStatus.length and getExecutorMemoryStatus.size
On Thu, Jul 24, 2014 at 4:53 PM, Nicolas Mai <ni...@gmail.com> wrote:
> Thanks, this is what I needed :) I should have searched more...
>
> Something I noticed though: after the SparkContext is initialized, I had to
> wait for a few seconds until sc.getExecutorStorageStatus.length returns the
> correct number of workers in my cluster (otherwise it returns 1, for the
> driver)...
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Getting-the-number-of-slaves-tp10604p10619.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
Re: Getting the number of slaves
Posted by Nicolas Mai <ni...@gmail.com>.
Thanks, this is what I needed :) I should have searched more...
Something I noticed though: after the SparkContext is initialized, I had to
wait for a few seconds until sc.getExecutorStorageStatus.length returns the
correct number of workers in my cluster (otherwise it returns 1, for the
driver)...
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Getting-the-number-of-slaves-tp10604p10619.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: Getting the number of slaves
Posted by "Evan R. Sparks" <ev...@gmail.com>.
Try sc.getExecutorStorageStatus().length
SparkContext's getExecutorMemoryStatus or getExecutorStorageStatus will
give you back an object per executor - the StorageStatus objects are what
drives a lot of the Spark Web UI.
https://spark.apache.org/docs/1.0.1/api/scala/index.html#org.apache.spark.SparkContext
On Thu, Jul 24, 2014 at 11:16 AM, Nicolas Mai <ni...@gmail.com> wrote:
> Hi,
>
> Is there a way to get the number of slaves/workers during runtime?
>
> I searched online but didn't find anything :/ The application I'm working
> will run on different clusters corresponding to different deployment stages
> (beta -> prod). It would be great to get the number of slaves currently in
> use, in order set the level of parallelism and RDD partitions, based on
> that
> number.
>
> Thanks!
> Nicolas
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Getting-the-number-of-slaves-tp10604.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>