How to make sure that function is executed on each active executor?

[Parag Chaudhari <pa...@gmail.com>]

Hi,

I am working on some use case where I want to perform some action on each
active executor of application once. How to run some function on each
active executor associated with current spark application?

num_executors = len(self._jsc.sc().statusTracker().getExecutorInfos()) - 1
if num_executors > 0:
    dummyRDD = self.parallelize(range(num_executors), num_executors)
    dummyRDD.foreachPartition(functionfoo)

Will it guarantee that function foo will be executed on each active
executor? Or will it miss few executors if there are more than 1 core per
executor?

Deeply appreciate help and time.


*Thanks,Parag Chaudhari*