You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Mingyu Kim <mk...@palantir.com> on 2015/03/13 08:04:32 UTC

toLocalIterator creates as many jobs as # of partitions, and it ends up spamming Spark UI

Hi all,

RDD.toLocalIterator() creates as many jobs as # of partitions and it spams Spark UI especially when the method is used on an RDD with hundreds or thousands of partitions.

Does anyone have a way to work around this issue? What do people think about introducing a SparkContext local property (analogous to “spark.scheduler.pool” set as a thread-local property) that determines if the job info should be shown on the Spark UI?

Thanks,
Mingyu