You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2015/09/22 12:41:04 UTC
[jira] [Resolved] (SPARK-10730) spark receivers not evenly
distributed among all cluster slaves
[ https://issues.apache.org/jira/browse/SPARK-10730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-10730.
-------------------------------
Resolution: Not A Problem
[~patrizio.munzi] please first read https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
I don't think this qualifies as a problem, not with this much information. Spark schedules according to many heuristics, and it's not necessarily ideal to spread work over machines. In your case you're working with a local collection that's been sent to the cluster; it may be optimal to execute locally.
> spark receivers not evenly distributed among all cluster slaves
> ---------------------------------------------------------------
>
> Key: SPARK-10730
> URL: https://issues.apache.org/jira/browse/SPARK-10730
> Project: Spark
> Issue Type: Bug
> Reporter: Patrizio Munzi
>
> I'm running a yarn cluster of 8 executors and the spark receivers are not evenly distributed among all cluster slaves.
> Sometimes they all get executed in the same machine!!
> I also tried to reinforce the concept you added after the last same bug submission, that is running a dummy job to ensure all slaves have joined the cluster.
> Here's my code:
> log.info("Starting a dummy job to ensure all slaves have registered.");
> final int dummyListLength = 1000;
> final List<Integer> dummyList = new ArrayList<Integer>();
> for(int i = 1; i <= dummyListLength; i++) {
> dummyList.add(i);
> }
> jssc.sparkContext()
> .parallelize(dummyList, dummyListLength)
> .mapToPair(new DummyPairFunction())
> .reduceByKey(new DummyReduceByKeyFunction())
> .collect();
> log.info("Dummy job completed.");
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org