slower worker node in the cluster

[haopu <hw...@qilinsoft.com>]

In a standalone cluster, is there way to specify the stage to be running on a
faster worker?

That stage is reading HDFS file and then doing some filter operations.  The
tasks are assigned to the slower worker also, but the slower worker delays
to launch because it's running some tasks from other stages.

So I think it may be better to assign stage to a worker. Any suggestions?

And will the cluster on Yarn help?



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/slower-worker-node-in-the-cluster-tp9125.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: slower worker node in the cluster

[Chris Fregly <ch...@fregly.com>]

perhaps creating Fair Scheduler Pools might help?  there's no way to pin
certain nodes to a pool, but you can specify minShares (cpu's).  not sure
if that would help, but worth looking in to.


On Tue, Jul 8, 2014 at 7:37 PM, haopu <hw...@qilinsoft.com> wrote:

> In a standalone cluster, is there way to specify the stage to be running
> on a
> faster worker?
>
> That stage is reading HDFS file and then doing some filter operations.  The
> tasks are assigned to the slower worker also, but the slower worker delays
> to launch because it's running some tasks from other stages.
>
> So I think it may be better to assign stage to a worker. Any suggestions?
>
> And will the cluster on Yarn help?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/slower-worker-node-in-the-cluster-tp9125.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>