You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Robert James <sr...@gmail.com> on 2014/07/06 16:10:40 UTC

Addind and subtracting workers on Spark EC2 cluster

If I've created a Spark EC2 cluster, how can I add or take away workers?

Also: If I use EC2 spot instances, what happens when Amazon removes
them? Will my computation be saved in any way, or will I need to
restart from scratch?

Finally: The spark-ec2 scripts seem to use Hadoop 1.  How can I
configure them to use Hadoop 2?

Re: Addind and subtracting workers on Spark EC2 cluster

Posted by Nicholas Chammas <ni...@gmail.com>.

On Sun, Jul 6, 2014 at 10:10 AM, Robert James <sr...@gmail.com>
wrote:

If I've created a Spark EC2 cluster, how can I add or take away workers?

There is a manual process by which this is possible, but I’m not sure of
the procedure. There is also SPARK-2008
<https://issues.apache.org/jira/browse/SPARK-2008>, which tracks adding the
ability to add/remove slaves to an existing cluster to spark-ec2.

Nick