Reg: Add/Remove slave nodes spark-ec2

[Sirisha Devineni <Si...@persistent.co.in>]

Hi All,

I have created a spark cluster on EC2 using spark-ec2 script. Whenever more data is there to be processed I would like to add new slaves to existing cluster and would like to remove slave node when the data to be processed is low.

It seems currently spark-ec2 doesn't have option to add/remove slaves to existing cluster. Could you please suggest how can we achieve this?

One liner problem statement: How to add/remove slaves to/from Spark cluster on EC2 using spark-ec2?

Suggestions on when to add/remove slaves is much appreciated.

Thanks & Regards,
Sirisha Devineni

DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

Re: Reg: Add/Remove slave nodes spark-ec2

[Mayur Rustagi <ma...@gmail.com>]

You'll have to restart the cluster.. create copy of your existing slave..
add it to slave files in master & restart the cluster

Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>



On Tue, Jun 3, 2014 at 4:30 PM, Sirisha Devineni <
Sirisha_Devineni@persistent.co.in> wrote:

>  Hi All,
>
>
>
> I have created a spark cluster on EC2 using spark-ec2 script. Whenever
> more data is there to be processed I would like to add new slaves to
> existing cluster and would like to remove slave node when the data to be
> processed is low.
>
>
>
> It seems currently spark-ec2 doesn’t have option to add/remove slaves to
> existing cluster. Could you please suggest how can we achieve this?
>
>
>
> *One liner problem statement*: How to add/remove slaves to/from Spark
> cluster on EC2 using spark-ec2?
>
>
>
> Suggestions on when to add/remove slaves is much appreciated.
>
>
>
> Thanks & Regards,
>
> Sirisha Devineni
>
> DISCLAIMER ========== This e-mail may contain privileged and confidential
> information which is the property of Persistent Systems Ltd. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Persistent Systems Ltd. does not accept any liability for
> virus infected mails.
>