Suggestion for ec2 script

[Mingyu Kim <mk...@palantir.com>]

Hi all,

I found it confusing that "./spark-ec2 start² actually reinstalls the
cluster, which ends up wiping out all the configurations. How about renaming
³start² to ³install² and add a real light-weight ³start² for frequently
starting and stopping ec2 instances for mostly cost reasons? The
light-weight ³start² can simply bring up the instances and invoke
³bin/start-all.sh². Thoughts?

Mingyu

Re: Suggestion for ec2 script

[Mingyu Kim <mk...@palantir.com>]

I see. I didn¹t catch that. Thanks for the clarification!

Mingyu

From:  Matei Zaharia <ma...@gmail.com>
Reply-To:  "user@spark.incubator.apache.org"
<us...@spark.incubator.apache.org>
Date:  Friday, January 24, 2014 at 2:37 PM
To:  "user@spark.incubator.apache.org" <us...@spark.incubator.apache.org>
Subject:  Re: Suggestion for ec2 script

The issue is that ³start² is designed to restart the cluster after pausing
the instances with ³stop², and because some instances¹ DNS names changed, we
need to modify the master URL and the list of slaves in the various config
files. If you make other changes in these config files, they will indeed be
lost. One solution might be to create your own setup script to run on the
instances after ³start².

Matei 

On Jan 24, 2014, at 2:19 PM, Mingyu Kim <mk...@palantir.com> wrote:

> Hi all,
> 
> I found it confusing that "./spark-ec2 start² actually reinstalls the cluster,
> which ends up wiping out all the configurations. How about renaming ³start² to
> ³install² and add a real light-weight ³start² for frequently starting and
> stopping ec2 instances for mostly cost reasons? The light-weight ³start² can
> simply bring up the instances and invoke ³bin/start-all.sh². Thoughts?
> 
> Mingyu

Re: Suggestion for ec2 script

[Matei Zaharia <ma...@gmail.com>]

The issue is that “start” is designed to restart the cluster after pausing the instances with “stop”, and because some instances’ DNS names changed, we need to modify the master URL and the list of slaves in the various config files. If you make other changes in these config files, they will indeed be lost. One solution might be to create your own setup script to run on the instances after “start”.

Matei 

On Jan 24, 2014, at 2:19 PM, Mingyu Kim <mk...@palantir.com> wrote:

> Hi all,
> 
> I found it confusing that "./spark-ec2 start” actually reinstalls the cluster, which ends up wiping out all the configurations. How about renaming “start” to “install” and add a real light-weight “start” for frequently starting and stopping ec2 instances for mostly cost reasons? The light-weight “start” can simply bring up the instances and invoke “bin/start-all.sh”. Thoughts?
> 
> Mingyu