AMP Camp 3 cluster setup

[Shivaram Venkataraman <sh...@gmail.com>]

Hi

As a part of AMP Camp 3 to be held on Aug 29, 30th we will be hosting
hands-on exercises that will allow users to try out Spark, Shark,
Mesos, MLBase, BlinkDB etc.

Here are some notes on how to setup a cluster in case you want to
follow along with our live stream  NOTE: The exercises have not been
released yet and will be made available during the camp. You can
register to be notified at http://ampcamp.berkeley.edu/3

To launch a cluster, the steps involved are:

1. Clone the training-scripts git repo:

git clone https://github.com/amplab/training-scripts.git

2. Launch the cluster using spark-ec2

cd training-scripts
./spark-ec2 -i <key_file> -k <name_of_key_pair> --copy launch amplab-training

Where <name_of_key_pair> is the name of your EC2 key pair (that you
gave it when you created it), <key_file> is the private key file for
your key pair.

The scripts will launch 6 m1.xlarge machines (1 master + 5 slaves) and
each machine will have two EBS volumes attached to it.

This command may take a 30-40 minutes or longer and should produce a
bunch of output as it first spins up the nodes for your cluster, sets
up BDAS on them, and performs a large file copy of the datasets we’ll
use in the camp from an EBS volume to your instance of HDFS.

Let me know if you have any questions !

Thanks
Shivaram

Re: AMP Camp 3 cluster setup

[Matei Zaharia <ma...@gmail.com>]

By the way, an important note: Make sure you *shut down* your cluster after using it. Otherwise, Amazon will keep charging you money for it! I've seen some people get caught by that in the past.

For others following this list, it's probably fine to start the cluster tomorrow morning (Pacific time) instead of today.

Matei

On Aug 28, 2013, at 3:21 PM, seanm <re...@gmail.com> wrote:

> Shivaram-
> 
> I just spun up a cluster and poked around a bit, it worked great!
> 
> Thank you!
> 
> Sean
> 
> 
> 
> On Wednesday, August 28, 2013 3:08:04 PM UTC-7, Shivaram wrote:
> Hi 
> 
> As a part of AMP Camp 3 to be held on Aug 29, 30th we will be hosting 
> hands-on exercises that will allow users to try out Spark, Shark, 
> Mesos, MLBase, BlinkDB etc. 
> 
> Here are some notes on how to setup a cluster in case you want to 
> follow along with our live stream  NOTE: The exercises have not been 
> released yet and will be made available during the camp. You can 
> register to be notified at http://ampcamp.berkeley.edu/3 
> 
> To launch a cluster, the steps involved are: 
> 
> 1. Clone the training-scripts git repo: 
> 
> git clone https://github.com/amplab/training-scripts.git 
> 
> 2. Launch the cluster using spark-ec2 
> 
> cd training-scripts 
> ./spark-ec2 -i <key_file> -k <name_of_key_pair> --copy launch amplab-training 
> 
> Where <name_of_key_pair> is the name of your EC2 key pair (that you 
> gave it when you created it), <key_file> is the private key file for 
> your key pair. 
> 
> The scripts will launch 6 m1.xlarge machines (1 master + 5 slaves) and 
> each machine will have two EBS volumes attached to it. 
> 
> This command may take a 30-40 minutes or longer and should produce a 
> bunch of output as it first spins up the nodes for your cluster, sets 
> up BDAS on them, and performs a large file copy of the datasets we’ll 
> use in the camp from an EBS volume to your instance of HDFS. 
> 
> Let me know if you have any questions ! 
> 
> Thanks 
> Shivaram 
> 
> -- 
> You received this message because you are subscribed to the Google Groups "Spark Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spark-users+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.