Spark 1.0.1 EC2 - Launching Applications

[Josh Happoldt <jo...@trueffect.com>]

Hi All,

I've used the spark-ec2 scripts to build a simple 1.0.1 Standalone cluster on EC2.  It appears that the spark-submit script is not bundled with a spark-ec2 install.  Given that:  What is the recommended way to execute spark jobs on a standalone EC2 cluster?  Spark-submit provides extremely useful features that are still useful for EC2 deployments.

We've used workarounds like modifying the spark-classpath and using run-example in the past to run simple one-time EC2 jobs.  The 'Running Applications' section of the EC2-Scripts documentation does not mention how to actually submit jobs to the cluster either.

Thanks!

Josh

Re: Spark 1.0.1 EC2 - Launching Applications

[Matei Zaharia <ma...@gmail.com>]

The script should be there, in the spark/bin directory. What command did you use to launch the cluster?

Matei

On Jul 14, 2014, at 1:12 PM, Josh Happoldt <jo...@trueffect.com> wrote:

> Hi All,
> 
> I've used the spark-ec2 scripts to build a simple 1.0.1 Standalone cluster on EC2.  It appears that the spark-submit script is not bundled with a spark-ec2 install.  Given that:  What is the recommended way to execute spark jobs on a standalone EC2 cluster?  Spark-submit provides extremely useful features that are still useful for EC2 deployments.
> 
> We've used workarounds like modifying the spark-classpath and using run-example in the past to run simple one-time EC2 jobs.  The 'Running Applications' section of the EC2-Scripts documentation does not mention how to actually submit jobs to the cluster either.
> 
> Thanks!
> 
> Josh
> 
> 
>