You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@spark.apache.org by Ajay Nair <pr...@gmail.com> on 2014/05/03 17:42:53 UTC

Apache Spark running out of the spark shell

Hi,

I have written a code that works just about fine in the spark shell on EC2.
The ec2 script helped me configure my master and worker nodes. Now I want to
run the scala-spark code out side the interactive shell. How do I go about
doing it.

I was referring to the instructions mentioned here:
https://spark.apache.org/docs/0.9.1/quick-start.html

But this is confusing because it mentions about a simple project jar file
which I am not sure how to generate. I only have the file that runs directly
on my spark shell. Any easy intruction to get this quickly running as a job?

Thanks
AJ



--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-running-out-of-the-spark-shell-tp6459.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: Apache Spark running out of the spark shell

Posted by Nicolas Garneau <ng...@ngarneau.com>.

Hey AJ,

I have tried to run on a cluster yet, only on local mode.
I'll try to get something running on a cluster soon and keep you posted.

Nicolas Garneau

> On May 4, 2014, at 6:23 PM, Ajay Nair <pr...@gmail.com> wrote:
> 
> Now I got it to work .. well almost. However I needed to copy the project/
> folder to the spark-standalone folder as the package build was failing
> because it could not find buil properties. After the copy the build was
> successful. However when I run it I get errors but it still gives me the
> output.
> 
> [error] 14/05/04 21:58:19 INFO spark.SparkContext: Job finished: count at
> SimpleApp.scala:11, took 0.040651597 s
> [error] 14/05/04 21:58:19 INFO scheduler.TaskSetManager: Finished TID 3 in
> 17 ms on localhost (progress: 2/2)
> [info] Lines with a: 3, Lines with b: 2
> [error] 14/05/04 21:58:19 INFO scheduler.TaskSchedulerImpl: Removed TaskSet
> 1.0, whose tasks have all completed, from pool 
> [success] Total time: 5 s, completed May 4, 2014 9:58:20 PM
> 
> 
> You can see the [info] that contains the output. All the lines i get mention
> [errors], any reason why ?
> 
> I have configured my ec2 machines master and slave nodes and this code I
> think tries to run in the local mode.
> 
> 
> 
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-running-out-of-the-spark-shell-tp6459p6478.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>

Re: Apache Spark running out of the spark shell

Posted by Ajay Nair <pr...@gmail.com>.

Now I got it to work .. well almost. However I needed to copy the project/
folder to the spark-standalone folder as the package build was failing
because it could not find buil properties. After the copy the build was
successful. However when I run it I get errors but it still gives me the
output.

[error] 14/05/04 21:58:19 INFO spark.SparkContext: Job finished: count at
SimpleApp.scala:11, took 0.040651597 s
[error] 14/05/04 21:58:19 INFO scheduler.TaskSetManager: Finished TID 3 in
17 ms on localhost (progress: 2/2)
[info] Lines with a: 3, Lines with b: 2
[error] 14/05/04 21:58:19 INFO scheduler.TaskSchedulerImpl: Removed TaskSet
1.0, whose tasks have all completed, from pool 
[success] Total time: 5 s, completed May 4, 2014 9:58:20 PM


You can see the [info] that contains the output. All the lines i get mention
[errors], any reason why ?

I have configured my ec2 machines master and slave nodes and this code I
think tries to run in the local mode.



--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-running-out-of-the-spark-shell-tp6459p6478.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: Apache Spark running out of the spark shell

Posted by Nicolas Garneau <ng...@ngarneau.com>.

Hey AJ,

If you plan to launch your job on a cluster, consider using the spark-submit command.
Running this in the spark's home directory gives you a help on how to use this:

$ ./bin/spark-submit

I haven't tried it yet but considering this post, it will be the preferred way to launch jobs:
http://apache-spark-user-list.1001560.n3.nabble.com/Running-a-spark-submit-compatible-app-in-spark-shell-td4905.html

Cheers

Le 2014-05-04 à 13:35, Ajay Nair <pr...@gmail.com> a écrit :

> Thank you. I am trying this now
> 
> 
> 
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-running-out-of-the-spark-shell-tp6459p6472.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
> 

Nicolas Garneau
ngarneau@ngarneau.com

Re: Apache Spark running out of the spark shell

Posted by Ajay Nair <pr...@gmail.com>.

Thank you. I am trying this now



--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-running-out-of-the-spark-shell-tp6459p6472.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: Apache Spark running out of the spark shell

Posted by Nicolas Garneau <ng...@ngarneau.com>.

Hey AJ,

As I can see your path when running sbt is:

> $root/spark/spark-examples: sbt/sbt package

You should be within the app's folder that contains the simple.sbt, which is spark-standalone/;

> $root/spark/spark-examples/spark-standalone: sbt/sbt package
> $root/spark/spark-examples/spark-standalone: sbt/sbt run


Don't forget to move the sbt folder within your app's directory.

That being said, I think you can install sbt globally on your system so you'll be able to run the sbt command everywhere on your PC.
It'll be useful when creating multiple apps.

For example, the way I'm building it from A to Z:
$ git clone https://github.com/ngarneau/spark-standalone.git
$ cd spark-standalone
-- change the path of spark's home dir
$ sbt package (assuming sbt is installed globally)
$ sbt run (assuming sbt is installed globally)

Hope this helps!

Le 2014-05-03 à 13:38, Ajay Nair <pr...@gmail.com> a écrit :

> Quick question, where should I place your folder. Inside the spark directory.
> My Spark directory is in /root/spark
> So currently I tried pulling your github code in /root/spark/spark-examples
> and modified my home spark directory in the scala code.
> I copied the sbt folder within the spark-examples folder. But when I try
> running this command
> 
> $root/spark/spark-examples: sbt/sbt package
> 
> awk: cmd. line:1: fatal: cannot open file `./project/build.properties' for
> reading (No such file or directory)
> Launching sbt from sbt/sbt-launch-.jar
> Error: Invalid or corrupt jarfile sbt/sbt-launch-.jar
> 
> 
> However the sbt package runs fines (Expectedly) when i run it from
> /root/spark folder.
> 
> Anything I am doing wrong here?
> 
> 
> 
> 
> 
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-running-out-of-the-spark-shell-tp6459p6465.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
> 

Nicolas Garneau
ngarneau@ngarneau.com

Re: Apache Spark running out of the spark shell

Posted by Ajay Nair <pr...@gmail.com>.

Quick question, where should I place your folder. Inside the spark directory.
My Spark directory is in /root/spark
So currently I tried pulling your github code in /root/spark/spark-examples
and modified my home spark directory in the scala code.
I copied the sbt folder within the spark-examples folder. But when I try
running this command

$root/spark/spark-examples: sbt/sbt package

awk: cmd. line:1: fatal: cannot open file `./project/build.properties' for
reading (No such file or directory)
Launching sbt from sbt/sbt-launch-.jar
Error: Invalid or corrupt jarfile sbt/sbt-launch-.jar


However the sbt package runs fines (Expectedly) when i run it from
/root/spark folder.

Anything I am doing wrong here?





--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-running-out-of-the-spark-shell-tp6459p6465.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: Apache Spark running out of the spark shell

Posted by Nicolas Garneau <ng...@ngarneau.com>.

Sorry, the link went wrong. I meant here:
https://github.com/ngarneau/spark-standalone

Le 2014-05-03 à 13:23, Nicolas Garneau <ng...@ngarneau.com> a écrit :

> Hey AJ,
> 
> I created a little sample app using the spark's quick start.
> Have a look here.
> Assuming you used scala, using sbt is good for running your application in standalone mode.
> The configuration file which is "simple.sbt" in my repo, holds all the dependencies needed to build your app.
> 
> Hope this helps!
> 
> Le 2014-05-03 à 11:42, Ajay Nair <pr...@gmail.com> a écrit :
> 
>> Hi,
>> 
>> I have written a code that works just about fine in the spark shell on EC2.
>> The ec2 script helped me configure my master and worker nodes. Now I want to
>> run the scala-spark code out side the interactive shell. How do I go about
>> doing it.
>> 
>> I was referring to the instructions mentioned here:
>> https://spark.apache.org/docs/0.9.1/quick-start.html
>> 
>> But this is confusing because it mentions about a simple project jar file
>> which I am not sure how to generate. I only have the file that runs directly
>> on my spark shell. Any easy intruction to get this quickly running as a job?
>> 
>> Thanks
>> AJ
>> 
>> 
>> 
>> --
>> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-running-out-of-the-spark-shell-tp6459.html
>> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
>> 
> 
> Nicolas Garneau
> ngarneau@ngarneau.com
> 

Nicolas Garneau
418.569.3097
ngarneau@ngarneau.com

Re: Apache Spark running out of the spark shell

Posted by Ajay Nair <pr...@gmail.com>.

Thank you for the reply. Have you posted a link from where I follow the steps
?



--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-running-out-of-the-spark-shell-tp6459p6462.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: Apache Spark running out of the spark shell

Posted by Nicolas Garneau <ng...@ngarneau.com>.

Hey AJ,

I created a little sample app using the spark's quick start.
Have a look here.
Assuming you used scala, using sbt is good for running your application in standalone mode.
The configuration file which is "simple.sbt" in my repo, holds all the dependencies needed to build your app.

Hope this helps!

Le 2014-05-03 à 11:42, Ajay Nair <pr...@gmail.com> a écrit :

> Hi,
> 
> I have written a code that works just about fine in the spark shell on EC2.
> The ec2 script helped me configure my master and worker nodes. Now I want to
> run the scala-spark code out side the interactive shell. How do I go about
> doing it.
> 
> I was referring to the instructions mentioned here:
> https://spark.apache.org/docs/0.9.1/quick-start.html
> 
> But this is confusing because it mentions about a simple project jar file
> which I am not sure how to generate. I only have the file that runs directly
> on my spark shell. Any easy intruction to get this quickly running as a job?
> 
> Thanks
> AJ
> 
> 
> 
> --
> View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-running-out-of-the-spark-shell-tp6459.html
> Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
> 

Nicolas Garneau
ngarneau@ngarneau.com

Re: Apache Spark running out of the spark shell

Posted by Ajay Nair <pr...@gmail.com>.

Thank you. Let me try this quickly !



--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-running-out-of-the-spark-shell-tp6459p6463.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

Re: Apache Spark running out of the spark shell

Posted by Sandy Ryza <sa...@cloudera.com>.

Hi AJ,

You might find this helpful -
http://blog.cloudera.com/blog/2014/04/how-to-run-a-simple-apache-spark-app-in-cdh-5/

-Sandy


On Sat, May 3, 2014 at 8:42 AM, Ajay Nair <pr...@gmail.com> wrote:

> Hi,
>
> I have written a code that works just about fine in the spark shell on EC2.
> The ec2 script helped me configure my master and worker nodes. Now I want
> to
> run the scala-spark code out side the interactive shell. How do I go about
> doing it.
>
> I was referring to the instructions mentioned here:
> https://spark.apache.org/docs/0.9.1/quick-start.html
>
> But this is confusing because it mentions about a simple project jar file
> which I am not sure how to generate. I only have the file that runs
> directly
> on my spark shell. Any easy intruction to get this quickly running as a
> job?
>
> Thanks
> AJ
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Apache-Spark-running-out-of-the-spark-shell-tp6459.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>