You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Xi Shen <da...@gmail.com> on 2015/03/06 02:19:00 UTC

Spark code development practice

Hi,

I am new to Spark. I see every spark program has a main() function. I
wonder if I can run the spark program directly, without using spark-submit.
I think it will be easier for early development and debug.


Thanks,
David

Re: Spark code development practice

Posted by Koen Vantomme <ko...@gmail.com>.

use the spark-shell command and the shell will open 
type :paste abd then paste your code, after control-d

open spark-shell:
sparks/bin
./spark-shell

Verstuurd vanaf mijn iPhone

> Op 6-mrt.-2015 om 02:28 heeft "fightfate@163.com" <fi...@163.com> het volgende geschreven:
> 
> Hi,
> 
> You can first establish a scala ide to develop and debug your spark program, lets say, intellij idea or eclipse.
> 
> Thanks,
> Sun.
> 
> fightfate@163.com
>  
> From: Xi Shen
> Date: 2015-03-06 09:19
> To: user@spark.apache.org
> Subject: Spark code development practice
> Hi,
> 
> I am new to Spark. I see every spark program has a main() function. I wonder if I can run the spark program directly, without using spark-submit. I think it will be easier for early development and debug.
> 
> 
> Thanks,
> David
>

Re: Spark code development practice

Posted by "fightfate@163.com" <fi...@163.com>.

Hi,

You can first establish a scala ide to develop and debug your spark program, lets say, intellij idea or eclipse.

Thanks,
Sun.



fightfate@163.com
 
From: Xi Shen
Date: 2015-03-06 09:19
To: user@spark.apache.org
Subject: Spark code development practice
Hi,

I am new to Spark. I see every spark program has a main() function. I wonder if I can run the spark program directly, without using spark-submit. I think it will be easier for early development and debug.


Thanks,
David

Re: Spark code development practice

Posted by Stephen Boesch <ja...@gmail.com>.

Hi Xi,
  Yes,
  You can do the following:

val sc = new SparkContext("local[2]", "mptest")

// or .. val sc = new SparkContext("spark://master:7070", "mptest")

val fileDataRdd = sc.textFile("/path/to/dir")

val fileLines = fileDataRdd.take(100)


The key here - i.e. the answer to your specific question - is the creation
of the SparkContext : that is one of the things spark-submit is doing for
you

2015-03-05 17:19 GMT-08:00 Xi Shen <da...@gmail.com>:

> Hi,
>
> I am new to Spark. I see every spark program has a main() function. I
> wonder if I can run the spark program directly, without using spark-submit.
> I think it will be easier for early development and debug.
>
>
> Thanks,
> David
>
>