You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Eran Witkon <er...@gmail.com> on 2015/12/21 18:21:02 UTC
Using inteliJ for spark development
Any pointers how to use InteliJ for spark development?
Any way to use scala worksheet run like spark- shell?
Re: Using inteliJ for spark development
Posted by Dean Wampler <de...@gmail.com>.
The easiest way is to import an SBT project into IntelliJ that has the
Spark jars as dependencies. Then you can work in local mode (--master
local[*]). You could start with the build files in my "spark workshop",
https://github.com/deanwampler/spark-workshop and go from there.
I haven't done much with worksheets in IDEs and Spark. You would need to
create the SparkContext,
import org.apache.spark.SparkContext
val sc = new SparkContext("local[*]", "app")
Rerunning the worksheet will be tricky, though. You'll have to stop and
reconstruct the sc every time. Put sc.stop() at the end of the page.
dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Mon, Dec 21, 2015 at 11:21 AM, Eran Witkon <er...@gmail.com> wrote:
> Any pointers how to use InteliJ for spark development?
> Any way to use scala worksheet run like spark- shell?
>
Re: Using inteliJ for spark development
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
Both are similar, give both a go and choose the one you like.
On Dec 23, 2015 7:55 PM, "Eran Witkon" <er...@gmail.com> wrote:
> Thanks, so based on that article, should I use sbt or maven? Or either?
> Eran
> On Wed, 23 Dec 2015 at 13:05 Akhil Das <ak...@sigmoidanalytics.com> wrote:
>
>> You will have to point to your spark-assembly.jar since spark has a lot
>> of dependencies. You can read the answers discussed over here to have a
>> better understanding
>> http://stackoverflow.com/questions/3589562/why-maven-what-are-the-benefits
>>
>> Thanks
>> Best Regards
>>
>> On Wed, Dec 23, 2015 at 4:27 PM, Eran Witkon <er...@gmail.com>
>> wrote:
>>
>>> Thanks, all of these examples shows how to link to spark source and
>>> build it as part of my project. why should I do that? why not point
>>> directly to my spark.jar?
>>> Am I missing something?
>>> Eran
>>>
>>> On Wed, Dec 23, 2015 at 9:59 AM Akhil Das <ak...@sigmoidanalytics.com>
>>> wrote:
>>>
>>>> 1. Install sbt plugin on IntelliJ
>>>> 2. Create a new project/Import an sbt project like Dean suggested
>>>> 3. Happy Debugging.
>>>>
>>>> You can also refer to this article for more information
>>>> https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ
>>>>
>>>> Thanks
>>>> Best Regards
>>>>
>>>> On Mon, Dec 21, 2015 at 10:51 PM, Eran Witkon <er...@gmail.com>
>>>> wrote:
>>>>
>>>>> Any pointers how to use InteliJ for spark development?
>>>>> Any way to use scala worksheet run like spark- shell?
>>>>>
>>>>
>>>>
>>
Re: Using inteliJ for spark development
Posted by Eran Witkon <er...@gmail.com>.
Thanks, so based on that article, should I use sbt or maven? Or either?
Eran
On Wed, 23 Dec 2015 at 13:05 Akhil Das <ak...@sigmoidanalytics.com> wrote:
> You will have to point to your spark-assembly.jar since spark has a lot of
> dependencies. You can read the answers discussed over here to have a better
> understanding
> http://stackoverflow.com/questions/3589562/why-maven-what-are-the-benefits
>
> Thanks
> Best Regards
>
> On Wed, Dec 23, 2015 at 4:27 PM, Eran Witkon <er...@gmail.com> wrote:
>
>> Thanks, all of these examples shows how to link to spark source and build
>> it as part of my project. why should I do that? why not point directly to
>> my spark.jar?
>> Am I missing something?
>> Eran
>>
>> On Wed, Dec 23, 2015 at 9:59 AM Akhil Das <ak...@sigmoidanalytics.com>
>> wrote:
>>
>>> 1. Install sbt plugin on IntelliJ
>>> 2. Create a new project/Import an sbt project like Dean suggested
>>> 3. Happy Debugging.
>>>
>>> You can also refer to this article for more information
>>> https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ
>>>
>>> Thanks
>>> Best Regards
>>>
>>> On Mon, Dec 21, 2015 at 10:51 PM, Eran Witkon <er...@gmail.com>
>>> wrote:
>>>
>>>> Any pointers how to use InteliJ for spark development?
>>>> Any way to use scala worksheet run like spark- shell?
>>>>
>>>
>>>
>
Re: Using inteliJ for spark development
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
You will have to point to your spark-assembly.jar since spark has a lot of
dependencies. You can read the answers discussed over here to have a better
understanding
http://stackoverflow.com/questions/3589562/why-maven-what-are-the-benefits
Thanks
Best Regards
On Wed, Dec 23, 2015 at 4:27 PM, Eran Witkon <er...@gmail.com> wrote:
> Thanks, all of these examples shows how to link to spark source and build
> it as part of my project. why should I do that? why not point directly to
> my spark.jar?
> Am I missing something?
> Eran
>
> On Wed, Dec 23, 2015 at 9:59 AM Akhil Das <ak...@sigmoidanalytics.com>
> wrote:
>
>> 1. Install sbt plugin on IntelliJ
>> 2. Create a new project/Import an sbt project like Dean suggested
>> 3. Happy Debugging.
>>
>> You can also refer to this article for more information
>> https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ
>>
>> Thanks
>> Best Regards
>>
>> On Mon, Dec 21, 2015 at 10:51 PM, Eran Witkon <er...@gmail.com>
>> wrote:
>>
>>> Any pointers how to use InteliJ for spark development?
>>> Any way to use scala worksheet run like spark- shell?
>>>
>>
>>
Re: Using inteliJ for spark development
Posted by Eran Witkon <er...@gmail.com>.
Thanks, all of these examples shows how to link to spark source and build
it as part of my project. why should I do that? why not point directly to
my spark.jar?
Am I missing something?
Eran
On Wed, Dec 23, 2015 at 9:59 AM Akhil Das <ak...@sigmoidanalytics.com>
wrote:
> 1. Install sbt plugin on IntelliJ
> 2. Create a new project/Import an sbt project like Dean suggested
> 3. Happy Debugging.
>
> You can also refer to this article for more information
> https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ
>
> Thanks
> Best Regards
>
> On Mon, Dec 21, 2015 at 10:51 PM, Eran Witkon <er...@gmail.com>
> wrote:
>
>> Any pointers how to use InteliJ for spark development?
>> Any way to use scala worksheet run like spark- shell?
>>
>
>
Re: Using inteliJ for spark development
Posted by Akhil Das <ak...@sigmoidanalytics.com>.
1. Install sbt plugin on IntelliJ
2. Create a new project/Import an sbt project like Dean suggested
3. Happy Debugging.
You can also refer to this article for more information
https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ
Thanks
Best Regards
On Mon, Dec 21, 2015 at 10:51 PM, Eran Witkon <er...@gmail.com> wrote:
> Any pointers how to use InteliJ for spark development?
> Any way to use scala worksheet run like spark- shell?
>