You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Amit Rana <am...@gmail.com> on 2016/06/20 12:23:04 UTC
Spark dev-setup
Hi all,
I am interested in figuring out how pyspark works at core/internal level.
And would like to understand the code flow as well.
For that I need to run a simple example in debug mode so that I can trace
the data flow for pyspark.
Can anyone please guide me on how do I set up my development environment
for the same in intellij IDEA in Windows 7.
Thanks,
Amit Rana
Re: Spark dev-setup
Posted by Nishadi Kirielle <nd...@gmail.com>.
Thanks a lot for the guidelines.
I could successfully configure and debug
On Wed, Aug 24, 2016 at 7:05 PM, Jacek Laskowski <ja...@japila.pl> wrote:
> On Wed, Aug 24, 2016 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > no reason; the key thing is : not in cluster mode, as there your work
> happens elsewhere
>
> Right! Anything but cluster mode should make it easy (that leaves us
> with local).
>
> Jacek
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: Spark dev-setup
Posted by Jacek Laskowski <ja...@japila.pl>.
On Wed, Aug 24, 2016 at 2:32 PM, Steve Loughran <st...@hortonworks.com> wrote:
> no reason; the key thing is : not in cluster mode, as there your work happens elsewhere
Right! Anything but cluster mode should make it easy (that leaves us
with local).
Jacek
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Spark dev-setup
Posted by Steve Loughran <st...@hortonworks.com>.
> On 24 Aug 2016, at 11:38, Jacek Laskowski <ja...@japila.pl> wrote:
>
> On Wed, Aug 24, 2016 at 11:13 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
>> I'd recommend
>
> ...which I mostly agree to with some exceptions :)
>
>> -stark spark standalone from there
>
> Why spark standalone since the OP asked about "learning how query
> execution flow occurs in Spark SQL"? How about spark-shell in local
> mode? Possibly explain(true) + conf/log4j.properties as the code might
> get tricky to get right at the very beginning.
>
no reason; the key thing is : not in cluster mode, as there your work happens elsewhere
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Spark dev-setup
Posted by Jacek Laskowski <ja...@japila.pl>.
On Wed, Aug 24, 2016 at 11:13 AM, Steve Loughran <st...@hortonworks.com> wrote:
> I'd recommend
...which I mostly agree to with some exceptions :)
> -stark spark standalone from there
Why spark standalone since the OP asked about "learning how query
execution flow occurs in Spark SQL"? How about spark-shell in local
mode? Possibly explain(true) + conf/log4j.properties as the code might
get tricky to get right at the very beginning.
#justcurious
Jacek
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: Spark dev-setup
Posted by Steve Loughran <st...@hortonworks.com>.
On 24 Aug 2016, at 07:10, Nishadi Kirielle <nd...@gmail.com>> wrote:
Hi,
I'm engaged in learning how query execution flow occurs in Spark SQL. In order to understand the query execution flow, I'm attempting to run an example in debug mode with intellij IDEA. It would be great if anyone can help me with debug configurations.
I'd recommend
-check out the version of spark you want to use
-set breakpoints wherever you want
-use dev/make-distribution.sh to build the release in dist/
-stark spark standalone from there
-attach to it in the IDE debugger
submit the work/type in queries in the REPL
this gives you the full launch with the complete classpath and env setup.
Otherwise: pull it out into a junit test and try to use IDEAs test runner to run it.
Re: Spark dev-setup
Posted by Nishadi Kirielle <nd...@gmail.com>.
Hi,
I'm engaged in learning how query execution flow occurs in Spark SQL. In
order to understand the query execution flow, I'm attempting to run an
example in debug mode with intellij IDEA. It would be great if anyone can
help me with debug configurations.
Thanks & Regards
Nishadi
On Tue, Jun 21, 2016 at 4:49 PM, Akhil Das <ak...@hacked.work> wrote:
> You can read this documentation to get started with the setup
> https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#
> UsefulDeveloperTools-IntelliJ
>
> There was a pyspark setup discussion on SO over here
> http://stackoverflow.com/questions/33478218/write-and-
> run-pyspark-in-intellij-idea
>
> On Mon, Jun 20, 2016 at 7:23 PM, Amit Rana <am...@gmail.com>
> wrote:
>
>> Hi all,
>>
>> I am interested in figuring out how pyspark works at core/internal
>> level. And would like to understand the code flow as well.
>> For that I need to run a simple example in debug mode so that I can
>> trace the data flow for pyspark.
>> Can anyone please guide me on how do I set up my development environment
>> for the same in intellij IDEA in Windows 7.
>>
>> Thanks,
>> Amit Rana
>>
>
>
>
> --
> Cheers!
>
>
Re: Spark dev-setup
Posted by Akhil Das <ak...@hacked.work>.
You can read this documentation to get started with the setup
https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ
There was a pyspark setup discussion on SO over here
http://stackoverflow.com/questions/33478218/write-and-run-pyspark-in-intellij-idea
On Mon, Jun 20, 2016 at 7:23 PM, Amit Rana <am...@gmail.com> wrote:
> Hi all,
>
> I am interested in figuring out how pyspark works at core/internal level.
> And would like to understand the code flow as well.
> For that I need to run a simple example in debug mode so that I can
> trace the data flow for pyspark.
> Can anyone please guide me on how do I set up my development environment
> for the same in intellij IDEA in Windows 7.
>
> Thanks,
> Amit Rana
>
--
Cheers!