You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by canan chen <cc...@gmail.com> on 2015/08/19 10:44:09 UTC
What's the best practice for developing new features for spark ?
I want to work on one jira, but it is not easy to do unit test, because it
involves different components especially UI. spark building is pretty slow,
I don't want to build it each time to test my code change. I am wondering
how other people do ? Is there any experience can share ? Thanks
Re: What's the best practice for developing new features for spark ?
Posted by Zoltán Zvara <zo...@gmail.com>.
I personally build with SBT and run Spark on YARN with IntelliJ. You need
to connect to remote JVMs with a remote debugger. You also need to do
similar, if you use Python, because it will launch a JVM on the driver
aswell.
On Wed, Aug 19, 2015 at 2:10 PM canan chen <cc...@gmail.com> wrote:
> Thanks Ted. I notice another thread about running spark programmatically
> (client mode for standalone and yarn). Would it be much easier to debug
> spark if is is possible ? Hasn't anyone thought about it ?
>
> On Wed, Aug 19, 2015 at 5:50 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> See this thread:
>>
>>
>> http://search-hadoop.com/m/q3RTtdZv0d1btRHl/Spark+build+module&subj=Building+Spark+Building+just+one+module+
>>
>>
>>
>> > On Aug 19, 2015, at 1:44 AM, canan chen <cc...@gmail.com> wrote:
>> >
>> > I want to work on one jira, but it is not easy to do unit test, because
>> it involves different components especially UI. spark building is pretty
>> slow, I don't want to build it each time to test my code change. I am
>> wondering how other people do ? Is there any experience can share ? Thanks
>> >
>> >
>>
>
>
Re: What's the best practice for developing new features for spark ?
Posted by Zoltán Zvara <zo...@gmail.com>.
I personally build with SBT and run Spark on YARN with IntelliJ. You need
to connect to remote JVMs with a remote debugger. You also need to do
similar, if you use Python, because it will launch a JVM on the driver
aswell.
On Wed, Aug 19, 2015 at 2:10 PM canan chen <cc...@gmail.com> wrote:
> Thanks Ted. I notice another thread about running spark programmatically
> (client mode for standalone and yarn). Would it be much easier to debug
> spark if is is possible ? Hasn't anyone thought about it ?
>
> On Wed, Aug 19, 2015 at 5:50 PM, Ted Yu <yu...@gmail.com> wrote:
>
>> See this thread:
>>
>>
>> http://search-hadoop.com/m/q3RTtdZv0d1btRHl/Spark+build+module&subj=Building+Spark+Building+just+one+module+
>>
>>
>>
>> > On Aug 19, 2015, at 1:44 AM, canan chen <cc...@gmail.com> wrote:
>> >
>> > I want to work on one jira, but it is not easy to do unit test, because
>> it involves different components especially UI. spark building is pretty
>> slow, I don't want to build it each time to test my code change. I am
>> wondering how other people do ? Is there any experience can share ? Thanks
>> >
>> >
>>
>
>
Re: What's the best practice for developing new features for spark ?
Posted by canan chen <cc...@gmail.com>.
Thanks Ted. I notice another thread about running spark programmatically
(client mode for standalone and yarn). Would it be much easier to debug
spark if is is possible ? Hasn't anyone thought about it ?
On Wed, Aug 19, 2015 at 5:50 PM, Ted Yu <yu...@gmail.com> wrote:
> See this thread:
>
>
> http://search-hadoop.com/m/q3RTtdZv0d1btRHl/Spark+build+module&subj=Building+Spark+Building+just+one+module+
>
>
>
> > On Aug 19, 2015, at 1:44 AM, canan chen <cc...@gmail.com> wrote:
> >
> > I want to work on one jira, but it is not easy to do unit test, because
> it involves different components especially UI. spark building is pretty
> slow, I don't want to build it each time to test my code change. I am
> wondering how other people do ? Is there any experience can share ? Thanks
> >
> >
>
Re: What's the best practice for developing new features for spark ?
Posted by canan chen <cc...@gmail.com>.
Thanks Ted. I notice another thread about running spark programmatically
(client mode for standalone and yarn). Would it be much easier to debug
spark if is is possible ? Hasn't anyone thought about it ?
On Wed, Aug 19, 2015 at 5:50 PM, Ted Yu <yu...@gmail.com> wrote:
> See this thread:
>
>
> http://search-hadoop.com/m/q3RTtdZv0d1btRHl/Spark+build+module&subj=Building+Spark+Building+just+one+module+
>
>
>
> > On Aug 19, 2015, at 1:44 AM, canan chen <cc...@gmail.com> wrote:
> >
> > I want to work on one jira, but it is not easy to do unit test, because
> it involves different components especially UI. spark building is pretty
> slow, I don't want to build it each time to test my code change. I am
> wondering how other people do ? Is there any experience can share ? Thanks
> >
> >
>
Re: What's the best practice for developing new features for spark ?
Posted by Ted Yu <yu...@gmail.com>.
See this thread:
http://search-hadoop.com/m/q3RTtdZv0d1btRHl/Spark+build+module&subj=Building+Spark+Building+just+one+module+
> On Aug 19, 2015, at 1:44 AM, canan chen <cc...@gmail.com> wrote:
>
> I want to work on one jira, but it is not easy to do unit test, because it involves different components especially UI. spark building is pretty slow, I don't want to build it each time to test my code change. I am wondering how other people do ? Is there any experience can share ? Thanks
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: What's the best practice for developing new features for spark ?
Posted by Ted Yu <yu...@gmail.com>.
See this thread:
http://search-hadoop.com/m/q3RTtdZv0d1btRHl/Spark+build+module&subj=Building+Spark+Building+just+one+module+
> On Aug 19, 2015, at 1:44 AM, canan chen <cc...@gmail.com> wrote:
>
> I want to work on one jira, but it is not easy to do unit test, because it involves different components especially UI. spark building is pretty slow, I don't want to build it each time to test my code change. I am wondering how other people do ? Is there any experience can share ? Thanks
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org