You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Mukesh G <mu...@gmail.com> on 2014/04/07 18:20:36 UTC
Contributing to Spark
Hi,
How I contribute to Spark and it's associated projects?
Appreciate the help...
Thanks
Mukesh
Re: Contributing to Spark
Posted by Reynold Xin <rx...@databricks.com>.
Usually you can just run Spark in local mode on a single machine for most
dev/testing.
If you want to simulate a cluster locally using multiple Spark worker
processes, you can use the undocumented local cluster mode, e.g.
local-cluster[2,1,512]
this launches two worker processes, each with one core and 512m of ram.
On Wed, Apr 9, 2014 at 3:00 PM, Sujeet Varakhedi <sv...@gmail.com> wrote:
> Another starter question which probably should have asked before is what is
> the most efficient way to iterate quickly on dev/test. I am currently using
> a local cluster (via vagrant and shared folders) and also spark-shell.
>
> Sujeet
>
>
> On Tue, Apr 8, 2014 at 9:50 AM, Michael Ernest <mfernest@cloudera.com
> >wrote:
>
> > Ha ha! nice try, sheepherder! ;-)
> >
> >
> > On Tue, Apr 8, 2014 at 12:37 PM, Matei Zaharia <matei.zaharia@gmail.com
> > >wrote:
> >
> > > Shh, maybe I really wanted people to fix that one issue.
> > >
> > > On Apr 8, 2014, at 9:34 AM, Aaron Davidson <il...@gmail.com> wrote:
> > >
> > > > Matei's link seems to point to a specific starter project as part of
> > the
> > > > starter list, but here is the list itself:
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)
> > > >
> > > >
> > > > On Mon, Apr 7, 2014 at 10:22 PM, Matei Zaharia <
> > matei.zaharia@gmail.com
> > > >wrote:
> > > >
> > > >> I'd suggest looking for the issues labeled "Starter" on JIRA. You
> can
> > > find
> > > >> them here:
> > > >>
> > >
> >
> https://issues.apache.org/jira/browse/SPARK-1438?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)
> > > >>
> > > >> Matei
> > > >>
> > > >> On Apr 7, 2014, at 9:45 PM, Mukesh G <mu...@gmail.com> wrote:
> > > >>
> > > >>> Hi Sujeet,
> > > >>>
> > > >>> Thanks. I went thru the website and looks great. Is there a list
> of
> > > >>> items that I can choose from, for contribution?
> > > >>>
> > > >>> Thanks
> > > >>>
> > > >>> Mukesh
> > > >>>
> > > >>>
> > > >>> On Mon, Apr 7, 2014 at 10:14 PM, Sujeet Varakhedi
> > > >>> <sv...@gopivotal.com>wrote:
> > > >>>
> > > >>>> This is a good place to start:
> > > >>>>
> > >
> https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
> > > >>>>
> > > >>>> Sujeet
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Apr 7, 2014 at 9:20 AM, Mukesh G <mu...@gmail.com>
> wrote:
> > > >>>>
> > > >>>>> Hi,
> > > >>>>>
> > > >>>>> How I contribute to Spark and it's associated projects?
> > > >>>>>
> > > >>>>> Appreciate the help...
> > > >>>>>
> > > >>>>> Thanks
> > > >>>>>
> > > >>>>> Mukesh
> > > >>>>>
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
> >
> > --
> > Michael Ernest
> > Sr. Solutions Consultant
> > West Coast
> >
>
Re: Contributing to Spark
Posted by Sujeet Varakhedi <sv...@gmail.com>.
Another starter question which probably should have asked before is what is
the most efficient way to iterate quickly on dev/test. I am currently using
a local cluster (via vagrant and shared folders) and also spark-shell.
Sujeet
On Tue, Apr 8, 2014 at 9:50 AM, Michael Ernest <mf...@cloudera.com>wrote:
> Ha ha! nice try, sheepherder! ;-)
>
>
> On Tue, Apr 8, 2014 at 12:37 PM, Matei Zaharia <matei.zaharia@gmail.com
> >wrote:
>
> > Shh, maybe I really wanted people to fix that one issue.
> >
> > On Apr 8, 2014, at 9:34 AM, Aaron Davidson <il...@gmail.com> wrote:
> >
> > > Matei's link seems to point to a specific starter project as part of
> the
> > > starter list, but here is the list itself:
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)
> > >
> > >
> > > On Mon, Apr 7, 2014 at 10:22 PM, Matei Zaharia <
> matei.zaharia@gmail.com
> > >wrote:
> > >
> > >> I'd suggest looking for the issues labeled "Starter" on JIRA. You can
> > find
> > >> them here:
> > >>
> >
> https://issues.apache.org/jira/browse/SPARK-1438?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)
> > >>
> > >> Matei
> > >>
> > >> On Apr 7, 2014, at 9:45 PM, Mukesh G <mu...@gmail.com> wrote:
> > >>
> > >>> Hi Sujeet,
> > >>>
> > >>> Thanks. I went thru the website and looks great. Is there a list of
> > >>> items that I can choose from, for contribution?
> > >>>
> > >>> Thanks
> > >>>
> > >>> Mukesh
> > >>>
> > >>>
> > >>> On Mon, Apr 7, 2014 at 10:14 PM, Sujeet Varakhedi
> > >>> <sv...@gopivotal.com>wrote:
> > >>>
> > >>>> This is a good place to start:
> > >>>>
> > https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
> > >>>>
> > >>>> Sujeet
> > >>>>
> > >>>>
> > >>>> On Mon, Apr 7, 2014 at 9:20 AM, Mukesh G <mu...@gmail.com> wrote:
> > >>>>
> > >>>>> Hi,
> > >>>>>
> > >>>>> How I contribute to Spark and it's associated projects?
> > >>>>>
> > >>>>> Appreciate the help...
> > >>>>>
> > >>>>> Thanks
> > >>>>>
> > >>>>> Mukesh
> > >>>>>
> > >>>>
> > >>
> > >>
> >
> >
>
>
> --
> Michael Ernest
> Sr. Solutions Consultant
> West Coast
>
Re: Contributing to Spark
Posted by Michael Ernest <mf...@cloudera.com>.
Ha ha! nice try, sheepherder! ;-)
On Tue, Apr 8, 2014 at 12:37 PM, Matei Zaharia <ma...@gmail.com>wrote:
> Shh, maybe I really wanted people to fix that one issue.
>
> On Apr 8, 2014, at 9:34 AM, Aaron Davidson <il...@gmail.com> wrote:
>
> > Matei's link seems to point to a specific starter project as part of the
> > starter list, but here is the list itself:
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)
> >
> >
> > On Mon, Apr 7, 2014 at 10:22 PM, Matei Zaharia <matei.zaharia@gmail.com
> >wrote:
> >
> >> I'd suggest looking for the issues labeled "Starter" on JIRA. You can
> find
> >> them here:
> >>
> https://issues.apache.org/jira/browse/SPARK-1438?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)
> >>
> >> Matei
> >>
> >> On Apr 7, 2014, at 9:45 PM, Mukesh G <mu...@gmail.com> wrote:
> >>
> >>> Hi Sujeet,
> >>>
> >>> Thanks. I went thru the website and looks great. Is there a list of
> >>> items that I can choose from, for contribution?
> >>>
> >>> Thanks
> >>>
> >>> Mukesh
> >>>
> >>>
> >>> On Mon, Apr 7, 2014 at 10:14 PM, Sujeet Varakhedi
> >>> <sv...@gopivotal.com>wrote:
> >>>
> >>>> This is a good place to start:
> >>>>
> https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
> >>>>
> >>>> Sujeet
> >>>>
> >>>>
> >>>> On Mon, Apr 7, 2014 at 9:20 AM, Mukesh G <mu...@gmail.com> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> How I contribute to Spark and it's associated projects?
> >>>>>
> >>>>> Appreciate the help...
> >>>>>
> >>>>> Thanks
> >>>>>
> >>>>> Mukesh
> >>>>>
> >>>>
> >>
> >>
>
>
--
Michael Ernest
Sr. Solutions Consultant
West Coast
Re: Contributing to Spark
Posted by Matei Zaharia <ma...@gmail.com>.
Shh, maybe I really wanted people to fix that one issue.
On Apr 8, 2014, at 9:34 AM, Aaron Davidson <il...@gmail.com> wrote:
> Matei's link seems to point to a specific starter project as part of the
> starter list, but here is the list itself:
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)
>
>
> On Mon, Apr 7, 2014 at 10:22 PM, Matei Zaharia <ma...@gmail.com>wrote:
>
>> I'd suggest looking for the issues labeled "Starter" on JIRA. You can find
>> them here:
>> https://issues.apache.org/jira/browse/SPARK-1438?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)
>>
>> Matei
>>
>> On Apr 7, 2014, at 9:45 PM, Mukesh G <mu...@gmail.com> wrote:
>>
>>> Hi Sujeet,
>>>
>>> Thanks. I went thru the website and looks great. Is there a list of
>>> items that I can choose from, for contribution?
>>>
>>> Thanks
>>>
>>> Mukesh
>>>
>>>
>>> On Mon, Apr 7, 2014 at 10:14 PM, Sujeet Varakhedi
>>> <sv...@gopivotal.com>wrote:
>>>
>>>> This is a good place to start:
>>>> https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
>>>>
>>>> Sujeet
>>>>
>>>>
>>>> On Mon, Apr 7, 2014 at 9:20 AM, Mukesh G <mu...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> How I contribute to Spark and it's associated projects?
>>>>>
>>>>> Appreciate the help...
>>>>>
>>>>> Thanks
>>>>>
>>>>> Mukesh
>>>>>
>>>>
>>
>>
Re: Contributing to Spark
Posted by Aaron Davidson <il...@gmail.com>.
Matei's link seems to point to a specific starter project as part of the
starter list, but here is the list itself:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)
On Mon, Apr 7, 2014 at 10:22 PM, Matei Zaharia <ma...@gmail.com>wrote:
> I'd suggest looking for the issues labeled "Starter" on JIRA. You can find
> them here:
> https://issues.apache.org/jira/browse/SPARK-1438?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)
>
> Matei
>
> On Apr 7, 2014, at 9:45 PM, Mukesh G <mu...@gmail.com> wrote:
>
> > Hi Sujeet,
> >
> > Thanks. I went thru the website and looks great. Is there a list of
> > items that I can choose from, for contribution?
> >
> > Thanks
> >
> > Mukesh
> >
> >
> > On Mon, Apr 7, 2014 at 10:14 PM, Sujeet Varakhedi
> > <sv...@gopivotal.com>wrote:
> >
> >> This is a good place to start:
> >> https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
> >>
> >> Sujeet
> >>
> >>
> >> On Mon, Apr 7, 2014 at 9:20 AM, Mukesh G <mu...@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> How I contribute to Spark and it's associated projects?
> >>>
> >>> Appreciate the help...
> >>>
> >>> Thanks
> >>>
> >>> Mukesh
> >>>
> >>
>
>
Re: Contributing to Spark
Posted by Matei Zaharia <ma...@gmail.com>.
I’d suggest looking for the issues labeled “Starter” on JIRA. You can find them here: https://issues.apache.org/jira/browse/SPARK-1438?jql=project%20%3D%20SPARK%20AND%20labels%20%3D%20Starter%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)
Matei
On Apr 7, 2014, at 9:45 PM, Mukesh G <mu...@gmail.com> wrote:
> Hi Sujeet,
>
> Thanks. I went thru the website and looks great. Is there a list of
> items that I can choose from, for contribution?
>
> Thanks
>
> Mukesh
>
>
> On Mon, Apr 7, 2014 at 10:14 PM, Sujeet Varakhedi
> <sv...@gopivotal.com>wrote:
>
>> This is a good place to start:
>> https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
>>
>> Sujeet
>>
>>
>> On Mon, Apr 7, 2014 at 9:20 AM, Mukesh G <mu...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> How I contribute to Spark and it's associated projects?
>>>
>>> Appreciate the help...
>>>
>>> Thanks
>>>
>>> Mukesh
>>>
>>
Re: Contributing to Spark
Posted by Mukesh G <mu...@gmail.com>.
Hi Sujeet,
Thanks. I went thru the website and looks great. Is there a list of
items that I can choose from, for contribution?
Thanks
Mukesh
On Mon, Apr 7, 2014 at 10:14 PM, Sujeet Varakhedi
<sv...@gopivotal.com>wrote:
> This is a good place to start:
> https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
>
> Sujeet
>
>
> On Mon, Apr 7, 2014 at 9:20 AM, Mukesh G <mu...@gmail.com> wrote:
>
> > Hi,
> >
> > How I contribute to Spark and it's associated projects?
> >
> > Appreciate the help...
> >
> > Thanks
> >
> > Mukesh
> >
>
Re: Contributing to Spark
Posted by Sujeet Varakhedi <sv...@gopivotal.com>.
This is a good place to start:
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
Sujeet
On Mon, Apr 7, 2014 at 9:20 AM, Mukesh G <mu...@gmail.com> wrote:
> Hi,
>
> How I contribute to Spark and it's associated projects?
>
> Appreciate the help...
>
> Thanks
>
> Mukesh
>