You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Adam Berry <am...@yahoo-inc.com> on 2012/10/17 22:01:45 UTC
[DISCUSS] [PROPOSAL] Hadoop Development Tools
Hello,
Below is a proposal for a new incubator project. This idea came out of and had strong support on the general hadoop list, see the thread at http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser.
We are looking for feedback, and to see who else would be interested in contributing to this effort as a committer, as well as an additional mentor.
Cheers,
Adam Berry
Hadoop Development Tools Proposal
I’d like to propose the Hadoop Development Tools, a set of extensions to the Eclipse IDE to support developing against Apache Hadoop technologies.
= HDT (Hadoop Development Tools) =
== Abstract ==
Tools to support developing applications that use Apache Hadoop from within Eclipse.
== Proposal ==
Hadoop Development Tools are a set of extensions to Eclipse providing support for creating, launching and debugging distributed applications, as well as interacting with HDFS filesystems. This work will build on the existing Map Reduce Tools present in the Apache Hadoop project.
== Background ==
Map Reduce Tools have existed as part of contrib for Apache Hadoop. Unfortunately they are source tied to a single version of Hadoop, and development has stalled, with little movement past the Hadoop 0.20 line.
== Rationale ==
Support for newer versions of Hadoop from within Eclipse is regularly raised on the Hadoop mailing lists, so there is a clear need to drive these tools forward. Development tools generally are worked on separate from the target tools/platform, separating the tools out will allow for supporting multiple versions, so a developer could work with a heterogeneous environment.
== Initial Goals ==
* Give the tools project a home of its own.
* Port current MapReduce tools feature set to all current release lines of Hadoop in a single Eclipse install.
* Documentation and tutorials for all features.
* Publish Eclipse update site, and join Eclipse marketplace listing.
* Establish release cycle that combines support for Hadoop and Eclipse release cycles.
* Look to build support for YARN, MRUnit and possibly other Hadoop-related projects.
== Current Status ==
The source for the current MapReduceTools lives in the contrib section of the Hadoop source. In its current implementation it is tied to the version of Hadoop against which it is compiled. The layout and API that it was developed with means that it can only be used with the 0.20 or 1.0 Hadoop releases, the new layout and YARN api introduced with the 0.23 and 2.0 lines are not supported.
=== Meritocracy ===
Several people and companies have already expressed an interest in contributing to this project, and we hope to attract additional interest during the proposal discussion. We plan to invest and support a meritocracy that attracts, invites, and supports newcomers to build a vibrant and diverse community.
=== Community ===
The target community is developers who are working developing Map/Reduce applications against Hadoop. Given the success of Hadoop the target group is likely to be quite large. Separation from the Hadoop community would make it easier to support multiple versions of hadoop, as well as merging the release cycles of Hadoop and Eclipse to provide predictable iteration and improvement in the toolset.
=== Core Developers ===
The initial list of developers includes people experienced with developing against the Eclipse platform.
* Adam Berry (amberry at yahoo-inc dot com)
* Jeffrey Zemerick (jeffrrey at mtnfog dot com)
=== Alignment ===
Hadoop Development Tools aligns with both Hadoop and Eclipse. Hadoop as the platform for the development target, and Eclipse as the IDE platform used as the base for the tools.
== Known Risks ==
=== Orphaned Products ===
=== Inexperience with Open Source ===
Adam Berry has experience of the Eclipse open source community, and has been building familiarity with the Apache processes through patches to the existing source.
=== Reliance on Salaried Developers ===
Hadoop Development Tools will be developed with a mix of salaried and volunteer time.
=== Relationships with Other Apache Projects ===
Hadoop Development Tools is closely related to Apache Hadoop.
=== An Excessive Fascination with the Apache Brand ===
Given the success of Hadoop and associated projects, Apache is the natural place for the Hadoop Development Tools. Chris Mattman suggested the Apache Incubator as appropriate on the Hadoop general mailing list following the success that MRUnit had taking the path from Hadoop contrib to an Apache top level project.
== Documentation ==
Documentation for the current tools can be found at http://wiki.apache.org/hadoop/EclipsePlugIn
== Initial Source ==
http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/eclipse-plugin/
== Source and Intellectual Property Submission Plan ==
The source, and any suggested initial patches, are already hosted either in Apache’s Subversion or JIRA.
== External Dependencies ==
Eclipse Platform
Eclipse Java Development Tools
== Cryptography ==
Hadoop Development Tools likely does not fall into this area.
== Required Resources ==
=== Mailing lists ===
* hdt-dev
* hdt-commits
* hdt-user
=== Subversion Directory ===
* https://svn.apache.org/repos/asf/incubator/hdt
=== Issue Tracking ===
* JIRA Hadoop Development Tools (HDT)
=== Other Resources ===
* Jenkins/Hudson for builds and test running.
== Initial Committers ==
* Adam Berry (amberry at yahoo-inc dot com)
* Jeffrey Zemerick (jeffrrey at mtnfog dot com)
== Affiliations ==
* Adam Berry - Yahoo!
* Jeffrey Zemerick - Mountain Fog
== Sponsors ==
=== Champion ===
Chris Douglas
=== Nominated Mentors ===
Chris Douglas
Chris Mattman
=== Sponsoring Entity ===
Incubator PMC
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Simone Gianni <si...@apache.org>.
Hi Adam,
I've added myself as a committer.
Simone
2012/10/24 Adam Berry <am...@yahoo-inc.com>
> Hi all,
>
> I've updated the proposal wiki to reflect that Roman has now volunteered
> as champion, and Evert has joined as a committer.
>
> Cheers,
> Adam
>
>
> On Oct 19, 2012, at 1:59 PM, Adam Berry wrote:
>
> >
> > On Oct 19, 2012, at 11:32 AM, Adam Berry wrote:
> >
> >>
> >> Thanks for the reminder Simone.
> >>
> >> Its now up at
> http://wiki.apache.org/incubator/HadoopDevelopmentToolsProposal, and is
> of course linked off the project proposals page.
> >>
> >> So anyone who wishes to join in please go ahead and add yourselves
> there, and dropping a note here would also be helpful. Remember to be a
> mentor my understanding is you need to be a member of the Incubator PMC.
> >
> > Roman, I see you have just been announced as a member of the Incubator
> PMC, congratulations!
> >
> > If you are willing to be a mentor that would be great.
> >
> > Adam
> >>
> >> Cheers,
> >> Adam
> >>
> >> On Oct 19, 2012, at 9:15 AM, Simone Gianni wrote:
> >>
> >> Adam,
> >> I think you should write your proposal in the incubator wiki, if you
> >> haven't already, so that interested people can start adding themselves
> >> there.
> >>
> >> Simone
> >>
> >> 2012/10/19 Roman Shaposhnik <rv...@apache.org>>
> >>
> >> On Wed, Oct 17, 2012 at 1:01 PM, Adam Berry <amberry@yahoo-inc.com
> <ma...@yahoo-inc.com>> wrote:
> >> Hello,
> >>
> >> Below is a proposal for a new incubator project. This idea came out of
> >> and had strong support on
> >> the general hadoop list, see the thread at
> >>
> http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser
> >> .
> >>
> >> We are looking for feedback, and to see who else would be interested in
> >> contributing to this effort as a committer, as well as an additional
> mentor.
> >>
> >> In general I'd be very happy to see this project as part of the Apache
> >> Family and would
> >> love to help with mentor-ship duties and also possible integration
> >> with Apache Bigtop
> >> and collaboration with MRUnit.
> >>
> >> Future mentors -- please consider me volunteering for help on this one.
> >>
> >> Thanks,
> >> Roman.
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> <ma...@incubator.apache.org>
> >> For additional commands, e-mail: general-help@incubator.apache.org
> <ma...@incubator.apache.org>
> >>
> >>
> >>
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Adam Berry <am...@yahoo-inc.com>.
Hi all,
I've updated the proposal wiki to reflect that Roman has now volunteered as champion, and Evert has joined as a committer.
Cheers,
Adam
On Oct 19, 2012, at 1:59 PM, Adam Berry wrote:
>
> On Oct 19, 2012, at 11:32 AM, Adam Berry wrote:
>
>>
>> Thanks for the reminder Simone.
>>
>> Its now up at http://wiki.apache.org/incubator/HadoopDevelopmentToolsProposal, and is of course linked off the project proposals page.
>>
>> So anyone who wishes to join in please go ahead and add yourselves there, and dropping a note here would also be helpful. Remember to be a mentor my understanding is you need to be a member of the Incubator PMC.
>
> Roman, I see you have just been announced as a member of the Incubator PMC, congratulations!
>
> If you are willing to be a mentor that would be great.
>
> Adam
>>
>> Cheers,
>> Adam
>>
>> On Oct 19, 2012, at 9:15 AM, Simone Gianni wrote:
>>
>> Adam,
>> I think you should write your proposal in the incubator wiki, if you
>> haven't already, so that interested people can start adding themselves
>> there.
>>
>> Simone
>>
>> 2012/10/19 Roman Shaposhnik <rv...@apache.org>>
>>
>> On Wed, Oct 17, 2012 at 1:01 PM, Adam Berry <am...@yahoo-inc.com>> wrote:
>> Hello,
>>
>> Below is a proposal for a new incubator project. This idea came out of
>> and had strong support on
>> the general hadoop list, see the thread at
>> http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser
>> .
>>
>> We are looking for feedback, and to see who else would be interested in
>> contributing to this effort as a committer, as well as an additional mentor.
>>
>> In general I'd be very happy to see this project as part of the Apache
>> Family and would
>> love to help with mentor-ship duties and also possible integration
>> with Apache Bigtop
>> and collaboration with MRUnit.
>>
>> Future mentors -- please consider me volunteering for help on this one.
>>
>> Thanks,
>> Roman.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org<ma...@incubator.apache.org>
>> For additional commands, e-mail: general-help@incubator.apache.org<ma...@incubator.apache.org>
>>
>>
>>
>
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Adam Berry <am...@yahoo-inc.com>.
Hi all,
I've updated the proposal wiki to reflect that Roman has now volunteered as champion, and Evert has joined as a committer.
Cheers,
Adam
On Oct 19, 2012, at 1:59 PM, Adam Berry wrote:
>
> On Oct 19, 2012, at 11:32 AM, Adam Berry wrote:
>
>>
>> Thanks for the reminder Simone.
>>
>> Its now up at http://wiki.apache.org/incubator/HadoopDevelopmentToolsProposal, and is of course linked off the project proposals page.
>>
>> So anyone who wishes to join in please go ahead and add yourselves there, and dropping a note here would also be helpful. Remember to be a mentor my understanding is you need to be a member of the Incubator PMC.
>
> Roman, I see you have just been announced as a member of the Incubator PMC, congratulations!
>
> If you are willing to be a mentor that would be great.
>
> Adam
>>
>> Cheers,
>> Adam
>>
>> On Oct 19, 2012, at 9:15 AM, Simone Gianni wrote:
>>
>> Adam,
>> I think you should write your proposal in the incubator wiki, if you
>> haven't already, so that interested people can start adding themselves
>> there.
>>
>> Simone
>>
>> 2012/10/19 Roman Shaposhnik <rv...@apache.org>>
>>
>> On Wed, Oct 17, 2012 at 1:01 PM, Adam Berry <am...@yahoo-inc.com>> wrote:
>> Hello,
>>
>> Below is a proposal for a new incubator project. This idea came out of
>> and had strong support on
>> the general hadoop list, see the thread at
>> http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser
>> .
>>
>> We are looking for feedback, and to see who else would be interested in
>> contributing to this effort as a committer, as well as an additional mentor.
>>
>> In general I'd be very happy to see this project as part of the Apache
>> Family and would
>> love to help with mentor-ship duties and also possible integration
>> with Apache Bigtop
>> and collaboration with MRUnit.
>>
>> Future mentors -- please consider me volunteering for help on this one.
>>
>> Thanks,
>> Roman.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org<ma...@incubator.apache.org>
>> For additional commands, e-mail: general-help@incubator.apache.org<ma...@incubator.apache.org>
>>
>>
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Adam Berry <am...@yahoo-inc.com>.
On Oct 19, 2012, at 11:32 AM, Adam Berry wrote:
>
> Thanks for the reminder Simone.
>
> Its now up at http://wiki.apache.org/incubator/HadoopDevelopmentToolsProposal, and is of course linked off the project proposals page.
>
> So anyone who wishes to join in please go ahead and add yourselves there, and dropping a note here would also be helpful. Remember to be a mentor my understanding is you need to be a member of the Incubator PMC.
Roman, I see you have just been announced as a member of the Incubator PMC, congratulations!
If you are willing to be a mentor that would be great.
Adam
>
> Cheers,
> Adam
>
> On Oct 19, 2012, at 9:15 AM, Simone Gianni wrote:
>
> Adam,
> I think you should write your proposal in the incubator wiki, if you
> haven't already, so that interested people can start adding themselves
> there.
>
> Simone
>
> 2012/10/19 Roman Shaposhnik <rv...@apache.org>>
>
> On Wed, Oct 17, 2012 at 1:01 PM, Adam Berry <am...@yahoo-inc.com>> wrote:
> Hello,
>
> Below is a proposal for a new incubator project. This idea came out of
> and had strong support on
> the general hadoop list, see the thread at
> http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser
> .
>
> We are looking for feedback, and to see who else would be interested in
> contributing to this effort as a committer, as well as an additional mentor.
>
> In general I'd be very happy to see this project as part of the Apache
> Family and would
> love to help with mentor-ship duties and also possible integration
> with Apache Bigtop
> and collaboration with MRUnit.
>
> Future mentors -- please consider me volunteering for help on this one.
>
> Thanks,
> Roman.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org<ma...@incubator.apache.org>
> For additional commands, e-mail: general-help@incubator.apache.org<ma...@incubator.apache.org>
>
>
>
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Adam Berry <am...@yahoo-inc.com>.
On Oct 19, 2012, at 11:32 AM, Adam Berry wrote:
>
> Thanks for the reminder Simone.
>
> Its now up at http://wiki.apache.org/incubator/HadoopDevelopmentToolsProposal, and is of course linked off the project proposals page.
>
> So anyone who wishes to join in please go ahead and add yourselves there, and dropping a note here would also be helpful. Remember to be a mentor my understanding is you need to be a member of the Incubator PMC.
Roman, I see you have just been announced as a member of the Incubator PMC, congratulations!
If you are willing to be a mentor that would be great.
Adam
>
> Cheers,
> Adam
>
> On Oct 19, 2012, at 9:15 AM, Simone Gianni wrote:
>
> Adam,
> I think you should write your proposal in the incubator wiki, if you
> haven't already, so that interested people can start adding themselves
> there.
>
> Simone
>
> 2012/10/19 Roman Shaposhnik <rv...@apache.org>>
>
> On Wed, Oct 17, 2012 at 1:01 PM, Adam Berry <am...@yahoo-inc.com>> wrote:
> Hello,
>
> Below is a proposal for a new incubator project. This idea came out of
> and had strong support on
> the general hadoop list, see the thread at
> http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser
> .
>
> We are looking for feedback, and to see who else would be interested in
> contributing to this effort as a committer, as well as an additional mentor.
>
> In general I'd be very happy to see this project as part of the Apache
> Family and would
> love to help with mentor-ship duties and also possible integration
> with Apache Bigtop
> and collaboration with MRUnit.
>
> Future mentors -- please consider me volunteering for help on this one.
>
> Thanks,
> Roman.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org<ma...@incubator.apache.org>
> For additional commands, e-mail: general-help@incubator.apache.org<ma...@incubator.apache.org>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Adam Berry <am...@yahoo-inc.com>.
Thanks for the reminder Simone.
Its now up at http://wiki.apache.org/incubator/HadoopDevelopmentToolsProposal, and is of course linked off the project proposals page.
So anyone who wishes to join in please go ahead and add yourselves there, and dropping a note here would also be helpful. Remember to be a mentor my understanding is you need to be a member of the Incubator PMC.
Cheers,
Adam
On Oct 19, 2012, at 9:15 AM, Simone Gianni wrote:
Adam,
I think you should write your proposal in the incubator wiki, if you
haven't already, so that interested people can start adding themselves
there.
Simone
2012/10/19 Roman Shaposhnik <rv...@apache.org>>
On Wed, Oct 17, 2012 at 1:01 PM, Adam Berry <am...@yahoo-inc.com>> wrote:
Hello,
Below is a proposal for a new incubator project. This idea came out of
and had strong support on
the general hadoop list, see the thread at
http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser
.
We are looking for feedback, and to see who else would be interested in
contributing to this effort as a committer, as well as an additional mentor.
In general I'd be very happy to see this project as part of the Apache
Family and would
love to help with mentor-ship duties and also possible integration
with Apache Bigtop
and collaboration with MRUnit.
Future mentors -- please consider me volunteering for help on this one.
Thanks,
Roman.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org<ma...@incubator.apache.org>
For additional commands, e-mail: general-help@incubator.apache.org<ma...@incubator.apache.org>
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Adam Berry <am...@yahoo-inc.com>.
Thanks for the reminder Simone.
Its now up at http://wiki.apache.org/incubator/HadoopDevelopmentToolsProposal, and is of course linked off the project proposals page.
So anyone who wishes to join in please go ahead and add yourselves there, and dropping a note here would also be helpful. Remember to be a mentor my understanding is you need to be a member of the Incubator PMC.
Cheers,
Adam
On Oct 19, 2012, at 9:15 AM, Simone Gianni wrote:
Adam,
I think you should write your proposal in the incubator wiki, if you
haven't already, so that interested people can start adding themselves
there.
Simone
2012/10/19 Roman Shaposhnik <rv...@apache.org>>
On Wed, Oct 17, 2012 at 1:01 PM, Adam Berry <am...@yahoo-inc.com>> wrote:
Hello,
Below is a proposal for a new incubator project. This idea came out of
and had strong support on
the general hadoop list, see the thread at
http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser
.
We are looking for feedback, and to see who else would be interested in
contributing to this effort as a committer, as well as an additional mentor.
In general I'd be very happy to see this project as part of the Apache
Family and would
love to help with mentor-ship duties and also possible integration
with Apache Bigtop
and collaboration with MRUnit.
Future mentors -- please consider me volunteering for help on this one.
Thanks,
Roman.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org<ma...@incubator.apache.org>
For additional commands, e-mail: general-help@incubator.apache.org<ma...@incubator.apache.org>
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Simone Gianni <si...@apache.org>.
Adam,
I think you should write your proposal in the incubator wiki, if you
haven't already, so that interested people can start adding themselves
there.
Simone
2012/10/19 Roman Shaposhnik <rv...@apache.org>
> On Wed, Oct 17, 2012 at 1:01 PM, Adam Berry <am...@yahoo-inc.com> wrote:
> > Hello,
> >
> > Below is a proposal for a new incubator project. This idea came out of
> and had strong support on
> > the general hadoop list, see the thread at
> http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser
> .
> >
> > We are looking for feedback, and to see who else would be interested in
> contributing to this effort as a committer, as well as an additional mentor.
>
> In general I'd be very happy to see this project as part of the Apache
> Family and would
> love to help with mentor-ship duties and also possible integration
> with Apache Bigtop
> and collaboration with MRUnit.
>
> Future mentors -- please consider me volunteering for help on this one.
>
> Thanks,
> Roman.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
>
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Roman Shaposhnik <rv...@apache.org>.
On Wed, Oct 17, 2012 at 1:01 PM, Adam Berry <am...@yahoo-inc.com> wrote:
> Hello,
>
> Below is a proposal for a new incubator project. This idea came out of and had strong support on
> the general hadoop list, see the thread at http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser.
>
> We are looking for feedback, and to see who else would be interested in contributing to this effort as a committer, as well as an additional mentor.
In general I'd be very happy to see this project as part of the Apache
Family and would
love to help with mentor-ship duties and also possible integration
with Apache Bigtop
and collaboration with MRUnit.
Future mentors -- please consider me volunteering for help on this one.
Thanks,
Roman.
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Roman Shaposhnik <rv...@apache.org>.
On Wed, Oct 17, 2012 at 1:01 PM, Adam Berry <am...@yahoo-inc.com> wrote:
> Hello,
>
> Below is a proposal for a new incubator project. This idea came out of and had strong support on
> the general hadoop list, see the thread at http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser.
>
> We are looking for feedback, and to see who else would be interested in contributing to this effort as a committer, as well as an additional mentor.
In general I'd be very happy to see this project as part of the Apache
Family and would
love to help with mentor-ship duties and also possible integration
with Apache Bigtop
and collaboration with MRUnit.
Future mentors -- please consider me volunteering for help on this one.
Thanks,
Roman.
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Joe Stein <cr...@gmail.com>.
Does none of this include implementations of streaming or did I miss
something?
On Thu, Oct 18, 2012 at 4:16 AM, Steve Loughran <st...@gmail.com>wrote:
> it may be good to have a proposal that is broader than just eclipse; gives
> you flexibility in future even if the initial target is the eclipse plugin.
> Certainly "Hadoop Development Tools" is pretty broad
>
> On 17 October 2012 21:01, Adam Berry <am...@yahoo-inc.com> wrote:
>
> > Hello,
> >
> > Below is a proposal for a new incubator project. This idea came out of
> and
> > had strong support on the general hadoop list, see the thread at
> >
> http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser
> > .
> >
> > We are looking for feedback, and to see who else would be interested in
> > contributing to this effort as a committer, as well as an additional
> mentor.
> >
> > Cheers,
> > Adam Berry
> >
> >
> > Hadoop Development Tools Proposal
> >
> > I’d like to propose the Hadoop Development Tools, a set of extensions to
> > the Eclipse IDE to support developing against Apache Hadoop technologies.
> >
> > = HDT (Hadoop Development Tools) =
> >
> > == Abstract ==
> > Tools to support developing applications that use Apache Hadoop from
> > within Eclipse.
> >
> > == Proposal ==
> > Hadoop Development Tools are a set of extensions to Eclipse providing
> > support for creating, launching and debugging distributed applications,
> as
> > well as interacting with HDFS filesystems. This work will build on the
> > existing Map Reduce Tools present in the Apache Hadoop project.
> >
> > == Background ==
> > Map Reduce Tools have existed as part of contrib for Apache Hadoop.
> > Unfortunately they are source tied to a single version of Hadoop, and
> > development has stalled, with little movement past the Hadoop 0.20 line.
> >
> > == Rationale ==
> > Support for newer versions of Hadoop from within Eclipse is regularly
> > raised on the Hadoop mailing lists, so there is a clear need to drive
> these
> > tools forward. Development tools generally are worked on separate from
> the
> > target tools/platform, separating the tools out will allow for supporting
> > multiple versions, so a developer could work with a heterogeneous
> > environment.
> >
> > == Initial Goals ==
> > * Give the tools project a home of its own.
> > * Port current MapReduce tools feature set to all current release lines
> > of Hadoop in a single Eclipse install.
> > * Documentation and tutorials for all features.
> > * Publish Eclipse update site, and join Eclipse marketplace listing.
> > * Establish release cycle that combines support for Hadoop and Eclipse
> > release cycles.
> > * Look to build support for YARN, MRUnit and possibly other
> > Hadoop-related projects.
> >
> > == Current Status ==
> > The source for the current MapReduceTools lives in the contrib section of
> > the Hadoop source. In its current implementation it is tied to the
> version
> > of Hadoop against which it is compiled. The layout and API that it was
> > developed with means that it can only be used with the 0.20 or 1.0 Hadoop
> > releases, the new layout and YARN api introduced with the 0.23 and 2.0
> > lines are not supported.
> >
> >
> > === Meritocracy ===
> > Several people and companies have already expressed an interest in
> > contributing to this project, and we hope to attract additional interest
> > during the proposal discussion. We plan to invest and support a
> meritocracy
> > that attracts, invites, and supports newcomers to build a vibrant and
> > diverse community.
> >
> > === Community ===
> > The target community is developers who are working developing Map/Reduce
> > applications against Hadoop. Given the success of Hadoop the target group
> > is likely to be quite large. Separation from the Hadoop community would
> > make it easier to support multiple versions of hadoop, as well as merging
> > the release cycles of Hadoop and Eclipse to provide predictable iteration
> > and improvement in the toolset.
> >
> > === Core Developers ===
> > The initial list of developers includes people experienced with
> developing
> > against the Eclipse platform.
> > * Adam Berry (amberry at yahoo-inc dot com)
> > * Jeffrey Zemerick (jeffrrey at mtnfog dot com)
> >
> > === Alignment ===
> > Hadoop Development Tools aligns with both Hadoop and Eclipse. Hadoop as
> > the platform for the development target, and Eclipse as the IDE platform
> > used as the base for the tools.
> >
> > == Known Risks ==
> >
> > === Orphaned Products ===
> >
> > === Inexperience with Open Source ===
> > Adam Berry has experience of the Eclipse open source community, and has
> > been building familiarity with the Apache processes through patches to
> the
> > existing source.
> >
> > === Reliance on Salaried Developers ===
> > Hadoop Development Tools will be developed with a mix of salaried and
> > volunteer time.
> >
> > === Relationships with Other Apache Projects ===
> > Hadoop Development Tools is closely related to Apache Hadoop.
> >
> > === An Excessive Fascination with the Apache Brand ===
> > Given the success of Hadoop and associated projects, Apache is the
> > natural place for the Hadoop Development Tools. Chris Mattman suggested
> the
> > Apache Incubator as appropriate on the Hadoop general mailing list
> > following the success that MRUnit had taking the path from Hadoop contrib
> > to an Apache top level project.
> >
> > == Documentation ==
> > Documentation for the current tools can be found at
> > http://wiki.apache.org/hadoop/EclipsePlugIn
> >
> > == Initial Source ==
> >
> >
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/eclipse-plugin/
> >
> > == Source and Intellectual Property Submission Plan ==
> > The source, and any suggested initial patches, are already hosted either
> > in Apache’s Subversion or JIRA.
> >
> > == External Dependencies ==
> > Eclipse Platform
> > Eclipse Java Development Tools
> >
> > == Cryptography ==
> > Hadoop Development Tools likely does not fall into this area.
> >
> > == Required Resources ==
> > === Mailing lists ===
> > * hdt-dev
> > * hdt-commits
> > * hdt-user
> >
> > === Subversion Directory ===
> > * https://svn.apache.org/repos/asf/incubator/hdt
> >
> > === Issue Tracking ===
> > * JIRA Hadoop Development Tools (HDT)
> >
> > === Other Resources ===
> > * Jenkins/Hudson for builds and test running.
> >
> > == Initial Committers ==
> > * Adam Berry (amberry at yahoo-inc dot com)
> > * Jeffrey Zemerick (jeffrrey at mtnfog dot com)
> >
> > == Affiliations ==
> > * Adam Berry - Yahoo!
> > * Jeffrey Zemerick - Mountain Fog
> >
> > == Sponsors ==
> > === Champion ===
> > Chris Douglas
> >
> > === Nominated Mentors ===
> > Chris Douglas
> > Chris Mattman
> >
> > === Sponsoring Entity ===
> > Incubator PMC
> >
>
--
/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
*/
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
+1 to giving it a chance at Apache. I'm not sure what Extras buys this effort and I for one
am not really interested in helping it there but am sincerely willing to try while it's at the ASF.
Cheers,
Chris
On Oct 18, 2012, at 12:57 PM, Chris Douglas wrote:
> On Thu, Oct 18, 2012 at 11:56 AM, Jakob Homan <jg...@gmail.com> wrote:
>> This seems like it might be more at home at apache-extras.
>
> That may ultimately be its fate, but building a community around these
> tools is worthy of an attempt. This also shares many goals with
> MRUnit. Collaboration or even merging with that community could also
> make sense. The code is dead in its current home; letting someone run
> with it here to see where it leads seems reasonable.
>
>> On Thu, Oct 18, 2012 at 9:28 AM, Adam Berry <am...@yahoo-inc.com> wrote:
>>> I will update the proposal later on to include that, and additionally some idea of how we could support streaming from within the IDE.
>
> Seems out of scope. The goal of the project, as I gathered from the
> proposal, is to produce tools and plugins that complement development
> of (and on) Hadoop. There's no need to specify which features the
> community intends to build at this stage. -C
>
>>> Great comments though folks, keep 'em coming!
>>>
>>> Cheers,
>>> Adam
>>>
>>> On Oct 18, 2012, at 10:53 AM, Simone Gianni wrote:
>>>
>>>> I think it's a good idea, I used the old stuff at the time and it is
>>>> something that is missing.
>>>>
>>>> 2012/10/18 Steve Loughran <st...@gmail.com>
>>>>
>>>>> it may be good to have a proposal that is broader than just eclipse; gives
>>>>> you flexibility in future even if the initial target is the eclipse plugin.
>>>>> Certainly "Hadoop Development Tools" is pretty broad
>>>>>
>>>>
>>>> Steve you are right, but I'm afraid that given how the Eclipse framework
>>>> requires tight coupling with Eclipse stuff (SWT, workbench system etc..)
>>>> it's not easy to develop something that can easily be reused on different
>>>> platforms.
>>>>
>>>> However, it could be possible to distribute a stand alone RCP application
>>>> for those not using Eclipse, and eventually try to implement a "generic
>>>> Hadoop-UI bridge", as far as possible agnostic of specific Eclipse classes
>>>> and interfaces, that could eventually be reused for other IDEs.
>>>>
>>>> Simone
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>>> For additional commands, e-mail: general-help@incubator.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Brock Noland <br...@cloudera.com>.
On Thu, Oct 18, 2012 at 2:57 PM, Chris Douglas <cd...@apache.org> wrote:
> On Thu, Oct 18, 2012 at 11:56 AM, Jakob Homan <jg...@gmail.com> wrote:
>> This seems like it might be more at home at apache-extras.
>
> That may ultimately be its fate, but building a community around these
> tools is worthy of an attempt. This also shares many goals with
> MRUnit. Collaboration or even merging with that community could also
> make sense. The code is dead in its current home; letting someone run
> with it here to see where it leads seems reasonable.
+1, as a MRUnit committer I think there are possible collaboration points here.
Brock
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Chris Douglas <cd...@apache.org>.
On Thu, Oct 18, 2012 at 11:56 AM, Jakob Homan <jg...@gmail.com> wrote:
> This seems like it might be more at home at apache-extras.
That may ultimately be its fate, but building a community around these
tools is worthy of an attempt. This also shares many goals with
MRUnit. Collaboration or even merging with that community could also
make sense. The code is dead in its current home; letting someone run
with it here to see where it leads seems reasonable.
> On Thu, Oct 18, 2012 at 9:28 AM, Adam Berry <am...@yahoo-inc.com> wrote:
>> I will update the proposal later on to include that, and additionally some idea of how we could support streaming from within the IDE.
Seems out of scope. The goal of the project, as I gathered from the
proposal, is to produce tools and plugins that complement development
of (and on) Hadoop. There's no need to specify which features the
community intends to build at this stage. -C
>> Great comments though folks, keep 'em coming!
>>
>> Cheers,
>> Adam
>>
>> On Oct 18, 2012, at 10:53 AM, Simone Gianni wrote:
>>
>>> I think it's a good idea, I used the old stuff at the time and it is
>>> something that is missing.
>>>
>>> 2012/10/18 Steve Loughran <st...@gmail.com>
>>>
>>>> it may be good to have a proposal that is broader than just eclipse; gives
>>>> you flexibility in future even if the initial target is the eclipse plugin.
>>>> Certainly "Hadoop Development Tools" is pretty broad
>>>>
>>>
>>> Steve you are right, but I'm afraid that given how the Eclipse framework
>>> requires tight coupling with Eclipse stuff (SWT, workbench system etc..)
>>> it's not easy to develop something that can easily be reused on different
>>> platforms.
>>>
>>> However, it could be possible to distribute a stand alone RCP application
>>> for those not using Eclipse, and eventually try to implement a "generic
>>> Hadoop-UI bridge", as far as possible agnostic of specific Eclipse classes
>>> and interfaces, that could eventually be reused for other IDEs.
>>>
>>> Simone
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
>> For additional commands, e-mail: general-help@incubator.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Jakob Homan <jg...@gmail.com>.
This seems like it might be more at home at apache-extras.
On Thu, Oct 18, 2012 at 9:28 AM, Adam Berry <am...@yahoo-inc.com> wrote:
>
> Ultimately I would like this project to be the definitive place to go for development tools for Hadoop and related projects. So if there was a drive to support multiple IDEs that would be great and something I would be completely behind.
>
> The reason I proposed it this way is this seems like a reasonably scoped incubator project that has a clear need in the existing community. I think it would be better to deliver solid Eclipse based tools, than promise tools for all IDEs and end up not able to deliver anything.
>
> I think a design goal of separating out as much core work for reuse would be sensible even in the initial stages, so that we have an eye on later supporting other tools if the community drives us that way.
>
> I will update the proposal later on to include that, and additionally some idea of how we could support streaming from within the IDE.
>
> Great comments though folks, keep 'em coming!
>
> Cheers,
> Adam
>
> On Oct 18, 2012, at 10:53 AM, Simone Gianni wrote:
>
>> I think it's a good idea, I used the old stuff at the time and it is
>> something that is missing.
>>
>> 2012/10/18 Steve Loughran <st...@gmail.com>
>>
>>> it may be good to have a proposal that is broader than just eclipse; gives
>>> you flexibility in future even if the initial target is the eclipse plugin.
>>> Certainly "Hadoop Development Tools" is pretty broad
>>>
>>
>> Steve you are right, but I'm afraid that given how the Eclipse framework
>> requires tight coupling with Eclipse stuff (SWT, workbench system etc..)
>> it's not easy to develop something that can easily be reused on different
>> platforms.
>>
>> However, it could be possible to distribute a stand alone RCP application
>> for those not using Eclipse, and eventually try to implement a "generic
>> Hadoop-UI bridge", as far as possible agnostic of specific Eclipse classes
>> and interfaces, that could eventually be reused for other IDEs.
>>
>> Simone
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
> For additional commands, e-mail: general-help@incubator.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Adam Berry <am...@yahoo-inc.com>.
Ultimately I would like this project to be the definitive place to go for development tools for Hadoop and related projects. So if there was a drive to support multiple IDEs that would be great and something I would be completely behind.
The reason I proposed it this way is this seems like a reasonably scoped incubator project that has a clear need in the existing community. I think it would be better to deliver solid Eclipse based tools, than promise tools for all IDEs and end up not able to deliver anything.
I think a design goal of separating out as much core work for reuse would be sensible even in the initial stages, so that we have an eye on later supporting other tools if the community drives us that way.
I will update the proposal later on to include that, and additionally some idea of how we could support streaming from within the IDE.
Great comments though folks, keep 'em coming!
Cheers,
Adam
On Oct 18, 2012, at 10:53 AM, Simone Gianni wrote:
> I think it's a good idea, I used the old stuff at the time and it is
> something that is missing.
>
> 2012/10/18 Steve Loughran <st...@gmail.com>
>
>> it may be good to have a proposal that is broader than just eclipse; gives
>> you flexibility in future even if the initial target is the eclipse plugin.
>> Certainly "Hadoop Development Tools" is pretty broad
>>
>
> Steve you are right, but I'm afraid that given how the Eclipse framework
> requires tight coupling with Eclipse stuff (SWT, workbench system etc..)
> it's not easy to develop something that can easily be reused on different
> platforms.
>
> However, it could be possible to distribute a stand alone RCP application
> for those not using Eclipse, and eventually try to implement a "generic
> Hadoop-UI bridge", as far as possible agnostic of specific Eclipse classes
> and interfaces, that could eventually be reused for other IDEs.
>
> Simone
---------------------------------------------------------------------
To unsubscribe, e-mail: general-unsubscribe@incubator.apache.org
For additional commands, e-mail: general-help@incubator.apache.org
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Simone Gianni <si...@apache.org>.
I think it's a good idea, I used the old stuff at the time and it is
something that is missing.
2012/10/18 Steve Loughran <st...@gmail.com>
> it may be good to have a proposal that is broader than just eclipse; gives
> you flexibility in future even if the initial target is the eclipse plugin.
> Certainly "Hadoop Development Tools" is pretty broad
>
Steve you are right, but I'm afraid that given how the Eclipse framework
requires tight coupling with Eclipse stuff (SWT, workbench system etc..)
it's not easy to develop something that can easily be reused on different
platforms.
However, it could be possible to distribute a stand alone RCP application
for those not using Eclipse, and eventually try to implement a "generic
Hadoop-UI bridge", as far as possible agnostic of specific Eclipse classes
and interfaces, that could eventually be reused for other IDEs.
Simone
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Steve Loughran <st...@gmail.com>.
it may be good to have a proposal that is broader than just eclipse; gives
you flexibility in future even if the initial target is the eclipse plugin.
Certainly "Hadoop Development Tools" is pretty broad
On 17 October 2012 21:01, Adam Berry <am...@yahoo-inc.com> wrote:
> Hello,
>
> Below is a proposal for a new incubator project. This idea came out of and
> had strong support on the general hadoop list, see the thread at
> http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser
> .
>
> We are looking for feedback, and to see who else would be interested in
> contributing to this effort as a committer, as well as an additional mentor.
>
> Cheers,
> Adam Berry
>
>
> Hadoop Development Tools Proposal
>
> I’d like to propose the Hadoop Development Tools, a set of extensions to
> the Eclipse IDE to support developing against Apache Hadoop technologies.
>
> = HDT (Hadoop Development Tools) =
>
> == Abstract ==
> Tools to support developing applications that use Apache Hadoop from
> within Eclipse.
>
> == Proposal ==
> Hadoop Development Tools are a set of extensions to Eclipse providing
> support for creating, launching and debugging distributed applications, as
> well as interacting with HDFS filesystems. This work will build on the
> existing Map Reduce Tools present in the Apache Hadoop project.
>
> == Background ==
> Map Reduce Tools have existed as part of contrib for Apache Hadoop.
> Unfortunately they are source tied to a single version of Hadoop, and
> development has stalled, with little movement past the Hadoop 0.20 line.
>
> == Rationale ==
> Support for newer versions of Hadoop from within Eclipse is regularly
> raised on the Hadoop mailing lists, so there is a clear need to drive these
> tools forward. Development tools generally are worked on separate from the
> target tools/platform, separating the tools out will allow for supporting
> multiple versions, so a developer could work with a heterogeneous
> environment.
>
> == Initial Goals ==
> * Give the tools project a home of its own.
> * Port current MapReduce tools feature set to all current release lines
> of Hadoop in a single Eclipse install.
> * Documentation and tutorials for all features.
> * Publish Eclipse update site, and join Eclipse marketplace listing.
> * Establish release cycle that combines support for Hadoop and Eclipse
> release cycles.
> * Look to build support for YARN, MRUnit and possibly other
> Hadoop-related projects.
>
> == Current Status ==
> The source for the current MapReduceTools lives in the contrib section of
> the Hadoop source. In its current implementation it is tied to the version
> of Hadoop against which it is compiled. The layout and API that it was
> developed with means that it can only be used with the 0.20 or 1.0 Hadoop
> releases, the new layout and YARN api introduced with the 0.23 and 2.0
> lines are not supported.
>
>
> === Meritocracy ===
> Several people and companies have already expressed an interest in
> contributing to this project, and we hope to attract additional interest
> during the proposal discussion. We plan to invest and support a meritocracy
> that attracts, invites, and supports newcomers to build a vibrant and
> diverse community.
>
> === Community ===
> The target community is developers who are working developing Map/Reduce
> applications against Hadoop. Given the success of Hadoop the target group
> is likely to be quite large. Separation from the Hadoop community would
> make it easier to support multiple versions of hadoop, as well as merging
> the release cycles of Hadoop and Eclipse to provide predictable iteration
> and improvement in the toolset.
>
> === Core Developers ===
> The initial list of developers includes people experienced with developing
> against the Eclipse platform.
> * Adam Berry (amberry at yahoo-inc dot com)
> * Jeffrey Zemerick (jeffrrey at mtnfog dot com)
>
> === Alignment ===
> Hadoop Development Tools aligns with both Hadoop and Eclipse. Hadoop as
> the platform for the development target, and Eclipse as the IDE platform
> used as the base for the tools.
>
> == Known Risks ==
>
> === Orphaned Products ===
>
> === Inexperience with Open Source ===
> Adam Berry has experience of the Eclipse open source community, and has
> been building familiarity with the Apache processes through patches to the
> existing source.
>
> === Reliance on Salaried Developers ===
> Hadoop Development Tools will be developed with a mix of salaried and
> volunteer time.
>
> === Relationships with Other Apache Projects ===
> Hadoop Development Tools is closely related to Apache Hadoop.
>
> === An Excessive Fascination with the Apache Brand ===
> Given the success of Hadoop and associated projects, Apache is the
> natural place for the Hadoop Development Tools. Chris Mattman suggested the
> Apache Incubator as appropriate on the Hadoop general mailing list
> following the success that MRUnit had taking the path from Hadoop contrib
> to an Apache top level project.
>
> == Documentation ==
> Documentation for the current tools can be found at
> http://wiki.apache.org/hadoop/EclipsePlugIn
>
> == Initial Source ==
>
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/eclipse-plugin/
>
> == Source and Intellectual Property Submission Plan ==
> The source, and any suggested initial patches, are already hosted either
> in Apache’s Subversion or JIRA.
>
> == External Dependencies ==
> Eclipse Platform
> Eclipse Java Development Tools
>
> == Cryptography ==
> Hadoop Development Tools likely does not fall into this area.
>
> == Required Resources ==
> === Mailing lists ===
> * hdt-dev
> * hdt-commits
> * hdt-user
>
> === Subversion Directory ===
> * https://svn.apache.org/repos/asf/incubator/hdt
>
> === Issue Tracking ===
> * JIRA Hadoop Development Tools (HDT)
>
> === Other Resources ===
> * Jenkins/Hudson for builds and test running.
>
> == Initial Committers ==
> * Adam Berry (amberry at yahoo-inc dot com)
> * Jeffrey Zemerick (jeffrrey at mtnfog dot com)
>
> == Affiliations ==
> * Adam Berry - Yahoo!
> * Jeffrey Zemerick - Mountain Fog
>
> == Sponsors ==
> === Champion ===
> Chris Douglas
>
> === Nominated Mentors ===
> Chris Douglas
> Chris Mattman
>
> === Sponsoring Entity ===
> Incubator PMC
>
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Adam Berry <am...@yahoo-inc.com>.
Just to keep everyone up to date over here as well, the vote has started over on the general@incubator.apache.org list.
Cheers,
Adam
On Oct 17, 2012, at 5:06 PM, Adam Berry wrote:
> Hi Evert,
>
> that sounds interesting and certainly useful. If you could contribute that would be great. I'll check back in with you in a couple of days.
>
> Cheers,
> Adam
>
> On Oct 17, 2012, at 4:52 PM, Evert Lammerts wrote:
>
>> Hi Adam, thanks for the initiative, this is good news for us and our users as well - right now I'm advising ant targets for submitting MR jobs from Eclipse. I'll have a look to see whether / how we can contribute.
>>
>> Best,
>> Evert
>>
>> ________________________________________
>> From: Adam Berry [amberry@yahoo-inc.com]
>> Sent: Wednesday, October 17, 2012 10:01 PM
>> To: general@hadoop.apache.org; general@incubator.apache.org
>> Subject: [DISCUSS] [PROPOSAL] Hadoop Development Tools
>>
>> Hello,
>>
>> Below is a proposal for a new incubator project. This idea came out of and had strong support on the general hadoop list, see the thread at http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser.
>>
>> We are looking for feedback, and to see who else would be interested in contributing to this effort as a committer, as well as an additional mentor.
>>
>> Cheers,
>> Adam Berry
>>
>>
>> Hadoop Development Tools Proposal
>>
>> I’d like to propose the Hadoop Development Tools, a set of extensions to the Eclipse IDE to support developing against Apache Hadoop technologies.
>>
>> = HDT (Hadoop Development Tools) =
>>
>> == Abstract ==
>> Tools to support developing applications that use Apache Hadoop from within Eclipse.
>>
>> == Proposal ==
>> Hadoop Development Tools are a set of extensions to Eclipse providing support for creating, launching and debugging distributed applications, as well as interacting with HDFS filesystems. This work will build on the existing Map Reduce Tools present in the Apache Hadoop project.
>>
>> == Background ==
>> Map Reduce Tools have existed as part of contrib for Apache Hadoop. Unfortunately they are source tied to a single version of Hadoop, and development has stalled, with little movement past the Hadoop 0.20 line.
>>
>> == Rationale ==
>> Support for newer versions of Hadoop from within Eclipse is regularly raised on the Hadoop mailing lists, so there is a clear need to drive these tools forward. Development tools generally are worked on separate from the target tools/platform, separating the tools out will allow for supporting multiple versions, so a developer could work with a heterogeneous environment.
>>
>> == Initial Goals ==
>> * Give the tools project a home of its own.
>> * Port current MapReduce tools feature set to all current release lines of Hadoop in a single Eclipse install.
>> * Documentation and tutorials for all features.
>> * Publish Eclipse update site, and join Eclipse marketplace listing.
>> * Establish release cycle that combines support for Hadoop and Eclipse release cycles.
>> * Look to build support for YARN, MRUnit and possibly other Hadoop-related projects.
>>
>> == Current Status ==
>> The source for the current MapReduceTools lives in the contrib section of the Hadoop source. In its current implementation it is tied to the version of Hadoop against which it is compiled. The layout and API that it was developed with means that it can only be used with the 0.20 or 1.0 Hadoop releases, the new layout and YARN api introduced with the 0.23 and 2.0 lines are not supported.
>>
>>
>> === Meritocracy ===
>> Several people and companies have already expressed an interest in contributing to this project, and we hope to attract additional interest during the proposal discussion. We plan to invest and support a meritocracy that attracts, invites, and supports newcomers to build a vibrant and diverse community.
>>
>> === Community ===
>> The target community is developers who are working developing Map/Reduce applications against Hadoop. Given the success of Hadoop the target group is likely to be quite large. Separation from the Hadoop community would make it easier to support multiple versions of hadoop, as well as merging the release cycles of Hadoop and Eclipse to provide predictable iteration and improvement in the toolset.
>>
>> === Core Developers ===
>> The initial list of developers includes people experienced with developing against the Eclipse platform.
>> * Adam Berry (amberry at yahoo-inc dot com)
>> * Jeffrey Zemerick (jeffrrey at mtnfog dot com)
>>
>> === Alignment ===
>> Hadoop Development Tools aligns with both Hadoop and Eclipse. Hadoop as the platform for the development target, and Eclipse as the IDE platform used as the base for the tools.
>>
>> == Known Risks ==
>>
>> === Orphaned Products ===
>>
>> === Inexperience with Open Source ===
>> Adam Berry has experience of the Eclipse open source community, and has been building familiarity with the Apache processes through patches to the existing source.
>>
>> === Reliance on Salaried Developers ===
>> Hadoop Development Tools will be developed with a mix of salaried and volunteer time.
>>
>> === Relationships with Other Apache Projects ===
>> Hadoop Development Tools is closely related to Apache Hadoop.
>>
>> === An Excessive Fascination with the Apache Brand ===
>> Given the success of Hadoop and associated projects, Apache is the natural place for the Hadoop Development Tools. Chris Mattman suggested the Apache Incubator as appropriate on the Hadoop general mailing list following the success that MRUnit had taking the path from Hadoop contrib to an Apache top level project.
>>
>> == Documentation ==
>> Documentation for the current tools can be found at http://wiki.apache.org/hadoop/EclipsePlugIn
>>
>> == Initial Source ==
>> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/eclipse-plugin/
>>
>> == Source and Intellectual Property Submission Plan ==
>> The source, and any suggested initial patches, are already hosted either in Apache’s Subversion or JIRA.
>>
>> == External Dependencies ==
>> Eclipse Platform
>> Eclipse Java Development Tools
>>
>> == Cryptography ==
>> Hadoop Development Tools likely does not fall into this area.
>>
>> == Required Resources ==
>> === Mailing lists ===
>> * hdt-dev
>> * hdt-commits
>> * hdt-user
>>
>> === Subversion Directory ===
>> * https://svn.apache.org/repos/asf/incubator/hdt
>>
>> === Issue Tracking ===
>> * JIRA Hadoop Development Tools (HDT)
>>
>> === Other Resources ===
>> * Jenkins/Hudson for builds and test running.
>>
>> == Initial Committers ==
>> * Adam Berry (amberry at yahoo-inc dot com)
>> * Jeffrey Zemerick (jeffrrey at mtnfog dot com)
>>
>> == Affiliations ==
>> * Adam Berry - Yahoo!
>> * Jeffrey Zemerick - Mountain Fog
>>
>> == Sponsors ==
>> === Champion ===
>> Chris Douglas
>>
>> === Nominated Mentors ===
>> Chris Douglas
>> Chris Mattman
>>
>> === Sponsoring Entity ===
>> Incubator PMC
>
Re: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Adam Berry <am...@yahoo-inc.com>.
Hi Evert,
that sounds interesting and certainly useful. If you could contribute that would be great. I'll check back in with you in a couple of days.
Cheers,
Adam
On Oct 17, 2012, at 4:52 PM, Evert Lammerts wrote:
> Hi Adam, thanks for the initiative, this is good news for us and our users as well - right now I'm advising ant targets for submitting MR jobs from Eclipse. I'll have a look to see whether / how we can contribute.
>
> Best,
> Evert
>
> ________________________________________
> From: Adam Berry [amberry@yahoo-inc.com]
> Sent: Wednesday, October 17, 2012 10:01 PM
> To: general@hadoop.apache.org; general@incubator.apache.org
> Subject: [DISCUSS] [PROPOSAL] Hadoop Development Tools
>
> Hello,
>
> Below is a proposal for a new incubator project. This idea came out of and had strong support on the general hadoop list, see the thread at http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser.
>
> We are looking for feedback, and to see who else would be interested in contributing to this effort as a committer, as well as an additional mentor.
>
> Cheers,
> Adam Berry
>
>
> Hadoop Development Tools Proposal
>
> I’d like to propose the Hadoop Development Tools, a set of extensions to the Eclipse IDE to support developing against Apache Hadoop technologies.
>
> = HDT (Hadoop Development Tools) =
>
> == Abstract ==
> Tools to support developing applications that use Apache Hadoop from within Eclipse.
>
> == Proposal ==
> Hadoop Development Tools are a set of extensions to Eclipse providing support for creating, launching and debugging distributed applications, as well as interacting with HDFS filesystems. This work will build on the existing Map Reduce Tools present in the Apache Hadoop project.
>
> == Background ==
> Map Reduce Tools have existed as part of contrib for Apache Hadoop. Unfortunately they are source tied to a single version of Hadoop, and development has stalled, with little movement past the Hadoop 0.20 line.
>
> == Rationale ==
> Support for newer versions of Hadoop from within Eclipse is regularly raised on the Hadoop mailing lists, so there is a clear need to drive these tools forward. Development tools generally are worked on separate from the target tools/platform, separating the tools out will allow for supporting multiple versions, so a developer could work with a heterogeneous environment.
>
> == Initial Goals ==
> * Give the tools project a home of its own.
> * Port current MapReduce tools feature set to all current release lines of Hadoop in a single Eclipse install.
> * Documentation and tutorials for all features.
> * Publish Eclipse update site, and join Eclipse marketplace listing.
> * Establish release cycle that combines support for Hadoop and Eclipse release cycles.
> * Look to build support for YARN, MRUnit and possibly other Hadoop-related projects.
>
> == Current Status ==
> The source for the current MapReduceTools lives in the contrib section of the Hadoop source. In its current implementation it is tied to the version of Hadoop against which it is compiled. The layout and API that it was developed with means that it can only be used with the 0.20 or 1.0 Hadoop releases, the new layout and YARN api introduced with the 0.23 and 2.0 lines are not supported.
>
>
> === Meritocracy ===
> Several people and companies have already expressed an interest in contributing to this project, and we hope to attract additional interest during the proposal discussion. We plan to invest and support a meritocracy that attracts, invites, and supports newcomers to build a vibrant and diverse community.
>
> === Community ===
> The target community is developers who are working developing Map/Reduce applications against Hadoop. Given the success of Hadoop the target group is likely to be quite large. Separation from the Hadoop community would make it easier to support multiple versions of hadoop, as well as merging the release cycles of Hadoop and Eclipse to provide predictable iteration and improvement in the toolset.
>
> === Core Developers ===
> The initial list of developers includes people experienced with developing against the Eclipse platform.
> * Adam Berry (amberry at yahoo-inc dot com)
> * Jeffrey Zemerick (jeffrrey at mtnfog dot com)
>
> === Alignment ===
> Hadoop Development Tools aligns with both Hadoop and Eclipse. Hadoop as the platform for the development target, and Eclipse as the IDE platform used as the base for the tools.
>
> == Known Risks ==
>
> === Orphaned Products ===
>
> === Inexperience with Open Source ===
> Adam Berry has experience of the Eclipse open source community, and has been building familiarity with the Apache processes through patches to the existing source.
>
> === Reliance on Salaried Developers ===
> Hadoop Development Tools will be developed with a mix of salaried and volunteer time.
>
> === Relationships with Other Apache Projects ===
> Hadoop Development Tools is closely related to Apache Hadoop.
>
> === An Excessive Fascination with the Apache Brand ===
> Given the success of Hadoop and associated projects, Apache is the natural place for the Hadoop Development Tools. Chris Mattman suggested the Apache Incubator as appropriate on the Hadoop general mailing list following the success that MRUnit had taking the path from Hadoop contrib to an Apache top level project.
>
> == Documentation ==
> Documentation for the current tools can be found at http://wiki.apache.org/hadoop/EclipsePlugIn
>
> == Initial Source ==
> http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/eclipse-plugin/
>
> == Source and Intellectual Property Submission Plan ==
> The source, and any suggested initial patches, are already hosted either in Apache’s Subversion or JIRA.
>
> == External Dependencies ==
> Eclipse Platform
> Eclipse Java Development Tools
>
> == Cryptography ==
> Hadoop Development Tools likely does not fall into this area.
>
> == Required Resources ==
> === Mailing lists ===
> * hdt-dev
> * hdt-commits
> * hdt-user
>
> === Subversion Directory ===
> * https://svn.apache.org/repos/asf/incubator/hdt
>
> === Issue Tracking ===
> * JIRA Hadoop Development Tools (HDT)
>
> === Other Resources ===
> * Jenkins/Hudson for builds and test running.
>
> == Initial Committers ==
> * Adam Berry (amberry at yahoo-inc dot com)
> * Jeffrey Zemerick (jeffrrey at mtnfog dot com)
>
> == Affiliations ==
> * Adam Berry - Yahoo!
> * Jeffrey Zemerick - Mountain Fog
>
> == Sponsors ==
> === Champion ===
> Chris Douglas
>
> === Nominated Mentors ===
> Chris Douglas
> Chris Mattman
>
> === Sponsoring Entity ===
> Incubator PMC
RE: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Posted by Evert Lammerts <Ev...@sara.nl>.
Hi Adam, thanks for the initiative, this is good news for us and our users as well - right now I'm advising ant targets for submitting MR jobs from Eclipse. I'll have a look to see whether / how we can contribute.
Best,
Evert
________________________________________
From: Adam Berry [amberry@yahoo-inc.com]
Sent: Wednesday, October 17, 2012 10:01 PM
To: general@hadoop.apache.org; general@incubator.apache.org
Subject: [DISCUSS] [PROPOSAL] Hadoop Development Tools
Hello,
Below is a proposal for a new incubator project. This idea came out of and had strong support on the general hadoop list, see the thread at http://mail-archives.apache.org/mod_mbox/hadoop-general/201209.mbox/browser.
We are looking for feedback, and to see who else would be interested in contributing to this effort as a committer, as well as an additional mentor.
Cheers,
Adam Berry
Hadoop Development Tools Proposal
I’d like to propose the Hadoop Development Tools, a set of extensions to the Eclipse IDE to support developing against Apache Hadoop technologies.
= HDT (Hadoop Development Tools) =
== Abstract ==
Tools to support developing applications that use Apache Hadoop from within Eclipse.
== Proposal ==
Hadoop Development Tools are a set of extensions to Eclipse providing support for creating, launching and debugging distributed applications, as well as interacting with HDFS filesystems. This work will build on the existing Map Reduce Tools present in the Apache Hadoop project.
== Background ==
Map Reduce Tools have existed as part of contrib for Apache Hadoop. Unfortunately they are source tied to a single version of Hadoop, and development has stalled, with little movement past the Hadoop 0.20 line.
== Rationale ==
Support for newer versions of Hadoop from within Eclipse is regularly raised on the Hadoop mailing lists, so there is a clear need to drive these tools forward. Development tools generally are worked on separate from the target tools/platform, separating the tools out will allow for supporting multiple versions, so a developer could work with a heterogeneous environment.
== Initial Goals ==
* Give the tools project a home of its own.
* Port current MapReduce tools feature set to all current release lines of Hadoop in a single Eclipse install.
* Documentation and tutorials for all features.
* Publish Eclipse update site, and join Eclipse marketplace listing.
* Establish release cycle that combines support for Hadoop and Eclipse release cycles.
* Look to build support for YARN, MRUnit and possibly other Hadoop-related projects.
== Current Status ==
The source for the current MapReduceTools lives in the contrib section of the Hadoop source. In its current implementation it is tied to the version of Hadoop against which it is compiled. The layout and API that it was developed with means that it can only be used with the 0.20 or 1.0 Hadoop releases, the new layout and YARN api introduced with the 0.23 and 2.0 lines are not supported.
=== Meritocracy ===
Several people and companies have already expressed an interest in contributing to this project, and we hope to attract additional interest during the proposal discussion. We plan to invest and support a meritocracy that attracts, invites, and supports newcomers to build a vibrant and diverse community.
=== Community ===
The target community is developers who are working developing Map/Reduce applications against Hadoop. Given the success of Hadoop the target group is likely to be quite large. Separation from the Hadoop community would make it easier to support multiple versions of hadoop, as well as merging the release cycles of Hadoop and Eclipse to provide predictable iteration and improvement in the toolset.
=== Core Developers ===
The initial list of developers includes people experienced with developing against the Eclipse platform.
* Adam Berry (amberry at yahoo-inc dot com)
* Jeffrey Zemerick (jeffrrey at mtnfog dot com)
=== Alignment ===
Hadoop Development Tools aligns with both Hadoop and Eclipse. Hadoop as the platform for the development target, and Eclipse as the IDE platform used as the base for the tools.
== Known Risks ==
=== Orphaned Products ===
=== Inexperience with Open Source ===
Adam Berry has experience of the Eclipse open source community, and has been building familiarity with the Apache processes through patches to the existing source.
=== Reliance on Salaried Developers ===
Hadoop Development Tools will be developed with a mix of salaried and volunteer time.
=== Relationships with Other Apache Projects ===
Hadoop Development Tools is closely related to Apache Hadoop.
=== An Excessive Fascination with the Apache Brand ===
Given the success of Hadoop and associated projects, Apache is the natural place for the Hadoop Development Tools. Chris Mattman suggested the Apache Incubator as appropriate on the Hadoop general mailing list following the success that MRUnit had taking the path from Hadoop contrib to an Apache top level project.
== Documentation ==
Documentation for the current tools can be found at http://wiki.apache.org/hadoop/EclipsePlugIn
== Initial Source ==
http://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/eclipse-plugin/
== Source and Intellectual Property Submission Plan ==
The source, and any suggested initial patches, are already hosted either in Apache’s Subversion or JIRA.
== External Dependencies ==
Eclipse Platform
Eclipse Java Development Tools
== Cryptography ==
Hadoop Development Tools likely does not fall into this area.
== Required Resources ==
=== Mailing lists ===
* hdt-dev
* hdt-commits
* hdt-user
=== Subversion Directory ===
* https://svn.apache.org/repos/asf/incubator/hdt
=== Issue Tracking ===
* JIRA Hadoop Development Tools (HDT)
=== Other Resources ===
* Jenkins/Hudson for builds and test running.
== Initial Committers ==
* Adam Berry (amberry at yahoo-inc dot com)
* Jeffrey Zemerick (jeffrrey at mtnfog dot com)
== Affiliations ==
* Adam Berry - Yahoo!
* Jeffrey Zemerick - Mountain Fog
== Sponsors ==
=== Champion ===
Chris Douglas
=== Nominated Mentors ===
Chris Douglas
Chris Mattman
=== Sponsoring Entity ===
Incubator PMC