You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Arun C Murthy <ac...@yahoo-inc.com> on 2011/05/10 09:14:37 UTC
Re: Developing on MAPREDUCE-279
Folks,
On Mar 21, 2011, Arun C Murthy wrote:
> We are very happy to have pushed MR-279 to the MR-279 dev-branch
> where we look forward to collaborating with everyone.
>
> This work is only partially complete. We plan to continue its
> development in the branch and do so in an aggressive manner - we want
> to get it done!
Just thought I'd give a heads up.
Things are progressing really well on MR-279 - all of the major
features we planned for are complete and checked-in; and we are
currently in a developer-testing phase. We would love for more eyes an
feedback.
We have put up some test plans at http://wiki.apache.org/hadoop/NextGenMapReduceDevTesting
.
I've been asked for some more docs offline and I'll get some of it on
the MAPREDUCE-279 jira soon (hopefully tmrw).
thanks,
Arun
Re: Status on MAPREDUCE-279 and plans forward
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On Jun 11, 2011, at 6:28 AM, Tom White wrote:
> Hi Arun,
>
> Great to hear that MR2 is nearly ready for trunk.
>
> MR2 is being built using Maven, so step 5 is about getting Maven to
> build all of MR I presume? The svn unsplit (HADOOP-7106) and
> Mavenization work in HADOOP-6671 should make this approach more
> tractable. Or were you thinking that there would be a hybrid Ant/Maven
> build during a transition period?
>
Good point Tom. I'm thinking of having a hybrid ant/maven transition
period to allow both to proceed independently.
> Also, for step 3, what did you think of the directory layout I
> suggested in http://bit.ly/mr-279-maven-layout?
>
Makes sense, maybe we should stage that too. Lets continue that
discussion on jira.
thanks!
Arun
> Cheers,
> Tom
>
> On Fri, Jun 10, 2011 at 12:44 PM, Arun C Murthy <ac...@yahoo-inc.com>
> wrote:
>> Another heads up.
>>
>> Things are going well, we have done scale testing (350 nodes) and
>> continue
>> to do more.
>>
>> We think we are nearly ready to merge MR-279 into trunk (yaay!).
>>
>> A proposal for plan forward:
>>
>> 1. I'd like to get MAPREDUCE-2400 committed to both trunk and
>> MR-279. This
>> is a very important patch by Tom which allows to submit jobs to
>> both JT and
>> the new RM - thanks Tom!
>> 2. Check-in the 'yarn' subfolder which is just the ResourceManager
>> and
>> NodeManager i.e. compute fabric to trunk.
>> 3. Rework the src tree in trunk to look like the MR-279 branch to
>> ensure no
>> changes to trunk are lost - this will ensure separation between the
>> 'classic' framework (JT/TT) and the MR runtime (map/reduce/sort/
>> shuffle
>> etc.)
>> 4. Re-play the 4-5 patches needed to the MR runtime from the MR-279
>> branch.
>> 5. Change build scripts to reflect the new reality.
>>
>> In terms of timelines, I'm hoping we can get this done in the
>> coming 2-3
>> weeks.
>>
>> Thoughts?
>>
>> thanks,
>> Arun
>>
Re: Status on MAPREDUCE-279 and plans forward
Posted by Tom White <to...@cloudera.com>.
Hi Arun,
Great to hear that MR2 is nearly ready for trunk.
MR2 is being built using Maven, so step 5 is about getting Maven to
build all of MR I presume? The svn unsplit (HADOOP-7106) and
Mavenization work in HADOOP-6671 should make this approach more
tractable. Or were you thinking that there would be a hybrid Ant/Maven
build during a transition period?
Also, for step 3, what did you think of the directory layout I
suggested in http://bit.ly/mr-279-maven-layout?
Cheers,
Tom
On Fri, Jun 10, 2011 at 12:44 PM, Arun C Murthy <ac...@yahoo-inc.com> wrote:
> Another heads up.
>
> Things are going well, we have done scale testing (350 nodes) and continue
> to do more.
>
> We think we are nearly ready to merge MR-279 into trunk (yaay!).
>
> A proposal for plan forward:
>
> 1. I'd like to get MAPREDUCE-2400 committed to both trunk and MR-279. This
> is a very important patch by Tom which allows to submit jobs to both JT and
> the new RM - thanks Tom!
> 2. Check-in the 'yarn' subfolder which is just the ResourceManager and
> NodeManager i.e. compute fabric to trunk.
> 3. Rework the src tree in trunk to look like the MR-279 branch to ensure no
> changes to trunk are lost - this will ensure separation between the
> 'classic' framework (JT/TT) and the MR runtime (map/reduce/sort/shuffle
> etc.)
> 4. Re-play the 4-5 patches needed to the MR runtime from the MR-279 branch.
> 5. Change build scripts to reflect the new reality.
>
> In terms of timelines, I'm hoping we can get this done in the coming 2-3
> weeks.
>
> Thoughts?
>
> thanks,
> Arun
>
Status on MAPREDUCE-279 and plans forward
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
Another heads up.
Things are going well, we have done scale testing (350 nodes) and
continue to do more.
We think we are nearly ready to merge MR-279 into trunk (yaay!).
A proposal for plan forward:
1. I'd like to get MAPREDUCE-2400 committed to both trunk and MR-279.
This is a very important patch by Tom which allows to submit jobs to
both JT and the new RM - thanks Tom!
2. Check-in the 'yarn' subfolder which is just the ResourceManager and
NodeManager i.e. compute fabric to trunk.
3. Rework the src tree in trunk to look like the MR-279 branch to
ensure no changes to trunk are lost - this will ensure separation
between the 'classic' framework (JT/TT) and the MR runtime (map/reduce/
sort/shuffle etc.)
4. Re-play the 4-5 patches needed to the MR runtime from the MR-279
branch.
5. Change build scripts to reflect the new reality.
In terms of timelines, I'm hoping we can get this done in the coming
2-3 weeks.
Thoughts?
thanks,
Arun