You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Arun C Murthy <ac...@yahoo-inc.com> on 2011/06/10 21:44:35 UTC

Status on MAPREDUCE-279 and plans forward

Another heads up.

Things are going well, we have done scale testing (350 nodes) and  
continue to do more.

We think we are nearly ready to merge MR-279 into trunk (yaay!).

A proposal for plan forward:

1. I'd like to get MAPREDUCE-2400 committed to both trunk and MR-279.  
This is a very important patch by Tom which allows to submit jobs to  
both JT and the new RM - thanks Tom!
2. Check-in the 'yarn' subfolder which is just the ResourceManager and  
NodeManager i.e. compute fabric to trunk.
3. Rework the src tree in trunk to look like the MR-279 branch to  
ensure no changes to trunk are lost - this will ensure separation  
between the 'classic' framework (JT/TT) and the MR runtime (map/reduce/ 
sort/shuffle etc.)
4. Re-play the 4-5 patches needed to the MR runtime from the MR-279  
branch.
5. Change build scripts to reflect the new reality.

In terms of timelines, I'm hoping we can get this done in the coming  
2-3 weeks.

Thoughts?

thanks,
Arun

Re: Status on MAPREDUCE-279 and plans forward

Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On Jun 11, 2011, at 6:28 AM, Tom White wrote:

> Hi Arun,
>
> Great to hear that MR2 is nearly ready for trunk.
>
> MR2 is being built using Maven, so step 5 is about getting Maven to
> build all of MR I presume? The svn unsplit (HADOOP-7106) and
> Mavenization work in HADOOP-6671 should make this approach more
> tractable. Or were you thinking that there would be a hybrid Ant/Maven
> build during a transition period?
>

Good point Tom. I'm thinking of having a hybrid ant/maven transition  
period to allow both to proceed independently.

> Also, for step 3, what did you think of the directory layout I
> suggested in http://bit.ly/mr-279-maven-layout?
>

Makes sense, maybe we should stage that too. Lets continue that  
discussion on jira.

thanks!
Arun

> Cheers,
> Tom
>
> On Fri, Jun 10, 2011 at 12:44 PM, Arun C Murthy <ac...@yahoo-inc.com>  
> wrote:
>> Another heads up.
>>
>> Things are going well, we have done scale testing (350 nodes) and  
>> continue
>> to do more.
>>
>> We think we are nearly ready to merge MR-279 into trunk (yaay!).
>>
>> A proposal for plan forward:
>>
>> 1. I'd like to get MAPREDUCE-2400 committed to both trunk and  
>> MR-279. This
>> is a very important patch by Tom which allows to submit jobs to  
>> both JT and
>> the new RM - thanks Tom!
>> 2. Check-in the 'yarn' subfolder which is just the ResourceManager  
>> and
>> NodeManager i.e. compute fabric to trunk.
>> 3. Rework the src tree in trunk to look like the MR-279 branch to  
>> ensure no
>> changes to trunk are lost - this will ensure separation between the
>> 'classic' framework (JT/TT) and the MR runtime (map/reduce/sort/ 
>> shuffle
>> etc.)
>> 4. Re-play the 4-5 patches needed to the MR runtime from the MR-279  
>> branch.
>> 5. Change build scripts to reflect the new reality.
>>
>> In terms of timelines, I'm hoping we can get this done in the  
>> coming 2-3
>> weeks.
>>
>> Thoughts?
>>
>> thanks,
>> Arun
>>

Re: Status on MAPREDUCE-279 and plans forward

Posted by Tom White <to...@cloudera.com>.
Hi Arun,

Great to hear that MR2 is nearly ready for trunk.

MR2 is being built using Maven, so step 5 is about getting Maven to
build all of MR I presume? The svn unsplit (HADOOP-7106) and
Mavenization work in HADOOP-6671 should make this approach more
tractable. Or were you thinking that there would be a hybrid Ant/Maven
build during a transition period?

Also, for step 3, what did you think of the directory layout I
suggested in http://bit.ly/mr-279-maven-layout?

Cheers,
Tom

On Fri, Jun 10, 2011 at 12:44 PM, Arun C Murthy <ac...@yahoo-inc.com> wrote:
> Another heads up.
>
> Things are going well, we have done scale testing (350 nodes) and continue
> to do more.
>
> We think we are nearly ready to merge MR-279 into trunk (yaay!).
>
> A proposal for plan forward:
>
> 1. I'd like to get MAPREDUCE-2400 committed to both trunk and MR-279. This
> is a very important patch by Tom which allows to submit jobs to both JT and
> the new RM - thanks Tom!
> 2. Check-in the 'yarn' subfolder which is just the ResourceManager and
> NodeManager i.e. compute fabric to trunk.
> 3. Rework the src tree in trunk to look like the MR-279 branch to ensure no
> changes to trunk are lost - this will ensure separation between the
> 'classic' framework (JT/TT) and the MR runtime (map/reduce/sort/shuffle
> etc.)
> 4. Re-play the 4-5 patches needed to the MR runtime from the MR-279 branch.
> 5. Change build scripts to reflect the new reality.
>
> In terms of timelines, I'm hoping we can get this done in the coming 2-3
> weeks.
>
> Thoughts?
>
> thanks,
> Arun
>