Native task branch progress

[Todd Lipcon <to...@apache.org>]

Hi all,

I wanted to give a quick status update on the MAPREDUCE-2841 feature branch
which was started earlier this summer.

Since establishing the branch, we've accomplished a number of key items:

- Imported the initial code from github
- Made integrations with other projects more pluggable using a
ServiceLoader mechanism
- Integrated the project build into our normal maven structure and release
tarballs
- Fixed several unit test failures that occurred in different developer
environments
- Fixed a couple of licensing issues with code
- Various code cleanup for style, etc.
- Fixed all of the compiler warnings

I've also spent some time doing comparative microbenchmarks of the mapside
sort between this implementation, the existing MR implementation, and
Facebook's implementation (available on github). I haven't posted those
yet, but I hope to do so later this week. The summary is that the native
implementation compares quite favorably to the other options.

This week, I'm hoping to close out a couple remaining items:

- A few small code cleanup items I have pending locally (eg improving
javadocs, etc)
- Some basic cluster verification and performance tests
- Post benchmark results mentioned above

After these items are done, I think we are ready to merge this feature to
trunk. After it is in trunk, we can continue to make improvements, add
documentation, etc. Because the patch is structured as completely new code
(no changes to existing functionality, manually enabled on a per-job
basis), this should be a "zero-risk" merge.

I'm anticipating calling this merge vote next week, and would appreciate if
other committers can plan to be ready to vote at that time. That is to say,
if you anticipate wanting to review the branch merge, this is your "heads
up" that the vote is coming. Please start your reviews this week.

Thanks
-Todd

Re: Native task branch progress

[Todd Lipcon <to...@apache.org>]

Hi folks,

Just to follow up here: benchmark and cluster validation results are
available on the MR-2841 JIRA. Over the next few days we're closing out a
couple small remaining issues and I'll start posting candidate merge
patches for Jenkins to test.

I anticipate calling the merge vote in parallel with the Jenkins precommit
checks and any small follow-ons that that might expose. If there's any
specific testing items you feel you need to see to support this merge,
please let me know now.

Thanks
-Todd


On Tue, Aug 26, 2014 at 11:50 AM, Todd Lipcon <to...@apache.org> wrote:

> Hi all,
>
> I wanted to give a quick status update on the MAPREDUCE-2841 feature
> branch which was started earlier this summer.
>
> Since establishing the branch, we've accomplished a number of key items:
>
> - Imported the initial code from github
> - Made integrations with other projects more pluggable using a
> ServiceLoader mechanism
> - Integrated the project build into our normal maven structure and release
> tarballs
> - Fixed several unit test failures that occurred in different developer
> environments
> - Fixed a couple of licensing issues with code
> - Various code cleanup for style, etc.
> - Fixed all of the compiler warnings
>
> I've also spent some time doing comparative microbenchmarks of the mapside
> sort between this implementation, the existing MR implementation, and
> Facebook's implementation (available on github). I haven't posted those
> yet, but I hope to do so later this week. The summary is that the native
> implementation compares quite favorably to the other options.
>
> This week, I'm hoping to close out a couple remaining items:
>
> - A few small code cleanup items I have pending locally (eg improving
> javadocs, etc)
> - Some basic cluster verification and performance tests
> - Post benchmark results mentioned above
>
> After these items are done, I think we are ready to merge this feature to
> trunk. After it is in trunk, we can continue to make improvements, add
> documentation, etc. Because the patch is structured as completely new code
> (no changes to existing functionality, manually enabled on a per-job
> basis), this should be a "zero-risk" merge.
>
> I'm anticipating calling this merge vote next week, and would appreciate
> if other committers can plan to be ready to vote at that time. That is to
> say, if you anticipate wanting to review the branch merge, this is your
> "heads up" that the vote is coming. Please start your reviews this week.
>
> Thanks
> -Todd
>



-- 
Todd Lipcon
Software Engineer, Cloudera