You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tez.apache.org by Jonathan Eagles <je...@gmail.com> on 2017/05/19 20:16:32 UTC
[VOTE] Merge TEZ-3334 into master
This vote is to merge the Tez Shuffle Handler feature branch (TEZ-3334)
into the master branch. Having done extensive testing and thanks to all the
feedback given, the Tez Shuffle Handler delivers on its promise to reduce
shuffle times for the auto-reduce case and pave the way for more shuffle
types in the future.
Please view a patch of the proposed merge attached to the TEZ-3334 jira
https://issues.apache.org/jira/browse/TEZ-3334
As to be the most useful and time efficient, please test and vote early if
you have any misgivings or questions about the readiness of the branch so
as to not to delay the merge process and increase the amount of work on my
part to continue to maintain this branch with master.
This vote will be open until Wednesday around the same time.
Here is my +1.
Regards,
jeagles
Re: [VOTE] Merge TEZ-3334 into master
Posted by Kuhu Shukla <ks...@yahoo-inc.com.INVALID>.
+1. Looks good to me.
- Applied patch and compiled against master. - Ran sample mrrsleep job with tez_shuffle.
Thanks and Regards,Kuhu
On Friday, May 19, 2017 6:04 PM, Siddharth Seth <ss...@apache.org> wrote:
+1. Looking forward to using this. Some of the CartesianProduct re-work
makes use of contiguous partitions as well.
Also, will create a 0.9 release soon after this goes in.
Thanks!
Sid
On Fri, May 19, 2017 at 1:16 PM, Jonathan Eagles <je...@gmail.com> wrote:
> This vote is to merge the Tez Shuffle Handler feature branch (TEZ-3334)
> into the master branch. Having done extensive testing and thanks to all the
> feedback given, the Tez Shuffle Handler delivers on its promise to reduce
> shuffle times for the auto-reduce case and pave the way for more shuffle
> types in the future.
>
> Please view a patch of the proposed merge attached to the TEZ-3334 jira
> https://issues.apache.org/jira/browse/TEZ-3334
>
> As to be the most useful and time efficient, please test and vote early if
> you have any misgivings or questions about the readiness of the branch so
> as to not to delay the merge process and increase the amount of work on my
> part to continue to maintain this branch with master.
>
> This vote will be open until Wednesday around the same time.
>
> Here is my +1.
>
> Regards,
> jeagles
>
Re: [VOTE] Merge TEZ-3334 into master
Posted by Siddharth Seth <ss...@apache.org>.
+1. Looking forward to using this. Some of the CartesianProduct re-work
makes use of contiguous partitions as well.
Also, will create a 0.9 release soon after this goes in.
Thanks!
Sid
On Fri, May 19, 2017 at 1:16 PM, Jonathan Eagles <je...@gmail.com> wrote:
> This vote is to merge the Tez Shuffle Handler feature branch (TEZ-3334)
> into the master branch. Having done extensive testing and thanks to all the
> feedback given, the Tez Shuffle Handler delivers on its promise to reduce
> shuffle times for the auto-reduce case and pave the way for more shuffle
> types in the future.
>
> Please view a patch of the proposed merge attached to the TEZ-3334 jira
> https://issues.apache.org/jira/browse/TEZ-3334
>
> As to be the most useful and time efficient, please test and vote early if
> you have any misgivings or questions about the readiness of the branch so
> as to not to delay the merge process and increase the amount of work on my
> part to continue to maintain this branch with master.
>
> This vote will be open until Wednesday around the same time.
>
> Here is my +1.
>
> Regards,
> jeagles
>
Re: [VOTE] Merge TEZ-3334 into master
Posted by Jonathan Eagles <je...@gmail.com>.
The merge is completed. Thanks for holding off. Please resume any and all
normal operations.
Regards,
jeagles
On Wed, May 24, 2017 at 3:54 PM, Jonathan Eagles <je...@gmail.com> wrote:
> With six +1 votes (5 binding and 1 non-binding) and zero -1 votes, this
> feature branch merge vote passes. I will plan on merging this feature
> branch soon. Please hold on off commits for a bit while I finalize this
> merge.
>
> Thanks again to all who contributed to this feature branch including
> contributing/reviewing patches, testing, voting, and community support.
>
> Regards,
> jeagles
>
> On Wed, May 24, 2017 at 2:08 PM, Jonathan Eagles <je...@gmail.com>
> wrote:
>
>> Ming, in its current form, Tez Shuffle Handler is not setup to handle
>> backwards compatibility with MapReduce. This could be easily added, the
>> complication is that MapReduce hard-codes its auxiliary shuffle service to
>> be named "mapreduce_shuffle". In addition, there are no compelling benefits
>> for using the Tez Shuffle Handler with mapreduce except not having to have
>> two services running. For a typical setup, yarn should define both
>> mapreduce_shuffle and tez_shuffle auxiliary services and run them
>> concurrently.
>>
>> jeagles
>>
>> On Tue, May 23, 2017 at 7:28 PM, Ming Ma <mi...@twitter.com.invalid>
>> wrote:
>>
>>> +1. Is Tez shuffle handler backward compatible with MR jobs? For a test
>>> cluster running Tez and MR jobs, if Tez shuffle handler is backward
>>> compatible, we can configure NM to only load Tez shuffle handler.
>>> Otherwise, will need to configure NM to load both Tez and MR shuffle
>>> handlers.
>>>
>>> On Tue, May 23, 2017 at 9:16 AM, Eric Badger
>>> <eb...@yahoo-inc.com.invalid>
>>> wrote:
>>>
>>> > +1 (non-binding) given the test failure followup jira ([TEZ-3735] Test
>>> > failures in TestTaskAttempt and TestAMContainerMap - ASF JIRA)
>>> > Eric
>>> >
>>> > |
>>> > | |
>>> > [TEZ-3735] Test failures in TestTaskAttempt and TestAMContainerMap -
>>> ASF
>>> > JIRA
>>> > | |
>>> >
>>> > |
>>> >
>>> >
>>> >
>>> >
>>> > On Monday, May 22, 2017 3:46 PM, Jason Lowe
>>> > <jl...@yahoo-inc.com.INVALID> wrote:
>>> >
>>> >
>>> > +1, looking forward to this functionality being available in an
>>> official
>>> > release!
>>> > Jason
>>> >
>>> >
>>> > On Friday, May 19, 2017 3:16 PM, Jonathan Eagles <
>>> jeagles@gmail.com>
>>> > wrote:
>>> >
>>> >
>>> > This vote is to merge the Tez Shuffle Handler feature branch
>>> (TEZ-3334)
>>> > into the master branch. Having done extensive testing and thanks to
>>> all the
>>> > feedback given, the Tez Shuffle Handler delivers on its promise to
>>> reduce
>>> > shuffle times for the auto-reduce case and pave the way for more
>>> shuffle
>>> > types in the future.
>>> >
>>> > Please view a patch of the proposed merge attached to the TEZ-3334 jira
>>> > https://issues.apache.org/jira/browse/TEZ-3334
>>> >
>>> > As to be the most useful and time efficient, please test and vote
>>> early if
>>> > you have any misgivings or questions about the readiness of the branch
>>> so
>>> > as to not to delay the merge process and increase the amount of work
>>> on my
>>> > part to continue to maintain this branch with master.
>>> >
>>> > This vote will be open until Wednesday around the same time.
>>> >
>>> > Here is my +1.
>>> >
>>> > Regards,
>>> > jeagles
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>>
>>
>>
>
Re: [VOTE] Merge TEZ-3334 into master
Posted by Jonathan Eagles <je...@gmail.com>.
With six +1 votes (5 binding and 1 non-binding) and zero -1 votes, this
feature branch merge vote passes. I will plan on merging this feature
branch soon. Please hold on off commits for a bit while I finalize this
merge.
Thanks again to all who contributed to this feature branch including
contributing/reviewing patches, testing, voting, and community support.
Regards,
jeagles
On Wed, May 24, 2017 at 2:08 PM, Jonathan Eagles <je...@gmail.com> wrote:
> Ming, in its current form, Tez Shuffle Handler is not setup to handle
> backwards compatibility with MapReduce. This could be easily added, the
> complication is that MapReduce hard-codes its auxiliary shuffle service to
> be named "mapreduce_shuffle". In addition, there are no compelling benefits
> for using the Tez Shuffle Handler with mapreduce except not having to have
> two services running. For a typical setup, yarn should define both
> mapreduce_shuffle and tez_shuffle auxiliary services and run them
> concurrently.
>
> jeagles
>
> On Tue, May 23, 2017 at 7:28 PM, Ming Ma <mi...@twitter.com.invalid>
> wrote:
>
>> +1. Is Tez shuffle handler backward compatible with MR jobs? For a test
>> cluster running Tez and MR jobs, if Tez shuffle handler is backward
>> compatible, we can configure NM to only load Tez shuffle handler.
>> Otherwise, will need to configure NM to load both Tez and MR shuffle
>> handlers.
>>
>> On Tue, May 23, 2017 at 9:16 AM, Eric Badger
>> <eb...@yahoo-inc.com.invalid>
>> wrote:
>>
>> > +1 (non-binding) given the test failure followup jira ([TEZ-3735] Test
>> > failures in TestTaskAttempt and TestAMContainerMap - ASF JIRA)
>> > Eric
>> >
>> > |
>> > | |
>> > [TEZ-3735] Test failures in TestTaskAttempt and TestAMContainerMap - ASF
>> > JIRA
>> > | |
>> >
>> > |
>> >
>> >
>> >
>> >
>> > On Monday, May 22, 2017 3:46 PM, Jason Lowe
>> > <jl...@yahoo-inc.com.INVALID> wrote:
>> >
>> >
>> > +1, looking forward to this functionality being available in an
>> official
>> > release!
>> > Jason
>> >
>> >
>> > On Friday, May 19, 2017 3:16 PM, Jonathan Eagles <jeagles@gmail.com
>> >
>> > wrote:
>> >
>> >
>> > This vote is to merge the Tez Shuffle Handler feature branch (TEZ-3334)
>> > into the master branch. Having done extensive testing and thanks to all
>> the
>> > feedback given, the Tez Shuffle Handler delivers on its promise to
>> reduce
>> > shuffle times for the auto-reduce case and pave the way for more shuffle
>> > types in the future.
>> >
>> > Please view a patch of the proposed merge attached to the TEZ-3334 jira
>> > https://issues.apache.org/jira/browse/TEZ-3334
>> >
>> > As to be the most useful and time efficient, please test and vote early
>> if
>> > you have any misgivings or questions about the readiness of the branch
>> so
>> > as to not to delay the merge process and increase the amount of work on
>> my
>> > part to continue to maintain this branch with master.
>> >
>> > This vote will be open until Wednesday around the same time.
>> >
>> > Here is my +1.
>> >
>> > Regards,
>> > jeagles
>> >
>> >
>> >
>> >
>> >
>> >
>>
>
>
Re: [VOTE] Merge TEZ-3334 into master
Posted by Jonathan Eagles <je...@gmail.com>.
Ming, in its current form, Tez Shuffle Handler is not setup to handle
backwards compatibility with MapReduce. This could be easily added, the
complication is that MapReduce hard-codes its auxiliary shuffle service to
be named "mapreduce_shuffle". In addition, there are no compelling benefits
for using the Tez Shuffle Handler with mapreduce except not having to have
two services running. For a typical setup, yarn should define both
mapreduce_shuffle and tez_shuffle auxiliary services and run them
concurrently.
jeagles
On Tue, May 23, 2017 at 7:28 PM, Ming Ma <mi...@twitter.com.invalid> wrote:
> +1. Is Tez shuffle handler backward compatible with MR jobs? For a test
> cluster running Tez and MR jobs, if Tez shuffle handler is backward
> compatible, we can configure NM to only load Tez shuffle handler.
> Otherwise, will need to configure NM to load both Tez and MR shuffle
> handlers.
>
> On Tue, May 23, 2017 at 9:16 AM, Eric Badger <ebadger@yahoo-inc.com.invalid
> >
> wrote:
>
> > +1 (non-binding) given the test failure followup jira ([TEZ-3735] Test
> > failures in TestTaskAttempt and TestAMContainerMap - ASF JIRA)
> > Eric
> >
> > |
> > | |
> > [TEZ-3735] Test failures in TestTaskAttempt and TestAMContainerMap - ASF
> > JIRA
> > | |
> >
> > |
> >
> >
> >
> >
> > On Monday, May 22, 2017 3:46 PM, Jason Lowe
> > <jl...@yahoo-inc.com.INVALID> wrote:
> >
> >
> > +1, looking forward to this functionality being available in an official
> > release!
> > Jason
> >
> >
> > On Friday, May 19, 2017 3:16 PM, Jonathan Eagles <je...@gmail.com>
> > wrote:
> >
> >
> > This vote is to merge the Tez Shuffle Handler feature branch (TEZ-3334)
> > into the master branch. Having done extensive testing and thanks to all
> the
> > feedback given, the Tez Shuffle Handler delivers on its promise to reduce
> > shuffle times for the auto-reduce case and pave the way for more shuffle
> > types in the future.
> >
> > Please view a patch of the proposed merge attached to the TEZ-3334 jira
> > https://issues.apache.org/jira/browse/TEZ-3334
> >
> > As to be the most useful and time efficient, please test and vote early
> if
> > you have any misgivings or questions about the readiness of the branch so
> > as to not to delay the merge process and increase the amount of work on
> my
> > part to continue to maintain this branch with master.
> >
> > This vote will be open until Wednesday around the same time.
> >
> > Here is my +1.
> >
> > Regards,
> > jeagles
> >
> >
> >
> >
> >
> >
>
Re: [VOTE] Merge TEZ-3334 into master
Posted by Ming Ma <mi...@twitter.com.INVALID>.
+1. Is Tez shuffle handler backward compatible with MR jobs? For a test
cluster running Tez and MR jobs, if Tez shuffle handler is backward
compatible, we can configure NM to only load Tez shuffle handler.
Otherwise, will need to configure NM to load both Tez and MR shuffle
handlers.
On Tue, May 23, 2017 at 9:16 AM, Eric Badger <eb...@yahoo-inc.com.invalid>
wrote:
> +1 (non-binding) given the test failure followup jira ([TEZ-3735] Test
> failures in TestTaskAttempt and TestAMContainerMap - ASF JIRA)
> Eric
>
> |
> | |
> [TEZ-3735] Test failures in TestTaskAttempt and TestAMContainerMap - ASF
> JIRA
> | |
>
> |
>
>
>
>
> On Monday, May 22, 2017 3:46 PM, Jason Lowe
> <jl...@yahoo-inc.com.INVALID> wrote:
>
>
> +1, looking forward to this functionality being available in an official
> release!
> Jason
>
>
> On Friday, May 19, 2017 3:16 PM, Jonathan Eagles <je...@gmail.com>
> wrote:
>
>
> This vote is to merge the Tez Shuffle Handler feature branch (TEZ-3334)
> into the master branch. Having done extensive testing and thanks to all the
> feedback given, the Tez Shuffle Handler delivers on its promise to reduce
> shuffle times for the auto-reduce case and pave the way for more shuffle
> types in the future.
>
> Please view a patch of the proposed merge attached to the TEZ-3334 jira
> https://issues.apache.org/jira/browse/TEZ-3334
>
> As to be the most useful and time efficient, please test and vote early if
> you have any misgivings or questions about the readiness of the branch so
> as to not to delay the merge process and increase the amount of work on my
> part to continue to maintain this branch with master.
>
> This vote will be open until Wednesday around the same time.
>
> Here is my +1.
>
> Regards,
> jeagles
>
>
>
>
>
>
Re: [VOTE] Merge TEZ-3334 into master
Posted by Eric Badger <eb...@yahoo-inc.com.INVALID>.
+1 (non-binding) given the test failure followup jira ([TEZ-3735] Test failures in TestTaskAttempt and TestAMContainerMap - ASF JIRA)
Eric
|
| |
[TEZ-3735] Test failures in TestTaskAttempt and TestAMContainerMap - ASF JIRA
| |
|
On Monday, May 22, 2017 3:46 PM, Jason Lowe <jl...@yahoo-inc.com.INVALID> wrote:
+1, looking forward to this functionality being available in an official release!
Jason
On Friday, May 19, 2017 3:16 PM, Jonathan Eagles <je...@gmail.com> wrote:
This vote is to merge the Tez Shuffle Handler feature branch (TEZ-3334)
into the master branch. Having done extensive testing and thanks to all the
feedback given, the Tez Shuffle Handler delivers on its promise to reduce
shuffle times for the auto-reduce case and pave the way for more shuffle
types in the future.
Please view a patch of the proposed merge attached to the TEZ-3334 jira
https://issues.apache.org/jira/browse/TEZ-3334
As to be the most useful and time efficient, please test and vote early if
you have any misgivings or questions about the readiness of the branch so
as to not to delay the merge process and increase the amount of work on my
part to continue to maintain this branch with master.
This vote will be open until Wednesday around the same time.
Here is my +1.
Regards,
jeagles
Re: [VOTE] Merge TEZ-3334 into master
Posted by Jason Lowe <jl...@yahoo-inc.com.INVALID>.
+1, looking forward to this functionality being available in an official release!
Jason
On Friday, May 19, 2017 3:16 PM, Jonathan Eagles <je...@gmail.com> wrote:
This vote is to merge the Tez Shuffle Handler feature branch (TEZ-3334)
into the master branch. Having done extensive testing and thanks to all the
feedback given, the Tez Shuffle Handler delivers on its promise to reduce
shuffle times for the auto-reduce case and pave the way for more shuffle
types in the future.
Please view a patch of the proposed merge attached to the TEZ-3334 jira
https://issues.apache.org/jira/browse/TEZ-3334
As to be the most useful and time efficient, please test and vote early if
you have any misgivings or questions about the readiness of the branch so
as to not to delay the merge process and increase the amount of work on my
part to continue to maintain this branch with master.
This vote will be open until Wednesday around the same time.
Here is my +1.
Regards,
jeagles