[DISCUSS] Aligning master with Hadoop 3 and create separate 0.9.x line

[Eric Wohlstadter <ew...@hortonworks.com>]

Hi all,
I’d like to propose that we move towards aligning the Tez master branch with support for Hadoop 3+ only.
 A separate branch and distribution (e.g. on Maven Central) would be created to maintain the 0.9.x line with support for Hadoop 2.7+.

This will help ensure that Tez can continue to move forward with other progress in the greater Hadoop community.
Since Hadoop 3 is not backward compatible with Hadoop 2, my opinion is that it is too difficult for Tez to maintain such backward compatibility


  *   Tez master branch would support only Hadoop 3+ moving forward
  *   Bug fixes would be required to be pushed to both to master and the 0.9.x line
  *   Major feature or performance improvements would be required to be pushed to both master and the 0.9.x line (unless they require Hadoop 3+)
  *   Minor feature or performance improvements can be pushed only to master
  *   A new release with Hadoop 3+ only support would be placed on high priority (possibly 0.10?)
     *   At a minimum the issues under TEZ-3903 would be required

Please help to provide any feedback or comments about this unofficial proposal.
This is not an official vote but it would help to get people’s thoughts/questions or unofficial (+1, -1).

Re: [DISCUSS] Aligning master with Hadoop 3 and create separate 0.9.x line

[Kuhu Shukla <ks...@oath.com.INVALID>]

+1 for the proposed plan.

On Mon, Mar 26, 2018 at 10:09 AM, Jason Lowe <jl...@oath.com.invalid> wrote:

> +1 for incrementing the required Hadoop version from 2.7 as long as we
> continue to push bugfixes to the 0.9 line for a while.  We currently
> have a "hadoop28" profile in Tez which is mostly compatible with
> Hadoop 3.x, but it does not get much testing.  There is no release
> vehicle for it, and it does not even get tested from the precommit
> build.  Promoting this or a 3.x profile to the main build is the most
> straightforward way to get it tested and released in an
> easy-to-consume form.
>
> This does mean we would need to maintain two release lines for a
> while, at least until users and downstream projects migrate away from
> Hadoop 2.7.  We've done two lines before (even three, if we consider
> the days of 0.9.x, 0.8.x, and 0.7.x all co-existing), and in this case
> I think the cost of maintaining those two lines is worth it to move
> the project forward as the stack migrates to Hadoop 3.x.
>
> Jason
>
>
>
> On Thu, Mar 22, 2018 at 7:16 PM, Eric Wohlstadter
> <ew...@hortonworks.com> wrote:
> > Hi all,
> > I’d like to propose that we move towards aligning the Tez master branch
> with support for Hadoop 3+ only.
> >  A separate branch and distribution (e.g. on Maven Central) would be
> created to maintain the 0.9.x line with support for Hadoop 2.7+.
> >
> > This will help ensure that Tez can continue to move forward with other
> progress in the greater Hadoop community.
> > Since Hadoop 3 is not backward compatible with Hadoop 2, my opinion is
> that it is too difficult for Tez to maintain such backward compatibility
> >
> >
> >   *   Tez master branch would support only Hadoop 3+ moving forward
> >   *   Bug fixes would be required to be pushed to both to master and the
> 0.9.x line
> >   *   Major feature or performance improvements would be required to be
> pushed to both master and the 0.9.x line (unless they require Hadoop 3+)
> >   *   Minor feature or performance improvements can be pushed only to
> master
> >   *   A new release with Hadoop 3+ only support would be placed on high
> priority (possibly 0.10?)
> >      *   At a minimum the issues under TEZ-3903 would be required
> >
> > Please help to provide any feedback or comments about this unofficial
> proposal.
> > This is not an official vote but it would help to get people’s
> thoughts/questions or unofficial (+1, -1).
> >
>

Re: [DISCUSS] Aligning master with Hadoop 3 and create separate 0.9.x line

[Jason Lowe <jl...@oath.com.INVALID>]

+1 for incrementing the required Hadoop version from 2.7 as long as we
continue to push bugfixes to the 0.9 line for a while.  We currently
have a "hadoop28" profile in Tez which is mostly compatible with
Hadoop 3.x, but it does not get much testing.  There is no release
vehicle for it, and it does not even get tested from the precommit
build.  Promoting this or a 3.x profile to the main build is the most
straightforward way to get it tested and released in an
easy-to-consume form.

This does mean we would need to maintain two release lines for a
while, at least until users and downstream projects migrate away from
Hadoop 2.7.  We've done two lines before (even three, if we consider
the days of 0.9.x, 0.8.x, and 0.7.x all co-existing), and in this case
I think the cost of maintaining those two lines is worth it to move
the project forward as the stack migrates to Hadoop 3.x.

Jason



On Thu, Mar 22, 2018 at 7:16 PM, Eric Wohlstadter
<ew...@hortonworks.com> wrote:
> Hi all,
> I’d like to propose that we move towards aligning the Tez master branch with support for Hadoop 3+ only.
>  A separate branch and distribution (e.g. on Maven Central) would be created to maintain the 0.9.x line with support for Hadoop 2.7+.
>
> This will help ensure that Tez can continue to move forward with other progress in the greater Hadoop community.
> Since Hadoop 3 is not backward compatible with Hadoop 2, my opinion is that it is too difficult for Tez to maintain such backward compatibility
>
>
>   *   Tez master branch would support only Hadoop 3+ moving forward
>   *   Bug fixes would be required to be pushed to both to master and the 0.9.x line
>   *   Major feature or performance improvements would be required to be pushed to both master and the 0.9.x line (unless they require Hadoop 3+)
>   *   Minor feature or performance improvements can be pushed only to master
>   *   A new release with Hadoop 3+ only support would be placed on high priority (possibly 0.10?)
>      *   At a minimum the issues under TEZ-3903 would be required
>
> Please help to provide any feedback or comments about this unofficial proposal.
> This is not an official vote but it would help to get people’s thoughts/questions or unofficial (+1, -1).
>