Re: [DISCUSS] Release going forward

[Ming Ma <mi...@twitter.com.INVALID>]

Another factor could be when we want to change the hadoop version that Tez
depends on so that Tez can use functionalities from more recent hadoop
distribution. Some examples are TEZ-3340, TEZ-3341 and TEZ-3342.


On Sat, Jul 9, 2016 at 10:51 AM, Siddharth Seth <ss...@apache.org> wrote:

>
> ---------- Forwarded message ----------
> From: Tsuyoshi Ozawa <oz...@apache.org>
> Date: Mon, Jun 27, 2016 at 9:43 AM
> Subject: Re: [DISCUSS] Release going forward
> To: dev <de...@tez.apache.org>
>
>
> +1(non-binding) to me for iterative release for maintenance release.
> It's useful for end users to get stable version as soon as it fixed.
>
> > For 0.9, I think it will be useful to adopt a model where features are
> added in point releases (the third digit changes). Also, make release more
> often - on a schedule - every 1.5-2 months, when a feature reaches a
> logical conclusion, and as and when required.
>
> About 0.9, my concern is that the point releases with the third digit
> changes can confuse end-users. IMHO, 0.9-alpha-1, 0.9-alpha-2 seems be
> more straight forward to me.
>
> Thanks,
> - Tsuyoshi
>
> On Thu, Jun 23, 2016 at 6:30 PM, Bikas Saha <bi...@apache.org> wrote:
> > Its good to target big features to major release lines. At the same time
> > have a regular cadence of dot releases helps get bits into user hands to
> try
> > out. This way we know a release is coming every 3-6 months and different
> > parts of the major feature could latch onto different releases. E.g. get
> > alpha out for trial in 1 release and then GA out in the next, hopefully
> > benefiting from trials/bug-reports from the alpha.
> >
> > Bikas
> >
> > -----Original Message-----
> > From: Hitesh Shah [mailto:hitesh@apache.org]
> > Sent: Thursday, June 23, 2016 9:12 AM
> > To: dev@tez.apache.org
> > Subject: Re: [DISCUSS] Release going forward
> >
> > +1 on a 0.8.4 release soon to keep a regular cadence on maint lines.
> >
> > A couple of points on the points on features/releases from master/0.9: I
> > think that we might be better off targeting a defined set of features
> for a
> > given release. For example, the new edges being introduced would make a
> good
> > set of features to introduce in 0.9. A completely different set could
> end up
> > being targeted to 0.10. It is likely that these new edges will require
> some
> > changes in the core to support them fully. In that regard, we could
> probably
> > do a release of master initially if there are no other major features
> being
> > targeted in the near future but we may need to branch off for 0.9 to
> allow
> > the new edges to stabilize and allow other communities like Hive to rely
> on
> > a more stable 0.9 maint line as compared to master-based releases which
> may
> > be stable but would make folks a bit nervous if we are also adding other
> > features :)
> >
> > But yes, agreed that we do need to figure out a plan to reduce the
> overall
> > no. of branches to keep active and support them whilst adding these new
> > features. Feature branches will likely help here and we may need to
> consider
> > a concept of branch committers to try and allow non-committers to work
> with
> > committers to progress faster on these branches.
> >
> > thanks
> > -- Hitesh
> >
> >
> >> On Jun 21, 2016, at 6:03 PM, Siddharth Seth <ss...@apache.org> wrote:
> >>
> >> Hi Folks,
> >> I'd like to start a discussion around how we maintain various branches
> >> (features vs bug-fixes), releases from various branches - stability,
> >> release schedules etc.
> >>
> >> This is how I look at the current active branches.
> >> branch-0.7 - Mainly gets bug-fixes.
> >> branch-0.8 - Targets improvements to external service integration
> >> (TEZ-2003), bug fixes master (0.9) - Gets everything for now.
> >>
> >> Feature development is not at the same pace that it has been in the
> past.
> >> The main features being worked on actively at the moment are
> >> TEZ-3209(Handling skewed data without ordering requirements) and
> >> TEZ-2104 (CrossProductEdge), and UI enhancements.
> >>
> >> Downstream projects typically need a released version to integrate
> >> with (projects don't want to depend upon a SNAPSHOT in their main
> > branches).
> >>
> >> For 0.9, I think it will be useful to adopt a model where features are
> >> added in point releases (the third digit changes). Also, make release
> >> more often - on a schedule - every 1.5-2 months, when a feature
> >> reaches a logical conclusion, and as and when required.
> >> The main features for these releases would be TEZ-2104, TEZ-3209, UI,
> >> and additional TEZ-2003 changes. Addition of more features can be
> >> discussed as required - i.e. whether to continue landing them on 0.9
> >> or create the next 'major' version. Bug fixes, would go into all active
> > branches.
> >>
> >> One thing we'll have to be careful about is to not destabilize the
> >> master branch. Half done features are OK - but anything which could
> >> destabilize the branch and needs multiple jiras to stabilize would be
> >> developed in a branch (so that release can continue off of master).
> >>
> >> The release notes which go out with a release can call out the new
> >> features in various 0.9 releases, and there level of stability.
> >>
> >> We can create a release from 0.9 once at least some set of features
> >> have gone in. Meanwhile, external services continues to evolve via 0.8
> >> releases (Will create a new release later this week).
> >>
> >>
> >> Thoughts?
> >>
> >> Thanks,
> >> Sid
> >
> >
> >
>
>

Re: [DISCUSS] Release going forward

[Hitesh Shah <hi...@apache.org>]

I would caution against bumping up the default hadoop version dependency to say something like 2.8.0 in the near future. Apache Hadoop’s .0 releases unfortunately are not stable as they do not undergo production rollout like testing as part of the release process. Additionally, I think commercial distros have a tendency to lag and a lot of users will therefore be on older versions of Hadoop. 

That said, using shims, I don’t see why Tez cannot be deployed against a newer version of Hadoop and potentially make use of newer features ( unless it impacts some core paths such as scheduling related features ). 

— Hitesh 

> On Jul 12, 2016, at 4:23 PM, Ming Ma <mi...@twitter.com.INVALID> wrote:
> 
> Another factor could be when we want to change the hadoop version that Tez
> depends on so that Tez can use functionalities from more recent hadoop
> distribution. Some examples are TEZ-3340, TEZ-3341 and TEZ-3342.
> 
> 
> On Sat, Jul 9, 2016 at 10:51 AM, Siddharth Seth <ss...@apache.org> wrote:
> 
>> 
>> ---------- Forwarded message ----------
>> From: Tsuyoshi Ozawa <oz...@apache.org>
>> Date: Mon, Jun 27, 2016 at 9:43 AM
>> Subject: Re: [DISCUSS] Release going forward
>> To: dev <de...@tez.apache.org>
>> 
>> 
>> +1(non-binding) to me for iterative release for maintenance release.
>> It's useful for end users to get stable version as soon as it fixed.
>> 
>>> For 0.9, I think it will be useful to adopt a model where features are
>> added in point releases (the third digit changes). Also, make release more
>> often - on a schedule - every 1.5-2 months, when a feature reaches a
>> logical conclusion, and as and when required.
>> 
>> About 0.9, my concern is that the point releases with the third digit
>> changes can confuse end-users. IMHO, 0.9-alpha-1, 0.9-alpha-2 seems be
>> more straight forward to me.
>> 
>> Thanks,
>> - Tsuyoshi
>> 
>> On Thu, Jun 23, 2016 at 6:30 PM, Bikas Saha <bi...@apache.org> wrote:
>>> Its good to target big features to major release lines. At the same time
>>> have a regular cadence of dot releases helps get bits into user hands to
>> try
>>> out. This way we know a release is coming every 3-6 months and different
>>> parts of the major feature could latch onto different releases. E.g. get
>>> alpha out for trial in 1 release and then GA out in the next, hopefully
>>> benefiting from trials/bug-reports from the alpha.
>>> 
>>> Bikas
>>> 
>>> -----Original Message-----
>>> From: Hitesh Shah [mailto:hitesh@apache.org]
>>> Sent: Thursday, June 23, 2016 9:12 AM
>>> To: dev@tez.apache.org
>>> Subject: Re: [DISCUSS] Release going forward
>>> 
>>> +1 on a 0.8.4 release soon to keep a regular cadence on maint lines.
>>> 
>>> A couple of points on the points on features/releases from master/0.9: I
>>> think that we might be better off targeting a defined set of features
>> for a
>>> given release. For example, the new edges being introduced would make a
>> good
>>> set of features to introduce in 0.9. A completely different set could
>> end up
>>> being targeted to 0.10. It is likely that these new edges will require
>> some
>>> changes in the core to support them fully. In that regard, we could
>> probably
>>> do a release of master initially if there are no other major features
>> being
>>> targeted in the near future but we may need to branch off for 0.9 to
>> allow
>>> the new edges to stabilize and allow other communities like Hive to rely
>> on
>>> a more stable 0.9 maint line as compared to master-based releases which
>> may
>>> be stable but would make folks a bit nervous if we are also adding other
>>> features :)
>>> 
>>> But yes, agreed that we do need to figure out a plan to reduce the
>> overall
>>> no. of branches to keep active and support them whilst adding these new
>>> features. Feature branches will likely help here and we may need to
>> consider
>>> a concept of branch committers to try and allow non-committers to work
>> with
>>> committers to progress faster on these branches.
>>> 
>>> thanks
>>> -- Hitesh
>>> 
>>> 
>>>> On Jun 21, 2016, at 6:03 PM, Siddharth Seth <ss...@apache.org> wrote:
>>>> 
>>>> Hi Folks,
>>>> I'd like to start a discussion around how we maintain various branches
>>>> (features vs bug-fixes), releases from various branches - stability,
>>>> release schedules etc.
>>>> 
>>>> This is how I look at the current active branches.
>>>> branch-0.7 - Mainly gets bug-fixes.
>>>> branch-0.8 - Targets improvements to external service integration
>>>> (TEZ-2003), bug fixes master (0.9) - Gets everything for now.
>>>> 
>>>> Feature development is not at the same pace that it has been in the
>> past.
>>>> The main features being worked on actively at the moment are
>>>> TEZ-3209(Handling skewed data without ordering requirements) and
>>>> TEZ-2104 (CrossProductEdge), and UI enhancements.
>>>> 
>>>> Downstream projects typically need a released version to integrate
>>>> with (projects don't want to depend upon a SNAPSHOT in their main
>>> branches).
>>>> 
>>>> For 0.9, I think it will be useful to adopt a model where features are
>>>> added in point releases (the third digit changes). Also, make release
>>>> more often - on a schedule - every 1.5-2 months, when a feature
>>>> reaches a logical conclusion, and as and when required.
>>>> The main features for these releases would be TEZ-2104, TEZ-3209, UI,
>>>> and additional TEZ-2003 changes. Addition of more features can be
>>>> discussed as required - i.e. whether to continue landing them on 0.9
>>>> or create the next 'major' version. Bug fixes, would go into all active
>>> branches.
>>>> 
>>>> One thing we'll have to be careful about is to not destabilize the
>>>> master branch. Half done features are OK - but anything which could
>>>> destabilize the branch and needs multiple jiras to stabilize would be
>>>> developed in a branch (so that release can continue off of master).
>>>> 
>>>> The release notes which go out with a release can call out the new
>>>> features in various 0.9 releases, and there level of stability.
>>>> 
>>>> We can create a release from 0.9 once at least some set of features
>>>> have gone in. Meanwhile, external services continues to evolve via 0.8
>>>> releases (Will create a new release later this week).
>>>> 
>>>> 
>>>> Thoughts?
>>>> 
>>>> Thanks,
>>>> Sid
>>> 
>>> 
>>> 
>> 
>>