[DISCUSS] : Apache Falcon minor release schedule

[Balu Vellanki Bala <bv...@hortonworks.com>]

Hello Team,

We had a discussion on the scope and schedule of Apache Falcon minor releases during the Falcon bi-weekly meeting. There were two approaches suggested. I request you to provide your feedback/opinion on what is the preferred way.

Approach 1 : Minor release should be feature based.
In this approach, the release manager will coordinate with the Falcon community and come up with an short wish-list of features that should go into next release. The list should be achievable in the timelines proposed. Once the list is decided upon, the release will happen only after the features are complete (including full testing).  The advantage is that minor releases will be feature complete and stable. Community will spend less time on debugging incomplete features.  The disadvantage is that the release timeline becomes unpredictable due to unforeseen feature delays.

Approach 2 : Minor release should be time bound.
In this approach, minor releases will be done on a regular time interval proposed to be once a month. Every month, if we have a single complete feature committed to Falcon, a new branch will be cut and a minor release will be made. Incomplete features can go into the release, but they will not be advertised. The advantage is that falcon will have faster and predictable release cycles. The disadvantage is that there could be incomplete features going into Falcon, leading to customers trying out and struggling with these features.

Thank you
Balu Vellanki

Re: [DISCUSS] : Apache Falcon minor release schedule

[Venkat Ranganathan <vr...@hortonworks.com>]

Yes, you are right - development and vibrancy is indeed more now thanks to more developers and contributors.   I think 6-8 weeks delivery is a good goal to have in that it brings the immediacy of integrated release of verified fixes and features but it will be a few more releases to iron out deployment mode issues that are still there to validate - for example kerberos enabled cluster, NN-HA, RM-HA (with/without WPR), Oozie-HA and with different Hive and Hadoop versions we support.

Thanks

Venkat



On 4/8/16, 6:15 AM, "Srikanth Sundarrajan" <sr...@hotmail.com> wrote:

>The last few releases appeared to be fast enough due to existence of a reasonably strong test suite to validate the builds. Are there still gaps on that front ? I would like to believe given the vibrant and active development in our project, we should have enough to ship every 6-8 weeks. Also critical and irritant bug fixes will also make it to out sooner.
>
>Regards
>Srikanth Sundarrajan
>
>> Subject: Re: [DISCUSS] : Apache Falcon minor release schedule
>> From: vranganathan@hortonworks.com
>> To: dev@falcon.apache.org
>> Date: Thu, 7 Apr 2016 15:06:16 +0000
>> 
>> While it is good to have frequent releases for maintaining the vibrancy of the product (it also makes features time bound and introduces some level of discipline),  we should have some threshold on the content update lest the releases become less useful.   I think a 6-8 week release cadence is aggressive as we have evidenced (and for a variety of reasons - features taking time to mature, integration tests with other products /configs by QE finds blockers late in the cycle).   In fact, running QE for a frequent release will be difficult across various deployment models a typical customer will expect - unless we decide to forego that or have a two task model - basic releases but  not all release features are immediately usable in all deployment scenarios.
>> 
>> Also we need to formulate, what it means to be a minor release (for example, a strictly bug fix release where no product dependency is affected by dependent product refresh, or feature content is also?  As we have seen, we have not changed some of the dependent product versions for a long time (HiveDR is on a separate addons partly because we did  not want to break users with dependency on existing Hive features).  
>> 
>> Venkat
>> 
>> 
>> 
>> 
>> 
>> 
>> On 4/7/16, 3:00 AM, "Ajay Yadav" <aj...@gmail.com> wrote:
>> 
>> >After observing both formats across several releases I have also become a
>> >strong supporter of Approach #2.
>> >Releasing early has improved the project in many ways. Earlier we were
>> >making releases after several months and we didn't invest time in improving
>> >the release and testing process. Early release forced us to invest time in
>> >these tasks and we have seen continuously better release candidates across
>> >last several releases.
>> >
>> >As for incomplete features going in, as Srikanth said this is a non-issue
>> >as long as we don't advertise those features being available and in
>> >practice it has been working out very well for us. Unlike approach 1 it
>> >encourages us to follow the Apache way by putting in small changes in
>> >community instead of dumping large features at the end and pushing them in
>> >to make in a release. It also helps us to get quick feedback on changes
>> >instead of making assumptions and making large changes based on those
>> >assumptions.
>> >
>> >Approach 1 also blocks certain changes if they are not supposed to be in
>> >the release or will affect a feature marked for the release. Though
>> >infrequent, in practice this hampers the community contribution in some
>> >cases.
>> >
>> >
>> >
>> >On Thu, Apr 7, 2016 at 8:51 AM, Srikanth Sundarrajan <sr...@hotmail.com>
>> >wrote:
>> >
>> >> I am personally a strong advocate of Approach #2. I believe "release early
>> >> release often" is the right model for open source projects. There are
>> >> numerous examples of where this is successfully adopted and community
>> >> benefiting from. If the overheads of a releases are reduced and managed
>> >> well, this ought to be a non-issue. While there are projects that do
>> >> releases one or even twice in a month, we can certainly do with a 6-8 week
>> >> release cycle.
>> >>
>> >> >> The disadvantage is that there could be incomplete features going into
>> >> Falcon, leading to customers trying out and struggling with these
>> >> features.
>> >>
>> >> This is only an issue if the feature is prematurely advertised as being
>> >> available, as long as new features don't break existing functionality.
>> >> Luckily we have a reasonable test suite.
>> >>
>> >> Regards
>> >> Srikanth Sundarrajan
>> >>
>> >>
>> >> > Subject: [DISCUSS] : Apache Falcon minor release schedule
>> >> > From: bvellanki@hortonworks.com
>> >> > To: dev@falcon.apache.org
>> >> > Date: Wed, 6 Apr 2016 18:39:11 +0000
>> >> >
>> >> > Hello Team,
>> >> >
>> >> > We had a discussion on the scope and schedule of Apache Falcon minor
>> >> releases during the Falcon bi-weekly meeting. There were two approaches
>> >> suggested. I request you to provide your feedback/opinion on what is the
>> >> preferred way.
>> >> >
>> >> > Approach 1 : Minor release should be feature based.
>> >> > In this approach, the release manager will coordinate with the Falcon
>> >> community and come up with an short wish-list of features that should go
>> >> into next release. The list should be achievable in the timelines proposed.
>> >> Once the list is decided upon, the release will happen only after the
>> >> features are complete (including full testing).  The advantage is that
>> >> minor releases will be feature complete and stable. Community will spend
>> >> less time on debugging incomplete features.  The disadvantage is that the
>> >> release timeline becomes unpredictable due to unforeseen feature delays.
>> >> >
>> >> > Approach 2 : Minor release should be time bound.
>> >> > In this approach, minor releases will be done on a regular time interval
>> >> proposed to be once a month. Every month, if we have a single complete
>> >> feature committed to Falcon, a new branch will be cut and a minor release
>> >> will be made. Incomplete features can go into the release, but they will
>> >> not be advertised. The advantage is that falcon will have faster and
>> >> predictable release cycles. The disadvantage is that there could be
>> >> incomplete features going into Falcon, leading to customers trying out and
>> >> struggling with these features.
>> >> >
>> >> > Thank you
>> >> > Balu Vellanki
>> >>
>> >>
> 		 	   		  

RE: [DISCUSS] : Apache Falcon minor release schedule

[Srikanth Sundarrajan <sr...@hotmail.com>]

The last few releases appeared to be fast enough due to existence of a reasonably strong test suite to validate the builds. Are there still gaps on that front ? I would like to believe given the vibrant and active development in our project, we should have enough to ship every 6-8 weeks. Also critical and irritant bug fixes will also make it to out sooner.

Regards
Srikanth Sundarrajan

> Subject: Re: [DISCUSS] : Apache Falcon minor release schedule
> From: vranganathan@hortonworks.com
> To: dev@falcon.apache.org
> Date: Thu, 7 Apr 2016 15:06:16 +0000
> 
> While it is good to have frequent releases for maintaining the vibrancy of the product (it also makes features time bound and introduces some level of discipline),  we should have some threshold on the content update lest the releases become less useful.   I think a 6-8 week release cadence is aggressive as we have evidenced (and for a variety of reasons - features taking time to mature, integration tests with other products /configs by QE finds blockers late in the cycle).   In fact, running QE for a frequent release will be difficult across various deployment models a typical customer will expect - unless we decide to forego that or have a two task model - basic releases but  not all release features are immediately usable in all deployment scenarios.
> 
> Also we need to formulate, what it means to be a minor release (for example, a strictly bug fix release where no product dependency is affected by dependent product refresh, or feature content is also?  As we have seen, we have not changed some of the dependent product versions for a long time (HiveDR is on a separate addons partly because we did  not want to break users with dependency on existing Hive features).  
> 
> Venkat
> 
> 
> 
> 
> 
> 
> On 4/7/16, 3:00 AM, "Ajay Yadav" <aj...@gmail.com> wrote:
> 
> >After observing both formats across several releases I have also become a
> >strong supporter of Approach #2.
> >Releasing early has improved the project in many ways. Earlier we were
> >making releases after several months and we didn't invest time in improving
> >the release and testing process. Early release forced us to invest time in
> >these tasks and we have seen continuously better release candidates across
> >last several releases.
> >
> >As for incomplete features going in, as Srikanth said this is a non-issue
> >as long as we don't advertise those features being available and in
> >practice it has been working out very well for us. Unlike approach 1 it
> >encourages us to follow the Apache way by putting in small changes in
> >community instead of dumping large features at the end and pushing them in
> >to make in a release. It also helps us to get quick feedback on changes
> >instead of making assumptions and making large changes based on those
> >assumptions.
> >
> >Approach 1 also blocks certain changes if they are not supposed to be in
> >the release or will affect a feature marked for the release. Though
> >infrequent, in practice this hampers the community contribution in some
> >cases.
> >
> >
> >
> >On Thu, Apr 7, 2016 at 8:51 AM, Srikanth Sundarrajan <sr...@hotmail.com>
> >wrote:
> >
> >> I am personally a strong advocate of Approach #2. I believe "release early
> >> release often" is the right model for open source projects. There are
> >> numerous examples of where this is successfully adopted and community
> >> benefiting from. If the overheads of a releases are reduced and managed
> >> well, this ought to be a non-issue. While there are projects that do
> >> releases one or even twice in a month, we can certainly do with a 6-8 week
> >> release cycle.
> >>
> >> >> The disadvantage is that there could be incomplete features going into
> >> Falcon, leading to customers trying out and struggling with these
> >> features.
> >>
> >> This is only an issue if the feature is prematurely advertised as being
> >> available, as long as new features don't break existing functionality.
> >> Luckily we have a reasonable test suite.
> >>
> >> Regards
> >> Srikanth Sundarrajan
> >>
> >>
> >> > Subject: [DISCUSS] : Apache Falcon minor release schedule
> >> > From: bvellanki@hortonworks.com
> >> > To: dev@falcon.apache.org
> >> > Date: Wed, 6 Apr 2016 18:39:11 +0000
> >> >
> >> > Hello Team,
> >> >
> >> > We had a discussion on the scope and schedule of Apache Falcon minor
> >> releases during the Falcon bi-weekly meeting. There were two approaches
> >> suggested. I request you to provide your feedback/opinion on what is the
> >> preferred way.
> >> >
> >> > Approach 1 : Minor release should be feature based.
> >> > In this approach, the release manager will coordinate with the Falcon
> >> community and come up with an short wish-list of features that should go
> >> into next release. The list should be achievable in the timelines proposed.
> >> Once the list is decided upon, the release will happen only after the
> >> features are complete (including full testing).  The advantage is that
> >> minor releases will be feature complete and stable. Community will spend
> >> less time on debugging incomplete features.  The disadvantage is that the
> >> release timeline becomes unpredictable due to unforeseen feature delays.
> >> >
> >> > Approach 2 : Minor release should be time bound.
> >> > In this approach, minor releases will be done on a regular time interval
> >> proposed to be once a month. Every month, if we have a single complete
> >> feature committed to Falcon, a new branch will be cut and a minor release
> >> will be made. Incomplete features can go into the release, but they will
> >> not be advertised. The advantage is that falcon will have faster and
> >> predictable release cycles. The disadvantage is that there could be
> >> incomplete features going into Falcon, leading to customers trying out and
> >> struggling with these features.
> >> >
> >> > Thank you
> >> > Balu Vellanki
> >>
> >>
 		 	   		  

Re: [DISCUSS] : Apache Falcon minor release schedule

[Venkat Ranganathan <vr...@hortonworks.com>]

While it is good to have frequent releases for maintaining the vibrancy of the product (it also makes features time bound and introduces some level of discipline),  we should have some threshold on the content update lest the releases become less useful.   I think a 6-8 week release cadence is aggressive as we have evidenced (and for a variety of reasons - features taking time to mature, integration tests with other products /configs by QE finds blockers late in the cycle).   In fact, running QE for a frequent release will be difficult across various deployment models a typical customer will expect - unless we decide to forego that or have a two task model - basic releases but  not all release features are immediately usable in all deployment scenarios.

Also we need to formulate, what it means to be a minor release (for example, a strictly bug fix release where no product dependency is affected by dependent product refresh, or feature content is also?  As we have seen, we have not changed some of the dependent product versions for a long time (HiveDR is on a separate addons partly because we did  not want to break users with dependency on existing Hive features).  

Venkat






On 4/7/16, 3:00 AM, "Ajay Yadav" <aj...@gmail.com> wrote:

>After observing both formats across several releases I have also become a
>strong supporter of Approach #2.
>Releasing early has improved the project in many ways. Earlier we were
>making releases after several months and we didn't invest time in improving
>the release and testing process. Early release forced us to invest time in
>these tasks and we have seen continuously better release candidates across
>last several releases.
>
>As for incomplete features going in, as Srikanth said this is a non-issue
>as long as we don't advertise those features being available and in
>practice it has been working out very well for us. Unlike approach 1 it
>encourages us to follow the Apache way by putting in small changes in
>community instead of dumping large features at the end and pushing them in
>to make in a release. It also helps us to get quick feedback on changes
>instead of making assumptions and making large changes based on those
>assumptions.
>
>Approach 1 also blocks certain changes if they are not supposed to be in
>the release or will affect a feature marked for the release. Though
>infrequent, in practice this hampers the community contribution in some
>cases.
>
>
>
>On Thu, Apr 7, 2016 at 8:51 AM, Srikanth Sundarrajan <sr...@hotmail.com>
>wrote:
>
>> I am personally a strong advocate of Approach #2. I believe "release early
>> release often" is the right model for open source projects. There are
>> numerous examples of where this is successfully adopted and community
>> benefiting from. If the overheads of a releases are reduced and managed
>> well, this ought to be a non-issue. While there are projects that do
>> releases one or even twice in a month, we can certainly do with a 6-8 week
>> release cycle.
>>
>> >> The disadvantage is that there could be incomplete features going into
>> Falcon, leading to customers trying out and struggling with these
>> features.
>>
>> This is only an issue if the feature is prematurely advertised as being
>> available, as long as new features don't break existing functionality.
>> Luckily we have a reasonable test suite.
>>
>> Regards
>> Srikanth Sundarrajan
>>
>>
>> > Subject: [DISCUSS] : Apache Falcon minor release schedule
>> > From: bvellanki@hortonworks.com
>> > To: dev@falcon.apache.org
>> > Date: Wed, 6 Apr 2016 18:39:11 +0000
>> >
>> > Hello Team,
>> >
>> > We had a discussion on the scope and schedule of Apache Falcon minor
>> releases during the Falcon bi-weekly meeting. There were two approaches
>> suggested. I request you to provide your feedback/opinion on what is the
>> preferred way.
>> >
>> > Approach 1 : Minor release should be feature based.
>> > In this approach, the release manager will coordinate with the Falcon
>> community and come up with an short wish-list of features that should go
>> into next release. The list should be achievable in the timelines proposed.
>> Once the list is decided upon, the release will happen only after the
>> features are complete (including full testing).  The advantage is that
>> minor releases will be feature complete and stable. Community will spend
>> less time on debugging incomplete features.  The disadvantage is that the
>> release timeline becomes unpredictable due to unforeseen feature delays.
>> >
>> > Approach 2 : Minor release should be time bound.
>> > In this approach, minor releases will be done on a regular time interval
>> proposed to be once a month. Every month, if we have a single complete
>> feature committed to Falcon, a new branch will be cut and a minor release
>> will be made. Incomplete features can go into the release, but they will
>> not be advertised. The advantage is that falcon will have faster and
>> predictable release cycles. The disadvantage is that there could be
>> incomplete features going into Falcon, leading to customers trying out and
>> struggling with these features.
>> >
>> > Thank you
>> > Balu Vellanki
>>
>>

Re: [DISCUSS] : Apache Falcon minor release schedule

[Ajay Yadav <aj...@gmail.com>]

After observing both formats across several releases I have also become a
strong supporter of Approach #2.
Releasing early has improved the project in many ways. Earlier we were
making releases after several months and we didn't invest time in improving
the release and testing process. Early release forced us to invest time in
these tasks and we have seen continuously better release candidates across
last several releases.

As for incomplete features going in, as Srikanth said this is a non-issue
as long as we don't advertise those features being available and in
practice it has been working out very well for us. Unlike approach 1 it
encourages us to follow the Apache way by putting in small changes in
community instead of dumping large features at the end and pushing them in
to make in a release. It also helps us to get quick feedback on changes
instead of making assumptions and making large changes based on those
assumptions.

Approach 1 also blocks certain changes if they are not supposed to be in
the release or will affect a feature marked for the release. Though
infrequent, in practice this hampers the community contribution in some
cases.



On Thu, Apr 7, 2016 at 8:51 AM, Srikanth Sundarrajan <sr...@hotmail.com>
wrote:

> I am personally a strong advocate of Approach #2. I believe "release early
> release often" is the right model for open source projects. There are
> numerous examples of where this is successfully adopted and community
> benefiting from. If the overheads of a releases are reduced and managed
> well, this ought to be a non-issue. While there are projects that do
> releases one or even twice in a month, we can certainly do with a 6-8 week
> release cycle.
>
> >> The disadvantage is that there could be incomplete features going into
> Falcon, leading to customers trying out and struggling with these
> features.
>
> This is only an issue if the feature is prematurely advertised as being
> available, as long as new features don't break existing functionality.
> Luckily we have a reasonable test suite.
>
> Regards
> Srikanth Sundarrajan
>
>
> > Subject: [DISCUSS] : Apache Falcon minor release schedule
> > From: bvellanki@hortonworks.com
> > To: dev@falcon.apache.org
> > Date: Wed, 6 Apr 2016 18:39:11 +0000
> >
> > Hello Team,
> >
> > We had a discussion on the scope and schedule of Apache Falcon minor
> releases during the Falcon bi-weekly meeting. There were two approaches
> suggested. I request you to provide your feedback/opinion on what is the
> preferred way.
> >
> > Approach 1 : Minor release should be feature based.
> > In this approach, the release manager will coordinate with the Falcon
> community and come up with an short wish-list of features that should go
> into next release. The list should be achievable in the timelines proposed.
> Once the list is decided upon, the release will happen only after the
> features are complete (including full testing).  The advantage is that
> minor releases will be feature complete and stable. Community will spend
> less time on debugging incomplete features.  The disadvantage is that the
> release timeline becomes unpredictable due to unforeseen feature delays.
> >
> > Approach 2 : Minor release should be time bound.
> > In this approach, minor releases will be done on a regular time interval
> proposed to be once a month. Every month, if we have a single complete
> feature committed to Falcon, a new branch will be cut and a minor release
> will be made. Incomplete features can go into the release, but they will
> not be advertised. The advantage is that falcon will have faster and
> predictable release cycles. The disadvantage is that there could be
> incomplete features going into Falcon, leading to customers trying out and
> struggling with these features.
> >
> > Thank you
> > Balu Vellanki
>
>

RE: [DISCUSS] : Apache Falcon minor release schedule

[Srikanth Sundarrajan <sr...@hotmail.com>]

I am personally a strong advocate of Approach #2. I believe "release early release often" is the right model for open source projects. There are numerous examples of where this is successfully adopted and community benefiting from. If the overheads of a releases are reduced and managed well, this ought to be a non-issue. While there are projects that do releases one or even twice in a month, we can certainly do with a 6-8 week release cycle. 

>> The disadvantage is that there could be incomplete features going into 
Falcon, leading to customers trying out and struggling with these 
features.

This is only an issue if the feature is prematurely advertised as being available, as long as new features don't break existing functionality. Luckily we have a reasonable test suite.

Regards
Srikanth Sundarrajan


> Subject: [DISCUSS] : Apache Falcon minor release schedule
> From: bvellanki@hortonworks.com
> To: dev@falcon.apache.org
> Date: Wed, 6 Apr 2016 18:39:11 +0000
> 
> Hello Team,
> 
> We had a discussion on the scope and schedule of Apache Falcon minor releases during the Falcon bi-weekly meeting. There were two approaches suggested. I request you to provide your feedback/opinion on what is the preferred way.
> 
> Approach 1 : Minor release should be feature based.
> In this approach, the release manager will coordinate with the Falcon community and come up with an short wish-list of features that should go into next release. The list should be achievable in the timelines proposed. Once the list is decided upon, the release will happen only after the features are complete (including full testing).  The advantage is that minor releases will be feature complete and stable. Community will spend less time on debugging incomplete features.  The disadvantage is that the release timeline becomes unpredictable due to unforeseen feature delays.
> 
> Approach 2 : Minor release should be time bound.
> In this approach, minor releases will be done on a regular time interval proposed to be once a month. Every month, if we have a single complete feature committed to Falcon, a new branch will be cut and a minor release will be made. Incomplete features can go into the release, but they will not be advertised. The advantage is that falcon will have faster and predictable release cycles. The disadvantage is that there could be incomplete features going into Falcon, leading to customers trying out and struggling with these features.
> 
> Thank you
> Balu Vellanki