You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@hudi.apache.org by Vinoth Chandar <vi...@apache.org> on 2021/12/13 18:01:02 UTC
Regular minor/patch releases
Hi all,
In the past we had plans for minor releases [1], but invariably we end up
doing major ones, which also deliver the bug fixes.
The reason was the cost involved in doing a release. We have made some good
progress towards regression/integration test, which prompts me to revive
this.
What does everyone think about a monthly bugfix release on the last
major/minor version. (not on every major release, we still don't have
enough contributors to pull that off IMO). So we would be trying to do a
0.10.1 early jan for e.g, in this model?
[1] https://cwiki.apache.org/confluence/display/HUDI/Release+Management
Thanks
Vinoth
Re: Regular minor/patch releases
Posted by Raymond Xu <xu...@gmail.com>.
+1 to patch release as needed
Question: for a minor release, do we cherry-pick commits needed from master
or only land patches until the minor release is cut? The former can be
quite troublesome in resolving conflicts.
--
Best,
Raymond
On Tue, Dec 14, 2021 at 5:07 PM Rubens Rodrigues <ru...@gmail.com>
wrote:
> +1
>
> I think it is really good for users
>
> Em ter., 14 de dez. de 2021 22:01, Y Ethan Guo <et...@gmail.com>
> escreveu:
>
> > +1 on packing bug fixes (at best effort) to minor releases.
> >
> > On Mon, Dec 13, 2021 at 12:06 PM Sivabalan <n....@gmail.com> wrote:
> >
> > > +1 in general. but yeah, not sure if we have resources to do this for
> > > every major release.
> > >
> > > On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <vi...@apache.org>
> > wrote:
> > >
> > >> Hi all,
> > >>
> > >> In the past we had plans for minor releases [1], but invariably we end
> > up
> > >> doing major ones, which also deliver the bug fixes.
> > >>
> > >> The reason was the cost involved in doing a release. We have made some
> > >> good
> > >> progress towards regression/integration test, which prompts me to
> revive
> > >> this.
> > >>
> > >> What does everyone think about a monthly bugfix release on the last
> > >> major/minor version. (not on every major release, we still don't have
> > >> enough contributors to pull that off IMO). So we would be trying to
> do a
> > >> 0.10.1 early jan for e.g, in this model?
> > >>
> > >> [1]
> https://cwiki.apache.org/confluence/display/HUDI/Release+Management
> > >>
> > >> Thanks
> > >> Vinoth
> > >>
> > >
> > >
> > > --
> > > Regards,
> > > -Sivabalan
> > >
> >
>
Re: Regular minor/patch releases
Posted by Raymond Xu <xu...@gmail.com>.
+1 to patch release as needed
Question: for a minor release, do we cherry-pick commits needed from master
or only land patches until the minor release is cut? The former can be
quite troublesome in resolving conflicts.
--
Best,
Raymond
On Tue, Dec 14, 2021 at 5:07 PM Rubens Rodrigues <ru...@gmail.com>
wrote:
> +1
>
> I think it is really good for users
>
> Em ter., 14 de dez. de 2021 22:01, Y Ethan Guo <et...@gmail.com>
> escreveu:
>
> > +1 on packing bug fixes (at best effort) to minor releases.
> >
> > On Mon, Dec 13, 2021 at 12:06 PM Sivabalan <n....@gmail.com> wrote:
> >
> > > +1 in general. but yeah, not sure if we have resources to do this for
> > > every major release.
> > >
> > > On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <vi...@apache.org>
> > wrote:
> > >
> > >> Hi all,
> > >>
> > >> In the past we had plans for minor releases [1], but invariably we end
> > up
> > >> doing major ones, which also deliver the bug fixes.
> > >>
> > >> The reason was the cost involved in doing a release. We have made some
> > >> good
> > >> progress towards regression/integration test, which prompts me to
> revive
> > >> this.
> > >>
> > >> What does everyone think about a monthly bugfix release on the last
> > >> major/minor version. (not on every major release, we still don't have
> > >> enough contributors to pull that off IMO). So we would be trying to
> do a
> > >> 0.10.1 early jan for e.g, in this model?
> > >>
> > >> [1]
> https://cwiki.apache.org/confluence/display/HUDI/Release+Management
> > >>
> > >> Thanks
> > >> Vinoth
> > >>
> > >
> > >
> > > --
> > > Regards,
> > > -Sivabalan
> > >
> >
>
Re: Regular minor/patch releases
Posted by Rubens Rodrigues <ru...@gmail.com>.
+1
I think it is really good for users
Em ter., 14 de dez. de 2021 22:01, Y Ethan Guo <et...@gmail.com>
escreveu:
> +1 on packing bug fixes (at best effort) to minor releases.
>
> On Mon, Dec 13, 2021 at 12:06 PM Sivabalan <n....@gmail.com> wrote:
>
> > +1 in general. but yeah, not sure if we have resources to do this for
> > every major release.
> >
> > On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <vi...@apache.org>
> wrote:
> >
> >> Hi all,
> >>
> >> In the past we had plans for minor releases [1], but invariably we end
> up
> >> doing major ones, which also deliver the bug fixes.
> >>
> >> The reason was the cost involved in doing a release. We have made some
> >> good
> >> progress towards regression/integration test, which prompts me to revive
> >> this.
> >>
> >> What does everyone think about a monthly bugfix release on the last
> >> major/minor version. (not on every major release, we still don't have
> >> enough contributors to pull that off IMO). So we would be trying to do a
> >> 0.10.1 early jan for e.g, in this model?
> >>
> >> [1] https://cwiki.apache.org/confluence/display/HUDI/Release+Management
> >>
> >> Thanks
> >> Vinoth
> >>
> >
> >
> > --
> > Regards,
> > -Sivabalan
> >
>
Re: Regular minor/patch releases
Posted by Y Ethan Guo <et...@gmail.com>.
+1 on packing bug fixes (at best effort) to minor releases.
On Mon, Dec 13, 2021 at 12:06 PM Sivabalan <n....@gmail.com> wrote:
> +1 in general. but yeah, not sure if we have resources to do this for
> every major release.
>
> On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <vi...@apache.org> wrote:
>
>> Hi all,
>>
>> In the past we had plans for minor releases [1], but invariably we end up
>> doing major ones, which also deliver the bug fixes.
>>
>> The reason was the cost involved in doing a release. We have made some
>> good
>> progress towards regression/integration test, which prompts me to revive
>> this.
>>
>> What does everyone think about a monthly bugfix release on the last
>> major/minor version. (not on every major release, we still don't have
>> enough contributors to pull that off IMO). So we would be trying to do a
>> 0.10.1 early jan for e.g, in this model?
>>
>> [1] https://cwiki.apache.org/confluence/display/HUDI/Release+Management
>>
>> Thanks
>> Vinoth
>>
>
>
> --
> Regards,
> -Sivabalan
>
Re: Regular minor/patch releases
Posted by Y Ethan Guo <et...@gmail.com>.
+1 on packing bug fixes (at best effort) to minor releases.
On Mon, Dec 13, 2021 at 12:06 PM Sivabalan <n....@gmail.com> wrote:
> +1 in general. but yeah, not sure if we have resources to do this for
> every major release.
>
> On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <vi...@apache.org> wrote:
>
>> Hi all,
>>
>> In the past we had plans for minor releases [1], but invariably we end up
>> doing major ones, which also deliver the bug fixes.
>>
>> The reason was the cost involved in doing a release. We have made some
>> good
>> progress towards regression/integration test, which prompts me to revive
>> this.
>>
>> What does everyone think about a monthly bugfix release on the last
>> major/minor version. (not on every major release, we still don't have
>> enough contributors to pull that off IMO). So we would be trying to do a
>> 0.10.1 early jan for e.g, in this model?
>>
>> [1] https://cwiki.apache.org/confluence/display/HUDI/Release+Management
>>
>> Thanks
>> Vinoth
>>
>
>
> --
> Regards,
> -Sivabalan
>
Re: Regular minor/patch releases
Posted by Danny Chan <da...@apache.org>.
> on top of the latest major version
Do you mean to include small features ? I would say that is not a good idea.
In general, we need the minor versions to converge the bugs introduced
by the major version, and 0.10.x is better because people know that
it is a quick follower after 0.10.0 to handle the bug fixes.
Best,
Danny
Vinoth Chandar <vi...@apache.org> 于2021年12月29日周三 10:12写道：
>
> Hi,
>
> I would love for us to get into the cadence of a regular bug fix/minor
> release, on top of the latest major version.
> What do you think about a minor release every month?
> Once there is a new major release, we will switch to issuing the next minor
> release on top of it. i.e once 0.11.0 is out, the next minor will be 0.11.1
> and not 0.10.x
>
> On Tue, Dec 28, 2021 at 11:08 AM Sivabalan <n....@gmail.com> wrote:
>
> > Following up on having regular minor bug fix release, this is what I am
> > thinning.
> > major release every 2 months to 2.5 months and a minor bug fix release on
> > the following month of major release. If incase major release gets pushed,
> > we can skip having a 2nd minor release for now(due to resource
> > availability).
> > If we have consensus, we can try to have a minor release 1 month after any
> > major releases. For eg, we had a major release by dec 10. And so jan 2nd
> > week is when we can target 0.10.1. we might need atleast a month from
> > major release to have accrued some bug fixes and hence.
> >
> > Open to hear thoughts from the community.
> >
> >
> >
> > On Wed, Dec 15, 2021 at 2:15 PM Vinoth Chandar <vi...@apache.org> wrote:
> >
> > > Hi all,
> > >
> > > Thanks for chiming in with the feedback. Looks like there is broad
> > support
> > > for this.
> > >
> > > Responding to few of the views below.
> > >
> > > >With the rush in features without enough tests, I'm afraid the major
> > > release version is never ready for production,
> > > While I agree with you, don't want to be very idealistic here either.
> > 0.10
> > > for e.g had a lot of testing on RCs and bug fixes after as well. And some
> > > of the features were hardened at places at Uber before we released, but
> > > open source major releases are generally rough (you can even see how
> > rough
> > > newer Spark versions are for e.g), and the community puts in the effort
> > to
> > > make it more and more stable going forward. Hudi's problem IMO has been
> > > that we have done only major releases from 0.6 to 0.10 (given our
> > resource
> > > crunch during the pandemic times). Now, is a good time to revisit this.
> > >
> > > >when fixing bugs against the master branch, the contributors/committers
> > > should also open a new PR
> > > We can try this and encourage this always. I am just worried that this
> > adds
> > > more burden on contributors and things may get missed. IMO we can pick
> > two
> > > RMs at any time. One for the next major release and one for the next
> > minor
> > > release and have them shepherd the bug fixes through? We mark JIRAs with
> > > two fix versions.
> > >
> > > >And for minor releases, there should only include the bug fixes, no
> > > breaking change, no feature, it should not be a hard work i think.
> > > +100 on this. otherwise it defeats the purpose of the minor release.
> > >
> > > Thanks
> > > Vinoth
> > >
> > > On Wed, Dec 15, 2021 at 7:22 AM leesf <le...@gmail.com> wrote:
> > >
> > > > +1
> > > >
> > > > We could create new branches such as release-0.10 as the master branch
> > > for
> > > > 0.10.0, 0.10.1 .etc version release, and when fixing bugs against the
> > > > master branch, the contributors/committers should also open a new PR
> > > > against the release-0.10 branch if needed. That would avoid
> > > cherry-picking
> > > > all bug fixes from master to release-0.10 at one time and cause so many
> > > > conflicts. You would see the Spark[1] and Flink[2] community also
> > > > maintaining a multi-master branch as well.
> > > >
> > > > [1] https://github.com/apache/spark/tree/branch-3.1
> > > > https://github.com/apache/spark/tree/branch-3.2
> > > > [2] https://github.com/apache/flink/tree/release-1.12
> > > > https://github.com/apache/flink/tree/release-1.13
> > > >
> > > > vino yang <ya...@gmail.com> 于2021年12月15日周三 18:12写道：
> > > >
> > > > > +1
> > > > >
> > > > > Agree that minor release mostly for bug fix purpose.
> > > > >
> > > > > Best,
> > > > > Vino
> > > > >
> > > > > Danny Chan <da...@apache.org> 于2021年12月15日周三 10:35写道：
> > > > >
> > > > > > I guess we must do that for current rapid development and
> > iteration.
> > > As
> > > > > for
> > > > > > the release 0.10.0, after the announcement of only a few days we
> > have
> > > > > > received a bunch of bugs reported by the github issues: such as
> > > > > >
> > > > > > - the empty meta file: https://github.com/apache/hudi/issues/4249
> > > > > > - and the timeline based marker files:
> > > > > > https://github.com/apache/hudi/issues/4230
> > > > > >
> > > > > > With the rush in features without enough tests, I'm afraid the
> > major
> > > > > > release version is never ready for production, unless there is
> > > > production
> > > > > > validation like in Uber internal.
> > > > > >
> > > > > > And for minor releases, there should only include the bug fixes, no
> > > > > > breaking change, no feature, it should not be a hard work i think.
> > > > > >
> > > > > > Best,
> > > > > > Danny
> > > > > >
> > > > > > Sivabalan <n....@gmail.com>于2021年12月14日 周二上午4:06写道：
> > > > > >
> > > > > > > +1 in general. but yeah, not sure if we have resources to do this
> > > for
> > > > > > every
> > > > > > > major release.
> > > > > > >
> > > > > > > On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <
> > vinoth@apache.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > In the past we had plans for minor releases [1], but invariably
> > > we
> > > > > end
> > > > > > up
> > > > > > > > doing major ones, which also deliver the bug fixes.
> > > > > > > >
> > > > > > > > The reason was the cost involved in doing a release. We have
> > made
> > > > > some
> > > > > > > good
> > > > > > > > progress towards regression/integration test, which prompts me
> > to
> > > > > > revive
> > > > > > > > this.
> > > > > > > >
> > > > > > > > What does everyone think about a monthly bugfix release on the
> > > last
> > > > > > > > major/minor version. (not on every major release, we still
> > don't
> > > > have
> > > > > > > > enough contributors to pull that off IMO). So we would be
> > trying
> > > to
> > > > > do
> > > > > > a
> > > > > > > > 0.10.1 early jan for e.g, in this model?
> > > > > > > >
> > > > > > > > [1]
> > > > > >
> > https://cwiki.apache.org/confluence/display/HUDI/Release+Management
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Vinoth
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Regards,
> > > > > > > -Sivabalan
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Regards,
> > -Sivabalan
> >
Re: Regular minor/patch releases
Posted by Vinoth Chandar <vi...@apache.org>.
Hi,
I would love for us to get into the cadence of a regular bug fix/minor
release, on top of the latest major version.
What do you think about a minor release every month?
Once there is a new major release, we will switch to issuing the next minor
release on top of it. i.e once 0.11.0 is out, the next minor will be 0.11.1
and not 0.10.x
On Tue, Dec 28, 2021 at 11:08 AM Sivabalan <n....@gmail.com> wrote:
> Following up on having regular minor bug fix release, this is what I am
> thinning.
> major release every 2 months to 2.5 months and a minor bug fix release on
> the following month of major release. If incase major release gets pushed,
> we can skip having a 2nd minor release for now(due to resource
> availability).
> If we have consensus, we can try to have a minor release 1 month after any
> major releases. For eg, we had a major release by dec 10. And so jan 2nd
> week is when we can target 0.10.1. we might need atleast a month from
> major release to have accrued some bug fixes and hence.
>
> Open to hear thoughts from the community.
>
>
>
> On Wed, Dec 15, 2021 at 2:15 PM Vinoth Chandar <vi...@apache.org> wrote:
>
> > Hi all,
> >
> > Thanks for chiming in with the feedback. Looks like there is broad
> support
> > for this.
> >
> > Responding to few of the views below.
> >
> > >With the rush in features without enough tests, I'm afraid the major
> > release version is never ready for production,
> > While I agree with you, don't want to be very idealistic here either.
> 0.10
> > for e.g had a lot of testing on RCs and bug fixes after as well. And some
> > of the features were hardened at places at Uber before we released, but
> > open source major releases are generally rough (you can even see how
> rough
> > newer Spark versions are for e.g), and the community puts in the effort
> to
> > make it more and more stable going forward. Hudi's problem IMO has been
> > that we have done only major releases from 0.6 to 0.10 (given our
> resource
> > crunch during the pandemic times). Now, is a good time to revisit this.
> >
> > >when fixing bugs against the master branch, the contributors/committers
> > should also open a new PR
> > We can try this and encourage this always. I am just worried that this
> adds
> > more burden on contributors and things may get missed. IMO we can pick
> two
> > RMs at any time. One for the next major release and one for the next
> minor
> > release and have them shepherd the bug fixes through? We mark JIRAs with
> > two fix versions.
> >
> > >And for minor releases, there should only include the bug fixes, no
> > breaking change, no feature, it should not be a hard work i think.
> > +100 on this. otherwise it defeats the purpose of the minor release.
> >
> > Thanks
> > Vinoth
> >
> > On Wed, Dec 15, 2021 at 7:22 AM leesf <le...@gmail.com> wrote:
> >
> > > +1
> > >
> > > We could create new branches such as release-0.10 as the master branch
> > for
> > > 0.10.0, 0.10.1 .etc version release, and when fixing bugs against the
> > > master branch, the contributors/committers should also open a new PR
> > > against the release-0.10 branch if needed. That would avoid
> > cherry-picking
> > > all bug fixes from master to release-0.10 at one time and cause so many
> > > conflicts. You would see the Spark[1] and Flink[2] community also
> > > maintaining a multi-master branch as well.
> > >
> > > [1] https://github.com/apache/spark/tree/branch-3.1
> > > https://github.com/apache/spark/tree/branch-3.2
> > > [2] https://github.com/apache/flink/tree/release-1.12
> > > https://github.com/apache/flink/tree/release-1.13
> > >
> > > vino yang <ya...@gmail.com> 于2021年12月15日周三 18:12写道：
> > >
> > > > +1
> > > >
> > > > Agree that minor release mostly for bug fix purpose.
> > > >
> > > > Best,
> > > > Vino
> > > >
> > > > Danny Chan <da...@apache.org> 于2021年12月15日周三 10:35写道：
> > > >
> > > > > I guess we must do that for current rapid development and
> iteration.
> > As
> > > > for
> > > > > the release 0.10.0, after the announcement of only a few days we
> have
> > > > > received a bunch of bugs reported by the github issues: such as
> > > > >
> > > > > - the empty meta file: https://github.com/apache/hudi/issues/4249
> > > > > - and the timeline based marker files:
> > > > > https://github.com/apache/hudi/issues/4230
> > > > >
> > > > > With the rush in features without enough tests, I'm afraid the
> major
> > > > > release version is never ready for production, unless there is
> > > production
> > > > > validation like in Uber internal.
> > > > >
> > > > > And for minor releases, there should only include the bug fixes, no
> > > > > breaking change, no feature, it should not be a hard work i think.
> > > > >
> > > > > Best,
> > > > > Danny
> > > > >
> > > > > Sivabalan <n....@gmail.com>于2021年12月14日 周二上午4:06写道：
> > > > >
> > > > > > +1 in general. but yeah, not sure if we have resources to do this
> > for
> > > > > every
> > > > > > major release.
> > > > > >
> > > > > > On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <
> vinoth@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > In the past we had plans for minor releases [1], but invariably
> > we
> > > > end
> > > > > up
> > > > > > > doing major ones, which also deliver the bug fixes.
> > > > > > >
> > > > > > > The reason was the cost involved in doing a release. We have
> made
> > > > some
> > > > > > good
> > > > > > > progress towards regression/integration test, which prompts me
> to
> > > > > revive
> > > > > > > this.
> > > > > > >
> > > > > > > What does everyone think about a monthly bugfix release on the
> > last
> > > > > > > major/minor version. (not on every major release, we still
> don't
> > > have
> > > > > > > enough contributors to pull that off IMO). So we would be
> trying
> > to
> > > > do
> > > > > a
> > > > > > > 0.10.1 early jan for e.g, in this model?
> > > > > > >
> > > > > > > [1]
> > > > >
> https://cwiki.apache.org/confluence/display/HUDI/Release+Management
> > > > > > >
> > > > > > > Thanks
> > > > > > > Vinoth
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > -Sivabalan
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Regards,
> -Sivabalan
>
Re: Regular minor/patch releases
Posted by Sivabalan <n....@gmail.com>.
Following up on having regular minor bug fix release, this is what I am
thinning.
major release every 2 months to 2.5 months and a minor bug fix release on
the following month of major release. If incase major release gets pushed,
we can skip having a 2nd minor release for now(due to resource
availability).
If we have consensus, we can try to have a minor release 1 month after any
major releases. For eg, we had a major release by dec 10. And so jan 2nd
week is when we can target 0.10.1. we might need atleast a month from
major release to have accrued some bug fixes and hence.
Open to hear thoughts from the community.
On Wed, Dec 15, 2021 at 2:15 PM Vinoth Chandar <vi...@apache.org> wrote:
> Hi all,
>
> Thanks for chiming in with the feedback. Looks like there is broad support
> for this.
>
> Responding to few of the views below.
>
> >With the rush in features without enough tests, I'm afraid the major
> release version is never ready for production,
> While I agree with you, don't want to be very idealistic here either. 0.10
> for e.g had a lot of testing on RCs and bug fixes after as well. And some
> of the features were hardened at places at Uber before we released, but
> open source major releases are generally rough (you can even see how rough
> newer Spark versions are for e.g), and the community puts in the effort to
> make it more and more stable going forward. Hudi's problem IMO has been
> that we have done only major releases from 0.6 to 0.10 (given our resource
> crunch during the pandemic times). Now, is a good time to revisit this.
>
> >when fixing bugs against the master branch, the contributors/committers
> should also open a new PR
> We can try this and encourage this always. I am just worried that this adds
> more burden on contributors and things may get missed. IMO we can pick two
> RMs at any time. One for the next major release and one for the next minor
> release and have them shepherd the bug fixes through? We mark JIRAs with
> two fix versions.
>
> >And for minor releases, there should only include the bug fixes, no
> breaking change, no feature, it should not be a hard work i think.
> +100 on this. otherwise it defeats the purpose of the minor release.
>
> Thanks
> Vinoth
>
> On Wed, Dec 15, 2021 at 7:22 AM leesf <le...@gmail.com> wrote:
>
> > +1
> >
> > We could create new branches such as release-0.10 as the master branch
> for
> > 0.10.0, 0.10.1 .etc version release, and when fixing bugs against the
> > master branch, the contributors/committers should also open a new PR
> > against the release-0.10 branch if needed. That would avoid
> cherry-picking
> > all bug fixes from master to release-0.10 at one time and cause so many
> > conflicts. You would see the Spark[1] and Flink[2] community also
> > maintaining a multi-master branch as well.
> >
> > [1] https://github.com/apache/spark/tree/branch-3.1
> > https://github.com/apache/spark/tree/branch-3.2
> > [2] https://github.com/apache/flink/tree/release-1.12
> > https://github.com/apache/flink/tree/release-1.13
> >
> > vino yang <ya...@gmail.com> 于2021年12月15日周三 18:12写道：
> >
> > > +1
> > >
> > > Agree that minor release mostly for bug fix purpose.
> > >
> > > Best,
> > > Vino
> > >
> > > Danny Chan <da...@apache.org> 于2021年12月15日周三 10:35写道：
> > >
> > > > I guess we must do that for current rapid development and iteration.
> As
> > > for
> > > > the release 0.10.0, after the announcement of only a few days we have
> > > > received a bunch of bugs reported by the github issues: such as
> > > >
> > > > - the empty meta file: https://github.com/apache/hudi/issues/4249
> > > > - and the timeline based marker files:
> > > > https://github.com/apache/hudi/issues/4230
> > > >
> > > > With the rush in features without enough tests, I'm afraid the major
> > > > release version is never ready for production, unless there is
> > production
> > > > validation like in Uber internal.
> > > >
> > > > And for minor releases, there should only include the bug fixes, no
> > > > breaking change, no feature, it should not be a hard work i think.
> > > >
> > > > Best,
> > > > Danny
> > > >
> > > > Sivabalan <n....@gmail.com>于2021年12月14日 周二上午4:06写道：
> > > >
> > > > > +1 in general. but yeah, not sure if we have resources to do this
> for
> > > > every
> > > > > major release.
> > > > >
> > > > > On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <vinoth@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > In the past we had plans for minor releases [1], but invariably
> we
> > > end
> > > > up
> > > > > > doing major ones, which also deliver the bug fixes.
> > > > > >
> > > > > > The reason was the cost involved in doing a release. We have made
> > > some
> > > > > good
> > > > > > progress towards regression/integration test, which prompts me to
> > > > revive
> > > > > > this.
> > > > > >
> > > > > > What does everyone think about a monthly bugfix release on the
> last
> > > > > > major/minor version. (not on every major release, we still don't
> > have
> > > > > > enough contributors to pull that off IMO). So we would be trying
> to
> > > do
> > > > a
> > > > > > 0.10.1 early jan for e.g, in this model?
> > > > > >
> > > > > > [1]
> > > > https://cwiki.apache.org/confluence/display/HUDI/Release+Management
> > > > > >
> > > > > > Thanks
> > > > > > Vinoth
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > > -Sivabalan
> > > > >
> > > >
> > >
> >
>
--
Regards,
-Sivabalan
Re: Regular minor/patch releases
Posted by Vinoth Chandar <vi...@apache.org>.
Hi all,
Thanks for chiming in with the feedback. Looks like there is broad support
for this.
Responding to few of the views below.
>With the rush in features without enough tests, I'm afraid the major
release version is never ready for production,
While I agree with you, don't want to be very idealistic here either. 0.10
for e.g had a lot of testing on RCs and bug fixes after as well. And some
of the features were hardened at places at Uber before we released, but
open source major releases are generally rough (you can even see how rough
newer Spark versions are for e.g), and the community puts in the effort to
make it more and more stable going forward. Hudi's problem IMO has been
that we have done only major releases from 0.6 to 0.10 (given our resource
crunch during the pandemic times). Now, is a good time to revisit this.
>when fixing bugs against the master branch, the contributors/committers
should also open a new PR
We can try this and encourage this always. I am just worried that this adds
more burden on contributors and things may get missed. IMO we can pick two
RMs at any time. One for the next major release and one for the next minor
release and have them shepherd the bug fixes through? We mark JIRAs with
two fix versions.
>And for minor releases, there should only include the bug fixes, no
breaking change, no feature, it should not be a hard work i think.
+100 on this. otherwise it defeats the purpose of the minor release.
Thanks
Vinoth
On Wed, Dec 15, 2021 at 7:22 AM leesf <le...@gmail.com> wrote:
> +1
>
> We could create new branches such as release-0.10 as the master branch for
> 0.10.0, 0.10.1 .etc version release, and when fixing bugs against the
> master branch, the contributors/committers should also open a new PR
> against the release-0.10 branch if needed. That would avoid cherry-picking
> all bug fixes from master to release-0.10 at one time and cause so many
> conflicts. You would see the Spark[1] and Flink[2] community also
> maintaining a multi-master branch as well.
>
> [1] https://github.com/apache/spark/tree/branch-3.1
> https://github.com/apache/spark/tree/branch-3.2
> [2] https://github.com/apache/flink/tree/release-1.12
> https://github.com/apache/flink/tree/release-1.13
>
> vino yang <ya...@gmail.com> 于2021年12月15日周三 18:12写道：
>
> > +1
> >
> > Agree that minor release mostly for bug fix purpose.
> >
> > Best,
> > Vino
> >
> > Danny Chan <da...@apache.org> 于2021年12月15日周三 10:35写道：
> >
> > > I guess we must do that for current rapid development and iteration. As
> > for
> > > the release 0.10.0, after the announcement of only a few days we have
> > > received a bunch of bugs reported by the github issues: such as
> > >
> > > - the empty meta file: https://github.com/apache/hudi/issues/4249
> > > - and the timeline based marker files:
> > > https://github.com/apache/hudi/issues/4230
> > >
> > > With the rush in features without enough tests, I'm afraid the major
> > > release version is never ready for production, unless there is
> production
> > > validation like in Uber internal.
> > >
> > > And for minor releases, there should only include the bug fixes, no
> > > breaking change, no feature, it should not be a hard work i think.
> > >
> > > Best,
> > > Danny
> > >
> > > Sivabalan <n....@gmail.com>于2021年12月14日 周二上午4:06写道：
> > >
> > > > +1 in general. but yeah, not sure if we have resources to do this for
> > > every
> > > > major release.
> > > >
> > > > On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <vi...@apache.org>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > In the past we had plans for minor releases [1], but invariably we
> > end
> > > up
> > > > > doing major ones, which also deliver the bug fixes.
> > > > >
> > > > > The reason was the cost involved in doing a release. We have made
> > some
> > > > good
> > > > > progress towards regression/integration test, which prompts me to
> > > revive
> > > > > this.
> > > > >
> > > > > What does everyone think about a monthly bugfix release on the last
> > > > > major/minor version. (not on every major release, we still don't
> have
> > > > > enough contributors to pull that off IMO). So we would be trying to
> > do
> > > a
> > > > > 0.10.1 early jan for e.g, in this model?
> > > > >
> > > > > [1]
> > > https://cwiki.apache.org/confluence/display/HUDI/Release+Management
> > > > >
> > > > > Thanks
> > > > > Vinoth
> > > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > > -Sivabalan
> > > >
> > >
> >
>
Re: Regular minor/patch releases
Posted by leesf <le...@gmail.com>.
+1
We could create new branches such as release-0.10 as the master branch for
0.10.0, 0.10.1 .etc version release, and when fixing bugs against the
master branch, the contributors/committers should also open a new PR
against the release-0.10 branch if needed. That would avoid cherry-picking
all bug fixes from master to release-0.10 at one time and cause so many
conflicts. You would see the Spark[1] and Flink[2] community also
maintaining a multi-master branch as well.
[1] https://github.com/apache/spark/tree/branch-3.1
https://github.com/apache/spark/tree/branch-3.2
[2] https://github.com/apache/flink/tree/release-1.12
https://github.com/apache/flink/tree/release-1.13
vino yang <ya...@gmail.com> 于2021年12月15日周三 18:12写道：
> +1
>
> Agree that minor release mostly for bug fix purpose.
>
> Best,
> Vino
>
> Danny Chan <da...@apache.org> 于2021年12月15日周三 10:35写道：
>
> > I guess we must do that for current rapid development and iteration. As
> for
> > the release 0.10.0, after the announcement of only a few days we have
> > received a bunch of bugs reported by the github issues: such as
> >
> > - the empty meta file: https://github.com/apache/hudi/issues/4249
> > - and the timeline based marker files:
> > https://github.com/apache/hudi/issues/4230
> >
> > With the rush in features without enough tests, I'm afraid the major
> > release version is never ready for production, unless there is production
> > validation like in Uber internal.
> >
> > And for minor releases, there should only include the bug fixes, no
> > breaking change, no feature, it should not be a hard work i think.
> >
> > Best,
> > Danny
> >
> > Sivabalan <n....@gmail.com>于2021年12月14日 周二上午4:06写道：
> >
> > > +1 in general. but yeah, not sure if we have resources to do this for
> > every
> > > major release.
> > >
> > > On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <vi...@apache.org>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > In the past we had plans for minor releases [1], but invariably we
> end
> > up
> > > > doing major ones, which also deliver the bug fixes.
> > > >
> > > > The reason was the cost involved in doing a release. We have made
> some
> > > good
> > > > progress towards regression/integration test, which prompts me to
> > revive
> > > > this.
> > > >
> > > > What does everyone think about a monthly bugfix release on the last
> > > > major/minor version. (not on every major release, we still don't have
> > > > enough contributors to pull that off IMO). So we would be trying to
> do
> > a
> > > > 0.10.1 early jan for e.g, in this model?
> > > >
> > > > [1]
> > https://cwiki.apache.org/confluence/display/HUDI/Release+Management
> > > >
> > > > Thanks
> > > > Vinoth
> > > >
> > >
> > >
> > > --
> > > Regards,
> > > -Sivabalan
> > >
> >
>
Re: Regular minor/patch releases
Posted by vino yang <ya...@gmail.com>.
+1
Agree that minor release mostly for bug fix purpose.
Best,
Vino
Danny Chan <da...@apache.org> 于2021年12月15日周三 10:35写道：
> I guess we must do that for current rapid development and iteration. As for
> the release 0.10.0, after the announcement of only a few days we have
> received a bunch of bugs reported by the github issues: such as
>
> - the empty meta file: https://github.com/apache/hudi/issues/4249
> - and the timeline based marker files:
> https://github.com/apache/hudi/issues/4230
>
> With the rush in features without enough tests, I'm afraid the major
> release version is never ready for production, unless there is production
> validation like in Uber internal.
>
> And for minor releases, there should only include the bug fixes, no
> breaking change, no feature, it should not be a hard work i think.
>
> Best,
> Danny
>
> Sivabalan <n....@gmail.com>于2021年12月14日 周二上午4:06写道：
>
> > +1 in general. but yeah, not sure if we have resources to do this for
> every
> > major release.
> >
> > On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <vi...@apache.org>
> wrote:
> >
> > > Hi all,
> > >
> > > In the past we had plans for minor releases [1], but invariably we end
> up
> > > doing major ones, which also deliver the bug fixes.
> > >
> > > The reason was the cost involved in doing a release. We have made some
> > good
> > > progress towards regression/integration test, which prompts me to
> revive
> > > this.
> > >
> > > What does everyone think about a monthly bugfix release on the last
> > > major/minor version. (not on every major release, we still don't have
> > > enough contributors to pull that off IMO). So we would be trying to do
> a
> > > 0.10.1 early jan for e.g, in this model?
> > >
> > > [1]
> https://cwiki.apache.org/confluence/display/HUDI/Release+Management
> > >
> > > Thanks
> > > Vinoth
> > >
> >
> >
> > --
> > Regards,
> > -Sivabalan
> >
>
Re: Regular minor/patch releases
Posted by Danny Chan <da...@apache.org>.
I guess we must do that for current rapid development and iteration. As for
the release 0.10.0, after the announcement of only a few days we have
received a bunch of bugs reported by the github issues: such as
- the empty meta file: https://github.com/apache/hudi/issues/4249
- and the timeline based marker files:
https://github.com/apache/hudi/issues/4230
With the rush in features without enough tests, I'm afraid the major
release version is never ready for production, unless there is production
validation like in Uber internal.
And for minor releases, there should only include the bug fixes, no
breaking change, no feature, it should not be a hard work i think.
Best,
Danny
Sivabalan <n....@gmail.com>于2021年12月14日 周二上午4:06写道：
> +1 in general. but yeah, not sure if we have resources to do this for every
> major release.
>
> On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <vi...@apache.org> wrote:
>
> > Hi all,
> >
> > In the past we had plans for minor releases [1], but invariably we end up
> > doing major ones, which also deliver the bug fixes.
> >
> > The reason was the cost involved in doing a release. We have made some
> good
> > progress towards regression/integration test, which prompts me to revive
> > this.
> >
> > What does everyone think about a monthly bugfix release on the last
> > major/minor version. (not on every major release, we still don't have
> > enough contributors to pull that off IMO). So we would be trying to do a
> > 0.10.1 early jan for e.g, in this model?
> >
> > [1] https://cwiki.apache.org/confluence/display/HUDI/Release+Management
> >
> > Thanks
> > Vinoth
> >
>
>
> --
> Regards,
> -Sivabalan
>
Re: Regular minor/patch releases
Posted by Sivabalan <n....@gmail.com>.
+1 in general. but yeah, not sure if we have resources to do this for every
major release.
On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <vi...@apache.org> wrote:
> Hi all,
>
> In the past we had plans for minor releases [1], but invariably we end up
> doing major ones, which also deliver the bug fixes.
>
> The reason was the cost involved in doing a release. We have made some good
> progress towards regression/integration test, which prompts me to revive
> this.
>
> What does everyone think about a monthly bugfix release on the last
> major/minor version. (not on every major release, we still don't have
> enough contributors to pull that off IMO). So we would be trying to do a
> 0.10.1 early jan for e.g, in this model?
>
> [1] https://cwiki.apache.org/confluence/display/HUDI/Release+Management
>
> Thanks
> Vinoth
>
--
Regards,
-Sivabalan
Re: Regular minor/patch releases
Posted by Sivabalan <n....@gmail.com>.
+1 in general. but yeah, not sure if we have resources to do this for every
major release.
On Mon, Dec 13, 2021 at 10:01 AM Vinoth Chandar <vi...@apache.org> wrote:
> Hi all,
>
> In the past we had plans for minor releases [1], but invariably we end up
> doing major ones, which also deliver the bug fixes.
>
> The reason was the cost involved in doing a release. We have made some good
> progress towards regression/integration test, which prompts me to revive
> this.
>
> What does everyone think about a monthly bugfix release on the last
> major/minor version. (not on every major release, we still don't have
> enough contributors to pull that off IMO). So we would be trying to do a
> 0.10.1 early jan for e.g, in this model?
>
> [1] https://cwiki.apache.org/confluence/display/HUDI/Release+Management
>
> Thanks
> Vinoth
>
--
Regards,
-Sivabalan