Patches to release branches

[Mithun Radhakrishnan <mi...@yahoo.com.INVALID>]

Greetings, Hive Dev.

In the past few months, my colleagues and I have been trying to roll Hive 0.13 out for wider use on Yahoo's Hadoop clusters. A "challenging" endeavour, shall we say.

Back when we were rolling out Hive 0.12, in spite of basing our builds on the Apache Hive 0.12 release branch, we ran into *several* problems that we wouldn't want to roll into production. (These have variously involved the ORC file-format, dynamic partitioning, metastore performance, query-plan serialization, and so on.) On the bright side, most of what we ran into was already found and rectified on trunk (now Hive 0.13). But those fixes didn't uniformly make it back to branch-0.12, then the current stable release. I fear we're now repeating this with Hive 0.13.

We've found that keeping up with the fixes on trunk is like trying to board a moving train while also trying to pull our shoes on. Patches often don't cleanly apply back to a release-branch because of unrelated changes on trunk. They sometimes depend silently on changes elsewhere. When we're lucky, tests fail. And when we're not, things go hilariously pear-shaped in production. Permit me the temerity of making the following suggestion:

1. For P1 bugs (i.e. involving data corruption, service unavailability, or serious failures without reasonable workarounds), along with a fix for trunk, I move that the current stable release branch also be patched. This will be much easier to accomplish alongside the trunk fix, than months down the line.
2. Of *course*, this doesn't apply to new features on trunk.

I realize this will involve a greater commitment from us (the community), certainly more effort for testing. But it'll ensure that current stable Hive release is both current and stable. =]

Thoughts?

Mithun

Re: Patches to release branches

[Mithun Radhakrishnan <mi...@yahoo.com.INVALID>]

Hello, Sergey.

I’m actually talking about the release-branches, not the release itself. I’m not as concerned about the release artifacts that we’ve pushed out (e.g. for download from Apache, or the maven artifacts). I’m more concerned about the release branch itself not getting the critical fixes that are available on trunk.

I’d like for us to keep the latest release *branch* current with critical fixes. I can understand that this means changing the branch off of which the release was made… In which case, maybe it makes sense to keep another branch going with the updates. The release manager might then choose to make a point-release off of that branch. But IMHO, it can’t be acceptable not to have these fixes available/applicable to the branch for the latest release. 

Mithun

On Sep 9, 2014, at 2:10 PM, Sergey Shelukhin <se...@hortonworks.com> wrote:

> From my experience in HBase, I can say that bug fix (or "dot", e.g. 0.96.1)
> releases are a common practice and work very well for users. The release
> manager of the corresponding major release vets the backports and makes
> bugfix releases on some cadence (i.e. monthly), without much of the usual
> major release overhead.
> 
> 
> On Tue, Sep 9, 2014 at 1:52 PM, Mithun Radhakrishnan <
> mithun.radhakrishnan@yahoo.com.invalid> wrote:
> 
>> Greetings, Hive Dev.
>> 
>> In the past few months, my colleagues and I have been trying to roll Hive
>> 0.13 out for wider use on Yahoo's Hadoop clusters. A "challenging"
>> endeavour, shall we say.
>> 
>> Back when we were rolling out Hive 0.12, in spite of basing our builds on
>> the Apache Hive 0.12 release branch, we ran into *several* problems that we
>> wouldn't want to roll into production. (These have variously involved the
>> ORC file-format, dynamic partitioning, metastore performance, query-plan
>> serialization, and so on.) On the bright side, most of what we ran into was
>> already found and rectified on trunk (now Hive 0.13). But those fixes
>> didn't uniformly make it back to branch-0.12, then the current stable
>> release. I fear we're now repeating this with Hive 0.13.
>> 
>> We've found that keeping up with the fixes on trunk is like trying to
>> board a moving train while also trying to pull our shoes on. Patches often
>> don't cleanly apply back to a release-branch because of unrelated changes
>> on trunk. They sometimes depend silently on changes elsewhere. When we're
>> lucky, tests fail. And when we're not, things go hilariously pear-shaped in
>> production. Permit me the temerity of making the following suggestion:
>> 
>> 1. For P1 bugs (i.e. involving data corruption, service unavailability, or
>> serious failures without reasonable workarounds), along with a fix for
>> trunk, I move that the current stable release branch also be patched. This
>> will be much easier to accomplish alongside the trunk fix, than months down
>> the line.
>> 2. Of *course*, this doesn't apply to new features on trunk.
>> 
>> I realize this will involve a greater commitment from us (the community),
>> certainly more effort for testing. But it'll ensure that current stable
>> Hive release is both current and stable. =]
>> 
>> Thoughts?
>> 
>> Mithun
>> 
>> 
> 
> -- 
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to 
> which it is addressed and may contain information that is confidential, 
> privileged and exempt from disclosure under applicable law. If the reader 
> of this message is not the intended recipient, you are hereby notified that 
> any printing, copying, dissemination, distribution, disclosure or 
> forwarding of this communication is strictly prohibited. If you have 
> received this communication in error, please contact the sender immediately 
> and delete it from your system. Thank You.

Re: Patches to release branches

[Sergey Shelukhin <se...@hortonworks.com>]

>From my experience in HBase, I can say that bug fix (or "dot", e.g. 0.96.1)
releases are a common practice and work very well for users. The release
manager of the corresponding major release vets the backports and makes
bugfix releases on some cadence (i.e. monthly), without much of the usual
major release overhead.


On Tue, Sep 9, 2014 at 1:52 PM, Mithun Radhakrishnan <
mithun.radhakrishnan@yahoo.com.invalid> wrote:

> Greetings, Hive Dev.
>
> In the past few months, my colleagues and I have been trying to roll Hive
> 0.13 out for wider use on Yahoo's Hadoop clusters. A "challenging"
> endeavour, shall we say.
>
> Back when we were rolling out Hive 0.12, in spite of basing our builds on
> the Apache Hive 0.12 release branch, we ran into *several* problems that we
> wouldn't want to roll into production. (These have variously involved the
> ORC file-format, dynamic partitioning, metastore performance, query-plan
> serialization, and so on.) On the bright side, most of what we ran into was
> already found and rectified on trunk (now Hive 0.13). But those fixes
> didn't uniformly make it back to branch-0.12, then the current stable
> release. I fear we're now repeating this with Hive 0.13.
>
> We've found that keeping up with the fixes on trunk is like trying to
> board a moving train while also trying to pull our shoes on. Patches often
> don't cleanly apply back to a release-branch because of unrelated changes
> on trunk. They sometimes depend silently on changes elsewhere. When we're
> lucky, tests fail. And when we're not, things go hilariously pear-shaped in
> production. Permit me the temerity of making the following suggestion:
>
> 1. For P1 bugs (i.e. involving data corruption, service unavailability, or
> serious failures without reasonable workarounds), along with a fix for
> trunk, I move that the current stable release branch also be patched. This
> will be much easier to accomplish alongside the trunk fix, than months down
> the line.
> 2. Of *course*, this doesn't apply to new features on trunk.
>
> I realize this will involve a greater commitment from us (the community),
> certainly more effort for testing. But it'll ensure that current stable
> Hive release is both current and stable. =]
>
> Thoughts?
>
> Mithun
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.