Proposal for 1.2 branch and release plan

[Todd Lipcon <to...@apache.org>]

Hi Kudu developers,

Back in mid-October I proposed a plan for the 1.1 and 1.2 releases[1]. 1.1
is now behind us, so I wanted to start a thread regarding 1.2.

The original proposal was to aim for a 1.2 release for mid-January. While
it was always planned to be a time-based release, we also discussed that
we’d likely be able to get some of the security features into it, as well
as a bunch of other work completed that’s been in flight the last month or
two.

Now that we’re getting a bit closer, I want to re-evaluate the timing and
scope a bit. Although we made some good progress on security features,
things have slowed a bit in recent weeks as people have shifted focus onto
other critical improvements and issues. Given that, and the fact that a lot
of people will be out for multiple weeks in December due to holidays, I
don’t think it’s realistic to guess that we’ll complete (and fully
test/stabilize) the security work in time for a mid-Jan release date.

On the other hand, there has been a lot of other good work in master
recently, both in terms of features as well as bug fixes. Here’s an
incomplete list of features and improvements from my skim of the git log
since 1.1.0 which are either complete or almost-complete:

- Improved consistency of reads (snapshot consistency / read-your-writes)
- File descriptor cache and container size limiting to prevent file system
corruption on RHEL 6
- Ability to bound memory usage for errors in C++ client (new API)
- Ability to list range partitions in Java API (new API)
- Performance improvements (AVX2) for BITSHUFFLE and DICT encoding
- Validation and “guard rails” for table metadata, column names, etc.

There are also a bunch of bug fixes, some of which might be a bit risky to
cherry-pick into a 1.1.x release due to the complexity of the patches. I’ve
spent a good amount of my last several weeks working on stress testing the
master branch, and can tell you that it’s significantly better than 1.1.

So, even without the security work complete, I think we’ve already got
enough juicy stuff lined up for 1.2 to make it enticing for users to
upgrade. We can then push the security work out to 1.3 some time in early
2017.

So, the next question is timing. I’d like to propose that we cut a
branch-1.2 from master this week, and give it a couple weeks of soak time
before release. This differs from previous releases in which we’ve cut a
branch only a day or two before the release candidate. My reasoning is as
follows:

1) Some of the changes since 1.1 have been fairly invasive. In particular,
the consistency work has done surgery on where and how timestamps are
assigned to writes, and a bug here could result in replicas diverging,
crashes, deadlocks, etc. Having a bit of time to soak, stress test, and
correctness-test this work before releasing it to the community would be
prudent.

2) Now that we are post-1.0, we have more and more users depending on Kudu
for production (or almost-production) workloads. As we enter a stage of
greater maturity, it’s probably smart to give each release a bit more
“soak” in a branch before giving it to users.

3) Given that a bunch of the above-mentioned features are currently
wrapping up, we’ll probably get back to working on authentication features,
which will cause quite a bit of churn. It would be better to avoid exposing
the upcoming 1.2 branch to this churn, and branching early is a good way to
insulate it.

Thoughts?

-Todd


[1] http://www.mail-archive.com/dev@kudu.apache.org/msg00237.html

Re: Proposal for 1.2 branch and release plan

[Todd Lipcon <to...@cloudera.com>]

OK, I've created branch-1.2.x. Please keep changes on this branch to
critical bug fixes (eg dataloss, easy-to-hit crashes, etc) and docs
changes. And open the flood gates for changes into master! :)

-Todd

On Fri, Dec 9, 2016 at 8:22 AM, Todd Lipcon <to...@cloudera.com> wrote:

> Modulo a few flaky test fixes in flight, I think the aforementioned
> features for 1.2 are now all checked in.
>
> Given there seems to be support for the plan outlined above, I'll create a
> branch for 1.2 some time in the next few hours. Please start thinking about
> release notes/docs changes that you'd like to get in for 1.2.
>
> -Todd
>
>
> On Thu, Dec 8, 2016 at 12:01 PM, Jordan Birdsell <jt...@apache.org>
> wrote:
>
>> +1
>>
>> On Wed, Dec 7, 2016 at 8:16 PM Jean-Daniel Cryans <jd...@apache.org>
>> wrote:
>>
>> > +1, let's get a 1.2.x branch soon.
>> >
>> > J-D
>> >
>> > On Tue, Dec 6, 2016 at 8:44 PM, Todd Lipcon <to...@apache.org> wrote:
>> >
>> > > Hi Kudu developers,
>> > >
>> > > Back in mid-October I proposed a plan for the 1.1 and 1.2 releases[1].
>> > 1.1
>> > > is now behind us, so I wanted to start a thread regarding 1.2.
>> > >
>> > > The original proposal was to aim for a 1.2 release for mid-January.
>> While
>> > > it was always planned to be a time-based release, we also discussed
>> that
>> > > we’d likely be able to get some of the security features into it, as
>> well
>> > > as a bunch of other work completed that’s been in flight the last
>> month
>> > or
>> > > two.
>> > >
>> > > Now that we’re getting a bit closer, I want to re-evaluate the timing
>> and
>> > > scope a bit. Although we made some good progress on security features,
>> > > things have slowed a bit in recent weeks as people have shifted focus
>> > onto
>> > > other critical improvements and issues. Given that, and the fact that
>> a
>> > lot
>> > > of people will be out for multiple weeks in December due to holidays,
>> I
>> > > don’t think it’s realistic to guess that we’ll complete (and fully
>> > > test/stabilize) the security work in time for a mid-Jan release date.
>> > >
>> > > On the other hand, there has been a lot of other good work in master
>> > > recently, both in terms of features as well as bug fixes. Here’s an
>> > > incomplete list of features and improvements from my skim of the git
>> log
>> > > since 1.1.0 which are either complete or almost-complete:
>> > >
>> > > - Improved consistency of reads (snapshot consistency /
>> read-your-writes)
>> > > - File descriptor cache and container size limiting to prevent file
>> > system
>> > > corruption on RHEL 6
>> > > - Ability to bound memory usage for errors in C++ client (new API)
>> > > - Ability to list range partitions in Java API (new API)
>> > > - Performance improvements (AVX2) for BITSHUFFLE and DICT encoding
>> > > - Validation and “guard rails” for table metadata, column names, etc.
>> > >
>> > > There are also a bunch of bug fixes, some of which might be a bit
>> risky
>> > to
>> > > cherry-pick into a 1.1.x release due to the complexity of the patches.
>> > I’ve
>> > > spent a good amount of my last several weeks working on stress testing
>> > the
>> > > master branch, and can tell you that it’s significantly better than
>> 1.1.
>> > >
>> > > So, even without the security work complete, I think we’ve already got
>> > > enough juicy stuff lined up for 1.2 to make it enticing for users to
>> > > upgrade. We can then push the security work out to 1.3 some time in
>> early
>> > > 2017.
>> > >
>> > > So, the next question is timing. I’d like to propose that we cut a
>> > > branch-1.2 from master this week, and give it a couple weeks of soak
>> time
>> > > before release. This differs from previous releases in which we’ve
>> cut a
>> > > branch only a day or two before the release candidate. My reasoning
>> is as
>> > > follows:
>> > >
>> > > 1) Some of the changes since 1.1 have been fairly invasive. In
>> > particular,
>> > > the consistency work has done surgery on where and how timestamps are
>> > > assigned to writes, and a bug here could result in replicas diverging,
>> > > crashes, deadlocks, etc. Having a bit of time to soak, stress test,
>> and
>> > > correctness-test this work before releasing it to the community would
>> be
>> > > prudent.
>> > >
>> > > 2) Now that we are post-1.0, we have more and more users depending on
>> > Kudu
>> > > for production (or almost-production) workloads. As we enter a stage
>> of
>> > > greater maturity, it’s probably smart to give each release a bit more
>> > > “soak” in a branch before giving it to users.
>> > >
>> > > 3) Given that a bunch of the above-mentioned features are currently
>> > > wrapping up, we’ll probably get back to working on authentication
>> > features,
>> > > which will cause quite a bit of churn. It would be better to avoid
>> > exposing
>> > > the upcoming 1.2 branch to this churn, and branching early is a good
>> way
>> > to
>> > > insulate it.
>> > >
>> > > Thoughts?
>> > >
>> > > -Todd
>> > >
>> > >
>> > > [1] http://www.mail-archive.com/dev@kudu.apache.org/msg00237.html
>> > >
>> >
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Proposal for 1.2 branch and release plan

[Todd Lipcon <to...@cloudera.com>]

Modulo a few flaky test fixes in flight, I think the aforementioned
features for 1.2 are now all checked in.

Given there seems to be support for the plan outlined above, I'll create a
branch for 1.2 some time in the next few hours. Please start thinking about
release notes/docs changes that you'd like to get in for 1.2.

-Todd


On Thu, Dec 8, 2016 at 12:01 PM, Jordan Birdsell <jt...@apache.org>
wrote:

> +1
>
> On Wed, Dec 7, 2016 at 8:16 PM Jean-Daniel Cryans <jd...@apache.org>
> wrote:
>
> > +1, let's get a 1.2.x branch soon.
> >
> > J-D
> >
> > On Tue, Dec 6, 2016 at 8:44 PM, Todd Lipcon <to...@apache.org> wrote:
> >
> > > Hi Kudu developers,
> > >
> > > Back in mid-October I proposed a plan for the 1.1 and 1.2 releases[1].
> > 1.1
> > > is now behind us, so I wanted to start a thread regarding 1.2.
> > >
> > > The original proposal was to aim for a 1.2 release for mid-January.
> While
> > > it was always planned to be a time-based release, we also discussed
> that
> > > we’d likely be able to get some of the security features into it, as
> well
> > > as a bunch of other work completed that’s been in flight the last month
> > or
> > > two.
> > >
> > > Now that we’re getting a bit closer, I want to re-evaluate the timing
> and
> > > scope a bit. Although we made some good progress on security features,
> > > things have slowed a bit in recent weeks as people have shifted focus
> > onto
> > > other critical improvements and issues. Given that, and the fact that a
> > lot
> > > of people will be out for multiple weeks in December due to holidays, I
> > > don’t think it’s realistic to guess that we’ll complete (and fully
> > > test/stabilize) the security work in time for a mid-Jan release date.
> > >
> > > On the other hand, there has been a lot of other good work in master
> > > recently, both in terms of features as well as bug fixes. Here’s an
> > > incomplete list of features and improvements from my skim of the git
> log
> > > since 1.1.0 which are either complete or almost-complete:
> > >
> > > - Improved consistency of reads (snapshot consistency /
> read-your-writes)
> > > - File descriptor cache and container size limiting to prevent file
> > system
> > > corruption on RHEL 6
> > > - Ability to bound memory usage for errors in C++ client (new API)
> > > - Ability to list range partitions in Java API (new API)
> > > - Performance improvements (AVX2) for BITSHUFFLE and DICT encoding
> > > - Validation and “guard rails” for table metadata, column names, etc.
> > >
> > > There are also a bunch of bug fixes, some of which might be a bit risky
> > to
> > > cherry-pick into a 1.1.x release due to the complexity of the patches.
> > I’ve
> > > spent a good amount of my last several weeks working on stress testing
> > the
> > > master branch, and can tell you that it’s significantly better than
> 1.1.
> > >
> > > So, even without the security work complete, I think we’ve already got
> > > enough juicy stuff lined up for 1.2 to make it enticing for users to
> > > upgrade. We can then push the security work out to 1.3 some time in
> early
> > > 2017.
> > >
> > > So, the next question is timing. I’d like to propose that we cut a
> > > branch-1.2 from master this week, and give it a couple weeks of soak
> time
> > > before release. This differs from previous releases in which we’ve cut
> a
> > > branch only a day or two before the release candidate. My reasoning is
> as
> > > follows:
> > >
> > > 1) Some of the changes since 1.1 have been fairly invasive. In
> > particular,
> > > the consistency work has done surgery on where and how timestamps are
> > > assigned to writes, and a bug here could result in replicas diverging,
> > > crashes, deadlocks, etc. Having a bit of time to soak, stress test, and
> > > correctness-test this work before releasing it to the community would
> be
> > > prudent.
> > >
> > > 2) Now that we are post-1.0, we have more and more users depending on
> > Kudu
> > > for production (or almost-production) workloads. As we enter a stage of
> > > greater maturity, it’s probably smart to give each release a bit more
> > > “soak” in a branch before giving it to users.
> > >
> > > 3) Given that a bunch of the above-mentioned features are currently
> > > wrapping up, we’ll probably get back to working on authentication
> > features,
> > > which will cause quite a bit of churn. It would be better to avoid
> > exposing
> > > the upcoming 1.2 branch to this churn, and branching early is a good
> way
> > to
> > > insulate it.
> > >
> > > Thoughts?
> > >
> > > -Todd
> > >
> > >
> > > [1] http://www.mail-archive.com/dev@kudu.apache.org/msg00237.html
> > >
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Proposal for 1.2 branch and release plan

[Jordan Birdsell <jt...@apache.org>]

+1

On Wed, Dec 7, 2016 at 8:16 PM Jean-Daniel Cryans <jd...@apache.org>
wrote:

> +1, let's get a 1.2.x branch soon.
>
> J-D
>
> On Tue, Dec 6, 2016 at 8:44 PM, Todd Lipcon <to...@apache.org> wrote:
>
> > Hi Kudu developers,
> >
> > Back in mid-October I proposed a plan for the 1.1 and 1.2 releases[1].
> 1.1
> > is now behind us, so I wanted to start a thread regarding 1.2.
> >
> > The original proposal was to aim for a 1.2 release for mid-January. While
> > it was always planned to be a time-based release, we also discussed that
> > we’d likely be able to get some of the security features into it, as well
> > as a bunch of other work completed that’s been in flight the last month
> or
> > two.
> >
> > Now that we’re getting a bit closer, I want to re-evaluate the timing and
> > scope a bit. Although we made some good progress on security features,
> > things have slowed a bit in recent weeks as people have shifted focus
> onto
> > other critical improvements and issues. Given that, and the fact that a
> lot
> > of people will be out for multiple weeks in December due to holidays, I
> > don’t think it’s realistic to guess that we’ll complete (and fully
> > test/stabilize) the security work in time for a mid-Jan release date.
> >
> > On the other hand, there has been a lot of other good work in master
> > recently, both in terms of features as well as bug fixes. Here’s an
> > incomplete list of features and improvements from my skim of the git log
> > since 1.1.0 which are either complete or almost-complete:
> >
> > - Improved consistency of reads (snapshot consistency / read-your-writes)
> > - File descriptor cache and container size limiting to prevent file
> system
> > corruption on RHEL 6
> > - Ability to bound memory usage for errors in C++ client (new API)
> > - Ability to list range partitions in Java API (new API)
> > - Performance improvements (AVX2) for BITSHUFFLE and DICT encoding
> > - Validation and “guard rails” for table metadata, column names, etc.
> >
> > There are also a bunch of bug fixes, some of which might be a bit risky
> to
> > cherry-pick into a 1.1.x release due to the complexity of the patches.
> I’ve
> > spent a good amount of my last several weeks working on stress testing
> the
> > master branch, and can tell you that it’s significantly better than 1.1.
> >
> > So, even without the security work complete, I think we’ve already got
> > enough juicy stuff lined up for 1.2 to make it enticing for users to
> > upgrade. We can then push the security work out to 1.3 some time in early
> > 2017.
> >
> > So, the next question is timing. I’d like to propose that we cut a
> > branch-1.2 from master this week, and give it a couple weeks of soak time
> > before release. This differs from previous releases in which we’ve cut a
> > branch only a day or two before the release candidate. My reasoning is as
> > follows:
> >
> > 1) Some of the changes since 1.1 have been fairly invasive. In
> particular,
> > the consistency work has done surgery on where and how timestamps are
> > assigned to writes, and a bug here could result in replicas diverging,
> > crashes, deadlocks, etc. Having a bit of time to soak, stress test, and
> > correctness-test this work before releasing it to the community would be
> > prudent.
> >
> > 2) Now that we are post-1.0, we have more and more users depending on
> Kudu
> > for production (or almost-production) workloads. As we enter a stage of
> > greater maturity, it’s probably smart to give each release a bit more
> > “soak” in a branch before giving it to users.
> >
> > 3) Given that a bunch of the above-mentioned features are currently
> > wrapping up, we’ll probably get back to working on authentication
> features,
> > which will cause quite a bit of churn. It would be better to avoid
> exposing
> > the upcoming 1.2 branch to this churn, and branching early is a good way
> to
> > insulate it.
> >
> > Thoughts?
> >
> > -Todd
> >
> >
> > [1] http://www.mail-archive.com/dev@kudu.apache.org/msg00237.html
> >
>

Re: Proposal for 1.2 branch and release plan

[Jean-Daniel Cryans <jd...@apache.org>]

+1, let's get a 1.2.x branch soon.

J-D

On Tue, Dec 6, 2016 at 8:44 PM, Todd Lipcon <to...@apache.org> wrote:

> Hi Kudu developers,
>
> Back in mid-October I proposed a plan for the 1.1 and 1.2 releases[1]. 1.1
> is now behind us, so I wanted to start a thread regarding 1.2.
>
> The original proposal was to aim for a 1.2 release for mid-January. While
> it was always planned to be a time-based release, we also discussed that
> we’d likely be able to get some of the security features into it, as well
> as a bunch of other work completed that’s been in flight the last month or
> two.
>
> Now that we’re getting a bit closer, I want to re-evaluate the timing and
> scope a bit. Although we made some good progress on security features,
> things have slowed a bit in recent weeks as people have shifted focus onto
> other critical improvements and issues. Given that, and the fact that a lot
> of people will be out for multiple weeks in December due to holidays, I
> don’t think it’s realistic to guess that we’ll complete (and fully
> test/stabilize) the security work in time for a mid-Jan release date.
>
> On the other hand, there has been a lot of other good work in master
> recently, both in terms of features as well as bug fixes. Here’s an
> incomplete list of features and improvements from my skim of the git log
> since 1.1.0 which are either complete or almost-complete:
>
> - Improved consistency of reads (snapshot consistency / read-your-writes)
> - File descriptor cache and container size limiting to prevent file system
> corruption on RHEL 6
> - Ability to bound memory usage for errors in C++ client (new API)
> - Ability to list range partitions in Java API (new API)
> - Performance improvements (AVX2) for BITSHUFFLE and DICT encoding
> - Validation and “guard rails” for table metadata, column names, etc.
>
> There are also a bunch of bug fixes, some of which might be a bit risky to
> cherry-pick into a 1.1.x release due to the complexity of the patches. I’ve
> spent a good amount of my last several weeks working on stress testing the
> master branch, and can tell you that it’s significantly better than 1.1.
>
> So, even without the security work complete, I think we’ve already got
> enough juicy stuff lined up for 1.2 to make it enticing for users to
> upgrade. We can then push the security work out to 1.3 some time in early
> 2017.
>
> So, the next question is timing. I’d like to propose that we cut a
> branch-1.2 from master this week, and give it a couple weeks of soak time
> before release. This differs from previous releases in which we’ve cut a
> branch only a day or two before the release candidate. My reasoning is as
> follows:
>
> 1) Some of the changes since 1.1 have been fairly invasive. In particular,
> the consistency work has done surgery on where and how timestamps are
> assigned to writes, and a bug here could result in replicas diverging,
> crashes, deadlocks, etc. Having a bit of time to soak, stress test, and
> correctness-test this work before releasing it to the community would be
> prudent.
>
> 2) Now that we are post-1.0, we have more and more users depending on Kudu
> for production (or almost-production) workloads. As we enter a stage of
> greater maturity, it’s probably smart to give each release a bit more
> “soak” in a branch before giving it to users.
>
> 3) Given that a bunch of the above-mentioned features are currently
> wrapping up, we’ll probably get back to working on authentication features,
> which will cause quite a bit of churn. It would be better to avoid exposing
> the upcoming 1.2 branch to this churn, and branching early is a good way to
> insulate it.
>
> Thoughts?
>
> -Todd
>
>
> [1] http://www.mail-archive.com/dev@kudu.apache.org/msg00237.html
>

Re: Proposal for 1.2 branch and release plan

[Mike Percy <mp...@apache.org>]

I agree that it would be nice to have another release before the holidays
and before we start merging major security features.

+1 from me.

Mike

On Wed, Dec 7, 2016 at 4:44 AM, Todd Lipcon <to...@apache.org> wrote:

> Hi Kudu developers,
>
> Back in mid-October I proposed a plan for the 1.1 and 1.2 releases[1]. 1.1
> is now behind us, so I wanted to start a thread regarding 1.2.
>
> The original proposal was to aim for a 1.2 release for mid-January. While
> it was always planned to be a time-based release, we also discussed that
> we’d likely be able to get some of the security features into it, as well
> as a bunch of other work completed that’s been in flight the last month or
> two.
>
> Now that we’re getting a bit closer, I want to re-evaluate the timing and
> scope a bit. Although we made some good progress on security features,
> things have slowed a bit in recent weeks as people have shifted focus onto
> other critical improvements and issues. Given that, and the fact that a lot
> of people will be out for multiple weeks in December due to holidays, I
> don’t think it’s realistic to guess that we’ll complete (and fully
> test/stabilize) the security work in time for a mid-Jan release date.
>
> On the other hand, there has been a lot of other good work in master
> recently, both in terms of features as well as bug fixes. Here’s an
> incomplete list of features and improvements from my skim of the git log
> since 1.1.0 which are either complete or almost-complete:
>
> - Improved consistency of reads (snapshot consistency / read-your-writes)
> - File descriptor cache and container size limiting to prevent file system
> corruption on RHEL 6
> - Ability to bound memory usage for errors in C++ client (new API)
> - Ability to list range partitions in Java API (new API)
> - Performance improvements (AVX2) for BITSHUFFLE and DICT encoding
> - Validation and “guard rails” for table metadata, column names, etc.
>
> There are also a bunch of bug fixes, some of which might be a bit risky to
> cherry-pick into a 1.1.x release due to the complexity of the patches. I’ve
> spent a good amount of my last several weeks working on stress testing the
> master branch, and can tell you that it’s significantly better than 1.1.
>
> So, even without the security work complete, I think we’ve already got
> enough juicy stuff lined up for 1.2 to make it enticing for users to
> upgrade. We can then push the security work out to 1.3 some time in early
> 2017.
>
> So, the next question is timing. I’d like to propose that we cut a
> branch-1.2 from master this week, and give it a couple weeks of soak time
> before release. This differs from previous releases in which we’ve cut a
> branch only a day or two before the release candidate. My reasoning is as
> follows:
>
> 1) Some of the changes since 1.1 have been fairly invasive. In particular,
> the consistency work has done surgery on where and how timestamps are
> assigned to writes, and a bug here could result in replicas diverging,
> crashes, deadlocks, etc. Having a bit of time to soak, stress test, and
> correctness-test this work before releasing it to the community would be
> prudent.
>
> 2) Now that we are post-1.0, we have more and more users depending on Kudu
> for production (or almost-production) workloads. As we enter a stage of
> greater maturity, it’s probably smart to give each release a bit more
> “soak” in a branch before giving it to users.
>
> 3) Given that a bunch of the above-mentioned features are currently
> wrapping up, we’ll probably get back to working on authentication features,
> which will cause quite a bit of churn. It would be better to avoid exposing
> the upcoming 1.2 branch to this churn, and branching early is a good way to
> insulate it.
>
> Thoughts?
>
> -Todd
>
>
> [1] http://www.mail-archive.com/dev@kudu.apache.org/msg00237.html
>