Re: [DISCUSS] CASSANDRA-13994

[Joshua McKenzie <jm...@apache.org>]

Where did we land on inclusion of this ticket in 4.0? Since it's now
flagged as blocking beta rel, I am now very interested in this question.

:)


On Wed, May 27, 2020 at 7:52 PM Jordan West <jo...@gmail.com> wrote:

> On Wed, May 27, 2020 at 1:23 PM Joshua McKenzie <jm...@apache.org>
> wrote:
>
> > Maybe. Do we just time box, say we're going to cut an RC and give it 4
> > weeks, if nothing awful surfaces we GA?
> >
>
> I've seen that work well in the past on other projects. I agree with the
> notion that RCs are real candidates for release if no one finds issues with
> them. Ideally we would have as little RCs as possible and have more
> alphas/betas.
>
> >
> > On Wed, May 27, 2020 at 4:12 PM Brandon Williams <dr...@gmail.com>
> wrote:
> >
> > > Absolutely my understanding.
> > >
> > > On Wed, May 27, 2020, 2:49 PM Jeremiah D Jordan <
> > jeremiah.jordan@gmail.com
> > > >
> > > wrote:
> > >
> > > > > A clear point to cut RC's doesn't surface from the above for me.
> > > > Releasing
> > > > > an RC before broad verification seems wrong, and cutting an RC
> after
> > > the
> > > > 4
> > > > > points above may as well be GA because it's all known scope.
> > > >
> > > > Isn’t the whole point of an RC is that it could be the GA?  It is a
> > > > “release candidate”, meaning if no one finds any issues with it, that
> > can
> > > > them become the release?  So that seems like exactly the right time
> to
> > > make
> > > > RC releases?
> > > >
> > > > > On May 27, 2020, at 2:45 PM, Joshua McKenzie <jmckenzie@apache.org
> >
> > > > wrote:
> > > > >
> > > > > I think we're all on the same page here; I was focusing more on the
> > > > release
> > > > > lifecycles and sequencing than the entire version cycle. Good to
> > > broaden
> > > > > scope I think.
> > > > >
> > > > > One thing we're not considering is the separation of API changes
> from
> > > > major
> > > > > changes and how that intersects with release milestones.
> > > > >
> > > > > Meaning:
> > > > > 1. alpha phase
> > > > > 2. Milestone: API freeze (all API changes pushed to next major)
> > > > > 3. beta phase
> > > > > 4. Verification phase (all major disruptive pushed to next major)
> > > > >
> > > > > A clear point to cut RC's doesn't surface from the above for me.
> > > > Releasing
> > > > > an RC before broad verification seems wrong, and cutting an RC
> after
> > > the
> > > > 4
> > > > > points above may as well be GA because it's all known scope.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > On Wed, May 27, 2020 at 3:28 PM Scott Andreas <
> scott@paradoxica.net>
> > > > wrote:
> > > > >
> > > > >> That makes sense to me, yep.
> > > > >>
> > > > >> My hope and expectation is that the time required for
> "verification
> > > > work"
> > > > >> will shrink dramatically in the not too distant future - ideally
> to
> > a
> > > > >> period of less than a month. In this world, the cost of missing
> one
> > > > train
> > > > >> is reduced to catching the next one.
> > > > >>
> > > > >> One of the main goals in shifting focus from "testing" and "test
> > > plans"
> > > > to
> > > > >> "test engineering" is automating as many aspects of release
> > > > qualification
> > > > >> as possible, with an asymptotic ideal as a function of compute
> > > capacity
> > > > and
> > > > >> time. While such automation will never be complete (it's likely
> that
> > > > >> development of new features will/must include qualification infra
> > > > changes
> > > > >> to exercise them), if we're able to apply the same rigor to major
> > > > releases
> > > > >> as we are to patchlevel builds with little incremental effort, I'd
> > be
> > > > >> thrilled.
> > > > >>
> > > > >> This is mostly a way of saying:
> > > > >> – I like the cadence/sequencing Benedict proposes below.
> > > > >> – I think improvements in test engineering can reduce/eliminate
> > > > >> invalidation and may increase the scope of what can be a candidate
> > for
> > > > >> merge on a given branch
> > > > >> – And if not, the cost of missing the train is lower because we'll
> > be
> > > > able
> > > > >> to deliver major releases more often.
> > > > >>
> > > > >> Scott
> > > > >>
> > > > >> ________________________________________
> > > > >> From: Jeremiah D Jordan <je...@gmail.com>
> > > > >> Sent: Wednesday, May 27, 2020 11:54 AM
> > > > >> To: Cassandra DEV
> > > > >> Subject: Re: [DISCUSS] CASSANDRA-13994
> > > > >>
> > > > >> +1 strongly agree.  If we aren’t going to let something go into
> > 4.0.0
> > > > >> because it would "invalidate testing” then we can not let such a
> > thing
> > > > go
> > > > >> into 4.0.1 unless we plan to re-do said testing for the patch
> > release.
> > > > >>
> > > > >>> On May 27, 2020, at 1:31 PM, Benedict Elliott Smith <
> > > > benedict@apache.org>
> > > > >> wrote:
> > > > >>>
> > > > >>> I'm being told this still isn't clear, so let me try in a
> > > bullet-point
> > > > >> timeline:
> > > > >>>
> > > > >>> * 4.0 Beta
> > > > >>> * 4.0 Verification Work
> > > > >>> * [Merge Window]
> > > > >>> * 4.0 GA
> > > > >>> * 4.0 Minor Releases
> > > > >>> * ...
> > > > >>> * 5.0 Dev
> > > > >>> * ...
> > > > >>> * 5.0 Verification Work
> > > > >>> * GA 5.0
> > > > >>>
> > > > >>> I think that anything that is prohibited from "[Merge Window]"
> > > because
> > > > >> it invalidates "4.0 Verification Work" must also be prohibited
> until
> > > > "5.0
> > > > >> Dev" because the next equivalent work that can now validate it
> > occurs
> > > > only
> > > > >> at "5.0 Verification Work"
> > > > >>>
> > > > >>> On 27/05/2020, 19:05, "Benedict Elliott Smith" <
> > benedict@apache.org
> > > >
> > > > >> wrote:
> > > > >>>
> > > > >>>   I'm not sure if I communicated my point very well.  I mean to
> say
> > > > >> that if the reason we are prohibiting a patch to land post-beta is
> > > > because
> > > > >> it invalidates work we only perform pre-ga, then it probably
> should
> > > not
> > > > be
> > > > >> permitted to land post-ga either, since it must also invalidate
> the
> > > same
> > > > >> work?
> > > > >>>
> > > > >>>   That is to say, if we're comfortable with work landing post-ga
> > > > >> because we believe it to be safe to release without our
> > > > pre-major-release
> > > > >> verification, we should be comfortable with it landing at any time
> > > > pre-ga
> > > > >> too.  Anything else seems inconsistent to me, and we should
> examine
> > > what
> > > > >> assumptions we're making that permit this inconsistency to arise.
> > > > >>>
> > > > >>>
> > > > >>>   On 27/05/2020, 18:49, "Joshua McKenzie" <jmckenzie@apache.org
> >
> > > > >> wrote:
> > > > >>>
> > > > >>>>
> > > > >>>> because it invalidates our pre-release verification, then it
> > should
> > > > not
> > > > >>>> land
> > > > >>>
> > > > >>>       until we next perform pre-release verification
> > > > >>>
> > > > >>>       At least for me there's a little softness around our
> > collective
> > > > >> alignment
> > > > >>>       on when pre-release verification takes place. If it's
> between
> > > > >> alpha-1 and
> > > > >>>       ga we don't want changes that would invalidate those
> changes
> > to
> > > > >> land during
> > > > >>>       that time frame. Different for beta-1 to ga. We also risk
> > > > >> invalidating
> > > > >>>       testing if we do any of that testing before wherever that
> > > cutoff
> > > > >> is, and a
> > > > >>>       lack of clarity on that cutoff further muddies those
> waters.
> > > > >>>
> > > > >>>       My very loosely held perspective is that beta-1 to ga is
> the
> > > > >> window in
> > > > >>>       which we apply the "don't do things that will invalidate
> > > > >> verification", and
> > > > >>>       we plan to do that verification during the beta phase. I
> > > *think*
> > > > >> this is
> > > > >>>       consistent w/the current framing of the lifecycle doc. That
> > > being
> > > > >> said, I
> > > > >>>       don't have strong religion on this so if we collectively
> want
> > > to
> > > > >> call it
> > > > >>>       "don't majorly disrupt from alpha-1 to ga", we can
> formalize
> > > that
> > > > >> in the
> > > > >>>       docs and go ahead and triage current open scope for 4.0 and
> > > move
> > > > >> things out.
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>       On Wed, May 27, 2020 at 12:59 PM Ekaterina Dimitrova <
> > > > >>>       ekaterina.dimitrova@datastax.com> wrote:
> > > > >>>
> > > > >>>> Thank you all for your input.
> > > > >>>> I think an important topic is again to revise the lifecycle and
> > > ensure
> > > > >> we
> > > > >>>> really have the vision on what is left until beta. I will start
> a
> > > > >> separate
> > > > >>>> thread on the flaky tests situation soon.
> > > > >>>>
> > > > >>>> For this particular ticket I see a couple of things:
> > > > >>>> - There are a lot of deletions of already not used code
> > > > >>>> - I implemented it still in alpha as per our agreement that this
> > > will
> > > > >> give
> > > > >>>> us enough time for testing. Probably Dinesh as a reviewer can
> give
> > > > some
> > > > >>>> valuable feedback/opinion on the patch.
> > > > >>>> - It definitely touches around important places but the
> important
> > > > thing
> > > > >> is
> > > > >>>> to see how exactly it touches, I think
> > > > >>>> - Considering it for alpha before the major testing in beta
> sounds
> > > > >>>> reasonable to me but I guess it also depends on people
> > availability
> > > to
> > > > >>>> review it in detail and the exact test plans afterwards
> > > > >>>>
> > > > >>>> On Wed, 27 May 2020 at 7:14, Benedict Elliott Smith <
> > > > >> benedict@apache.org>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> I think our "pre-beta" criteria should also be our "not in a
> > major"
> > > > >>>>> criteria.
> > > > >>>>>
> > > > >>>>> If work is prohibited because it invalidates our pre-release
> > > > >>>> verification,
> > > > >>>>> then it should not land until we next perform pre-release
> > > > verification,
> > > > >>>>> which only currently happens once per major.
> > > > >>>>>
> > > > >>>>> This could mean either landing less in a major, or permitting
> > more
> > > in
> > > > >>>> beta
> > > > >>>>> etc.
> > > > >>>>>
> > > > >>>>> On 26/05/2020, 19:24, "Joshua McKenzie" <jmckenzie@apache.org
> >
> > > > wrote:
> > > > >>>>>
> > > > >>>>>   I think an interesting question that informs when to stop
> > > accepting
> > > > >>>>>   specific changes in a release is when we expect any extensive
> > > > >>>>> pre-release
> > > > >>>>>   testing to take place.
> > > > >>>>>
> > > > >>>>>   If we go by our release lifecycle, gutting deprecated code
> > seems
> > > > >>>>> compatible
> > > > >>>>>   w/Alpha but I wouldn't endorse merging it into Beta:
> > > > >>>>>
> > > > >>>>>
> > > > >>
> > >
> https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
> > > > .
> > > > >>>>>   Since almost all of the 40_quality_testing epic stuff is also
> > > beta
> > > > >>>>> phase
> > > > >>>>>   and hasn't really taken off yet, it also seems like there
> will
> > be
> > > > >>>>> extensive
> > > > >>>>>   testing after this phase transition.
> > > > >>>>>
> > > > >>>>>   All that being said, I'd advocate for marking FixVer 4.x to
> > > > indicate
> > > > >>>>>   optionality and disallow merge of tickets like this after
> we're
> > > > done
> > > > >>>>>   w/alpha phase in keeping w/our lifecycle doc in general.
> > > > >>>>>
> > > > >>>>>   Does that make sense? Should we consider revisiting and
> > revising
> > > > the
> > > > >>>>>   lifecycle doc re: larger deprecation / changes and cycle
> > stages?
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>   On Tue, May 26, 2020 at 12:53 PM Oleksandr Petrov <
> > > > >>>>>   oleksandr.petrov@gmail.com> wrote:
> > > > >>>>>
> > > > >>>>>>> 1) Would you block the release over this ticket?
> > > > >>>>>>
> > > > >>>>>> I would definitely not block the release on this ticket.
> > > > >>>>>>
> > > > >>>>>>> 2) Would you prioritize this ticket over testing?
> > > > >>>>>>
> > > > >>>>>> Same here, I would prioritise testing.
> > > > >>>>>>
> > > > >>>>>>> 3) Does fixing this ticket make 4.0 a more stable release?
> > > > >>>>>>
> > > > >>>>>> I wanted to give some context: I wrote that in August 2018.
> > While
> > > I
> > > > >>>>> still
> > > > >>>>>> believe it is important to get rid of this code, I'm
> disinclined
> > > to
> > > > >>>>> merge
> > > > >>>>>> it into 4.0.
> > > > >>>>>>
> > > > >>>>>> Given that the patch is rather big (421 additions and 1,480
> > > > >>>>> deletions) and
> > > > >>>>>> touches many important places, including parser, I would be
> > > > >>>> extremely
> > > > >>>>>> cautious to merge it that late in release cycle. It would be
> > great
> > > > >>>>> to also
> > > > >>>>>> hear arguments that would justify the risk.
> > > > >>>>>>
> > > > >>>>>> Thank you for starting this discussion,
> > > > >>>>>> -- Alex
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> On Tue, May 26, 2020 at 5:20 PM Ekaterina Dimitrova <
> > > > >>>>>> ekaterina.dimitrova@datastax.com> wrote:
> > > > >>>>>>
> > > > >>>>>>> Dear all,
> > > > >>>>>>>
> > > > >>>>>>> Following the ticket review sent on 12th May I wanted to
> bring
> > up
> > > > >>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-13994:
> Remove
> > > > >>>>> COMPACT
> > > > >>>>>>>
> > > > >>>>>>> STORAGE internals before 4.0 release.
> > > > >>>>>>>
> > > > >>>>>>> It is already under review by Dinesh Joshi and Alex Petrov.
> > Not a
> > > > >>>>>>> blocker but already under review.
> > > > >>>>>>>
> > > > >>>>>>> Below are my responses to the questions brought up.
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> 1) Would you block the release over this
> > > > >>>>>>>
> > > > >>>>>>> ticket? - probably not
> > > > >>>>>>>
> > > > >>>>>>> 2) Would you prioritize this ticket over testing? - already
> > > > >>>>>>> implemented but if there are some big changes needed after
> the
> > > > >>>>> review,
> > > > >>>>>>> I doubt it we will want to prioritize over the testing
> > > > >>>>>>>
> > > > >>>>>>> 3) Does fixing
> > > > >>>>>>> this ticket make 4.0 a more stable release? - I will just
> cite
> > > > >>>> Alex
> > > > >>>>>>> Petrov who reported this Jira and I think the rest of us
> would
> > > > >>>>> agree
> > > > >>>>>>> with him here.
> > > > >>>>>>>
> > > > >>>>>>> "I would say it's quite important to clean up compact storage
> > > > >>>>>>> internals in 4.0 before the release. It should have no
> visible
> > > > >>>>>>> side-effects, but it'd be very good to have as it simplifies
> > > > >>>>> multiple
> > > > >>>>>>> code paths."
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> Ekaterina Dimitrova
> > > > >>>>>>> e. ekaterina.dimitrova@datastax.com
> > > > >>>>>>> w. www.datastax.com
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>>
> > > > >>>>>> --
> > > > >>>>>> alex p
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > > >>>>>
> > > ---------------------------------------------------------------------
> > > > >>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > >>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> > > > >>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > >  ---------------------------------------------------------------------
> > > > >>>   To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > >>>   For additional commands, e-mail: dev-help@cassandra.apache.org
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > > > >>>
> > ---------------------------------------------------------------------
> > > > >>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > >>> For additional commands, e-mail: dev-help@cassandra.apache.org
> > > > >>>
> > > > >>
> > > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > >> For additional commands, e-mail: dev-help@cassandra.apache.org
> > > > >>
> > > > >>
> > > > >>
> > ---------------------------------------------------------------------
> > > > >> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > >> For additional commands, e-mail: dev-help@cassandra.apache.org
> > > > >>
> > > > >>
> > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > For additional commands, e-mail: dev-help@cassandra.apache.org
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] CASSANDRA-13994

[Sylvain Lebresne <le...@gmail.com>]

There appears to be a rather important misunderstanding here.

Compact storage *is removed* from 4.0, it has been since at least
CASSANDRA-10857 which prevented 4.0 to start if any compact tables existed.
This was done 2.5+ years ago and is explicitly indicated in the NEWS file
since
then.

The ticket we're discussing, CASSANDRA-13994, truly is a minor and
unconsequential patch. It does not remove "COMPACT STORAGE", since it is
already removed, it removes some leftover "COMPACT STORAGE *internals*". In
other words, it removes *dead* code, code that cannot ever be run, and
that is just polluting the codebase. None of those changes are invasive. I
don't
know where the idea this patch was invasive came from, but it's just not.

It is not meant to have user-visible consequences and it won't impact any
(past
or future) testing in any meaningful way.

And yes, the patch does cover a non trivial amount of _lines_, but it's all
either the removal of clearly unreachable code, or near automatic changes.

So, tbc, my PMC and reviewer opinion is that this is fine to commit in
either
4.0-alpha or 4.0-beta. It cleans the internal, feels as safe as safe can be
and does not impact testing.

--
Sylvain


On Wed, Jun 24, 2020 at 9:03 PM Dinesh Joshi <dj...@apache.org> wrote:

> I might be missing some context here but I thought we did not intend to
> remove compact storage for 4.0 as it was deemed to be too invasive at this
> point. Did that decision change?
>
> Dinesh
>
> > On Jun 24, 2020, at 10:48 AM, Ekaterina Dimitrova <
> ekaterina.dimitrova@datastax.com> wrote:
> >
> > Dear all,
> > As we talked yesterday during our meeting, I am bringing back to the
> table
> > CASSANDRA-13994 <https://issues.apache.org/jira/browse/CASSANDRA-13994>.
> > Description
> >
> > 4.0 comes without thrift (after CASSANDRA-11115
> > <https://issues.apache.org/jira/browse/CASSANDRA-11115>) and COMPACT
> > STORAGE (after CASSANDRA-10857
> > <https://issues.apache.org/jira/browse/CASSANDRA-10857>), and since
> Compact
> > Storage flags are now disabled, all of the related functionality is
> useless.
> >
> > There are still some things to consider:
> > 1. One of the system tables (built indexes) was compact. For now, we just
> > added value column to it to make sure it's backwards-compatible, but we
> > might want to make sure it's just a "normal" table and doesn't have
> > redundant columns.
> > 2. Compact Tables were building indexes in KEYS mode. Removing it is
> > trivial, but this would mean that all built indexes will be defunct. We
> > could log a warning for now and ask users to migrate off those for now
> and
> > completely remove it from future releases. It's just a couple of classes
> > though.
> >
> >
> > Thank you Sylvain for the first pass of review.
> >
> > An option, as also Sylvain suggested, is to split the ticket in two
> tickets
> > - clean the Compact storage dead code in this one and open a new ticket
> to
> > handle the indexes.
> >
> > I am pasting here the latest comment, hopefully this makes it easier for
> > the audience:
> >
> > I have made a first pass of review and offered a few remarks above.
> >
> > But I think this ticket is hanging up on us deciding whether removing the
> > KEYS 2ndary index code is ok or not. And this yields, to me, the question
> > of what is the upgrade path to 4.0 for users that still have KEYS index
> > (which, reminder, could only be created with Thrift, but could *used*
> with
> > CQL and thus still be around).
> >
> > Because, while I haven't tested this myself, I suspect we have a hole
> here.
> >
> > Namely, KEYS index were compact tables, and 4.0 does not start if there
> is
> > still compact tables. And while for user tables, user are asked to use
> DROP
> > COMPACT STORAGE before upgrading, this cannot be done on KEYS index
> (there
> > is just no syntax to do it), so unless there is code I'm not aware of
> (and
> > please, someone correct me if I'm wrong), I don't think user can upgrade
> to
> > 4.0 at all if they still have KEYS index. They'd have to drop those index
> > first.
> >
> > So If I'm right here, this technically mean removing the KEYS index code
> in
> > 4.0 is fine, since you cannot upgrade in the first place if you have KEYS
> > index. But the more important question for 4.0 imo is what is the upgrade
> > path for users if they have a KEYS index in 3.X?
> >
> > Currently (without code changes), the only available option I can think
> of
> > is that before upgrade to 4.0, users would have to 1) drop their KEYS
> index
> > and then 2) re-create a "normal" (non-KEYS) equivalent index.
> >
> > Are we comfortable with that being the upgrade path for KEYS index?
> >
> > Personally, I'm not sure I am because this is not a seamless upgrade, as
> > between the 1) and 2) above, there is a window where there is no
> accessible
> > index, so if the user application rely on it, it means a period of
> downtime
> > for the application to perform the upgrade. However, if we want a more
> > seamless upgrade, we need to figure something out, and that probably
> > involve non trivial amounts of code and testing. And, playing devil's
> > advocate, KEYS index being so old, maybe nobody that plans to upgrade to
> > 4.0 have them anymore, and maybe it's not worth bothering?
> >
> > So I could use others opinions here.
> >
> > Tl;dr, this ticket raises the point that "Oops, I'm not sure we have
> though
> > through the question of upgrade to 4.0 for KEYS indexes". And tbc, it's
> not
> > directly related to this ticket, only indirectly, but it is still
> something
> > we need to figure out. And I'd say, before 4.0-alpha. But I'm happy to
> > create a separate ticket specific to that question if that helps.
> >
> >
> > Ekaterina
> >
> >>
> >>
> >> It was moved to alpha following the lifecycle document and the latest
> >> discussion on the ML. Sylvain made a review yesterday. To me the biggest
> >> question is about the indexing. (Posted questions on the ticket itself,
> >> waiting for opinion and agreement) For the rest of the ticket I support
> >> Sylvain that it is primarily removal of dead code.
> >>
> >> My understanding is either it lands in alpha before the big testing in
> >> beta or it is delayed for v.5
> >>
> >> Ekaterina
> >>
> >> On Wed, 10 Jun 2020 at 12:59, Joshua McKenzie <jm...@apache.org>
> >> wrote:
> >>
> >>> Where did we land on inclusion of this ticket in 4.0? Since it's now
> >>> flagged as blocking beta rel, I am now very interested in this
> question.
> >>>
> >>> :)
> >>>
> >>>
> >>> On Wed, May 27, 2020 at 7:52 PM Jordan West <jo...@gmail.com>
> wrote:
> >>>
> >>>> On Wed, May 27, 2020 at 1:23 PM Joshua McKenzie <jmckenzie@apache.org
> >
> >>>> wrote:
> >>>>
> >>>>> Maybe. Do we just time box, say we're going to cut an RC and give it
> 4
> >>>>> weeks, if nothing awful surfaces we GA?
> >>>>>
> >>>>
> >>>> I've seen that work well in the past on other projects. I agree with
> the
> >>>> notion that RCs are real candidates for release if no one finds issues
> >>> with
> >>>> them. Ideally we would have as little RCs as possible and have more
> >>>> alphas/betas.
> >>>>
> >>>>>
> >>>>> On Wed, May 27, 2020 at 4:12 PM Brandon Williams <dr...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> Absolutely my understanding.
> >>>>>>
> >>>>>> On Wed, May 27, 2020, 2:49 PM Jeremiah D Jordan <
> >>>>> jeremiah.jordan@gmail.com
> >>>>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>>> A clear point to cut RC's doesn't surface from the above for me.
> >>>>>>> Releasing
> >>>>>>>> an RC before broad verification seems wrong, and cutting an RC
> >>>> after
> >>>>>> the
> >>>>>>> 4
> >>>>>>>> points above may as well be GA because it's all known scope.
> >>>>>>>
> >>>>>>> Isn’t the whole point of an RC is that it could be the GA?  It is
> >>> a
> >>>>>>> “release candidate”, meaning if no one finds any issues with it,
> >>> that
> >>>>> can
> >>>>>>> them become the release?  So that seems like exactly the right
> >>> time
> >>>> to
> >>>>>> make
> >>>>>>> RC releases?
> >>>>>>>
> >>>>>>>> On May 27, 2020, at 2:45 PM, Joshua McKenzie <
> >>> jmckenzie@apache.org
> >>>>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> I think we're all on the same page here; I was focusing more on
> >>> the
> >>>>>>> release
> >>>>>>>> lifecycles and sequencing than the entire version cycle. Good to
> >>>>>> broaden
> >>>>>>>> scope I think.
> >>>>>>>>
> >>>>>>>> One thing we're not considering is the separation of API changes
> >>>> from
> >>>>>>> major
> >>>>>>>> changes and how that intersects with release milestones.
> >>>>>>>>
> >>>>>>>> Meaning:
> >>>>>>>> 1. alpha phase
> >>>>>>>> 2. Milestone: API freeze (all API changes pushed to next major)
> >>>>>>>> 3. beta phase
> >>>>>>>> 4. Verification phase (all major disruptive pushed to next
> >>> major)
> >>>>>>>>
> >>>>>>>> A clear point to cut RC's doesn't surface from the above for me.
> >>>>>>> Releasing
> >>>>>>>> an RC before broad verification seems wrong, and cutting an RC
> >>>> after
> >>>>>> the
> >>>>>>> 4
> >>>>>>>> points above may as well be GA because it's all known scope.
> >>>>>>>>
> >>>>>>>> Thoughts?
> >>>>>>>>
> >>>>>>>> On Wed, May 27, 2020 at 3:28 PM Scott Andreas <
> >>>> scott@paradoxica.net>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> That makes sense to me, yep.
> >>>>>>>>>
> >>>>>>>>> My hope and expectation is that the time required for
> >>>> "verification
> >>>>>>> work"
> >>>>>>>>> will shrink dramatically in the not too distant future -
> >>> ideally
> >>>> to
> >>>>> a
> >>>>>>>>> period of less than a month. In this world, the cost of missing
> >>>> one
> >>>>>>> train
> >>>>>>>>> is reduced to catching the next one.
> >>>>>>>>>
> >>>>>>>>> One of the main goals in shifting focus from "testing" and
> >>> "test
> >>>>>> plans"
> >>>>>>> to
> >>>>>>>>> "test engineering" is automating as many aspects of release
> >>>>>>> qualification
> >>>>>>>>> as possible, with an asymptotic ideal as a function of compute
> >>>>>> capacity
> >>>>>>> and
> >>>>>>>>> time. While such automation will never be complete (it's likely
> >>>> that
> >>>>>>>>> development of new features will/must include qualification
> >>> infra
> >>>>>>> changes
> >>>>>>>>> to exercise them), if we're able to apply the same rigor to
> >>> major
> >>>>>>> releases
> >>>>>>>>> as we are to patchlevel builds with little incremental effort,
> >>> I'd
> >>>>> be
> >>>>>>>>> thrilled.
> >>>>>>>>>
> >>>>>>>>> This is mostly a way of saying:
> >>>>>>>>> – I like the cadence/sequencing Benedict proposes below.
> >>>>>>>>> – I think improvements in test engineering can reduce/eliminate
> >>>>>>>>> invalidation and may increase the scope of what can be a
> >>> candidate
> >>>>> for
> >>>>>>>>> merge on a given branch
> >>>>>>>>> – And if not, the cost of missing the train is lower because
> >>> we'll
> >>>>> be
> >>>>>>> able
> >>>>>>>>> to deliver major releases more often.
> >>>>>>>>>
> >>>>>>>>> Scott
> >>>>>>>>>
> >>>>>>>>> ________________________________________
> >>>>>>>>> From: Jeremiah D Jordan <je...@gmail.com>
> >>>>>>>>> Sent: Wednesday, May 27, 2020 11:54 AM
> >>>>>>>>> To: Cassandra DEV
> >>>>>>>>> Subject: Re: [DISCUSS] CASSANDRA-13994
> >>>>>>>>>
> >>>>>>>>> +1 strongly agree.  If we aren’t going to let something go into
> >>>>> 4.0.0
> >>>>>>>>> because it would "invalidate testing” then we can not let such
> >>> a
> >>>>> thing
> >>>>>>> go
> >>>>>>>>> into 4.0.1 unless we plan to re-do said testing for the patch
> >>>>> release.
> >>>>>>>>>
> >>>>>>>>>> On May 27, 2020, at 1:31 PM, Benedict Elliott Smith <
> >>>>>>> benedict@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> I'm being told this still isn't clear, so let me try in a
> >>>>>> bullet-point
> >>>>>>>>> timeline:
> >>>>>>>>>>
> >>>>>>>>>> * 4.0 Beta
> >>>>>>>>>> * 4.0 Verification Work
> >>>>>>>>>> * [Merge Window]
> >>>>>>>>>> * 4.0 GA
> >>>>>>>>>> * 4.0 Minor Releases
> >>>>>>>>>> * ...
> >>>>>>>>>> * 5.0 Dev
> >>>>>>>>>> * ...
> >>>>>>>>>> * 5.0 Verification Work
> >>>>>>>>>> * GA 5.0
> >>>>>>>>>>
> >>>>>>>>>> I think that anything that is prohibited from "[Merge Window]"
> >>>>>> because
> >>>>>>>>> it invalidates "4.0 Verification Work" must also be prohibited
> >>>> until
> >>>>>>> "5.0
> >>>>>>>>> Dev" because the next equivalent work that can now validate it
> >>>>> occurs
> >>>>>>> only
> >>>>>>>>> at "5.0 Verification Work"
> >>>>>>>>>>
> >>>>>>>>>> On 27/05/2020, 19:05, "Benedict Elliott Smith" <
> >>>>> benedict@apache.org
> >>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>  I'm not sure if I communicated my point very well.  I mean
> >>> to
> >>>> say
> >>>>>>>>> that if the reason we are prohibiting a patch to land
> >>> post-beta is
> >>>>>>> because
> >>>>>>>>> it invalidates work we only perform pre-ga, then it probably
> >>>> should
> >>>>>> not
> >>>>>>> be
> >>>>>>>>> permitted to land post-ga either, since it must also invalidate
> >>>> the
> >>>>>> same
> >>>>>>>>> work?
> >>>>>>>>>>
> >>>>>>>>>>  That is to say, if we're comfortable with work landing
> >>> post-ga
> >>>>>>>>> because we believe it to be safe to release without our
> >>>>>>> pre-major-release
> >>>>>>>>> verification, we should be comfortable with it landing at any
> >>> time
> >>>>>>> pre-ga
> >>>>>>>>> too.  Anything else seems inconsistent to me, and we should
> >>>> examine
> >>>>>> what
> >>>>>>>>> assumptions we're making that permit this inconsistency to
> >>> arise.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>  On 27/05/2020, 18:49, "Joshua McKenzie" <
> >>> jmckenzie@apache.org
> >>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> because it invalidates our pre-release verification, then it
> >>>>> should
> >>>>>>> not
> >>>>>>>>>>> land
> >>>>>>>>>>
> >>>>>>>>>>      until we next perform pre-release verification
> >>>>>>>>>>
> >>>>>>>>>>      At least for me there's a little softness around our
> >>>>> collective
> >>>>>>>>> alignment
> >>>>>>>>>>      on when pre-release verification takes place. If it's
> >>>> between
> >>>>>>>>> alpha-1 and
> >>>>>>>>>>      ga we don't want changes that would invalidate those
> >>>> changes
> >>>>> to
> >>>>>>>>> land during
> >>>>>>>>>>      that time frame. Different for beta-1 to ga. We also
> >>> risk
> >>>>>>>>> invalidating
> >>>>>>>>>>      testing if we do any of that testing before wherever
> >>> that
> >>>>>> cutoff
> >>>>>>>>> is, and a
> >>>>>>>>>>      lack of clarity on that cutoff further muddies those
> >>>> waters.
> >>>>>>>>>>
> >>>>>>>>>>      My very loosely held perspective is that beta-1 to ga is
> >>>> the
> >>>>>>>>> window in
> >>>>>>>>>>      which we apply the "don't do things that will invalidate
> >>>>>>>>> verification", and
> >>>>>>>>>>      we plan to do that verification during the beta phase. I
> >>>>>> *think*
> >>>>>>>>> this is
> >>>>>>>>>>      consistent w/the current framing of the lifecycle doc.
> >>> That
> >>>>>> being
> >>>>>>>>> said, I
> >>>>>>>>>>      don't have strong religion on this so if we collectively
> >>>> want
> >>>>>> to
> >>>>>>>>> call it
> >>>>>>>>>>      "don't majorly disrupt from alpha-1 to ga", we can
> >>>> formalize
> >>>>>> that
> >>>>>>>>> in the
> >>>>>>>>>>      docs and go ahead and triage current open scope for 4.0
> >>> and
> >>>>>> move
> >>>>>>>>> things out.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>      On Wed, May 27, 2020 at 12:59 PM Ekaterina Dimitrova <
> >>>>>>>>>>      ekaterina.dimitrova@datastax.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thank you all for your input.
> >>>>>>>>>>> I think an important topic is again to revise the lifecycle
> >>> and
> >>>>>> ensure
> >>>>>>>>> we
> >>>>>>>>>>> really have the vision on what is left until beta. I will
> >>> start
> >>>> a
> >>>>>>>>> separate
> >>>>>>>>>>> thread on the flaky tests situation soon.
> >>>>>>>>>>>
> >>>>>>>>>>> For this particular ticket I see a couple of things:
> >>>>>>>>>>> - There are a lot of deletions of already not used code
> >>>>>>>>>>> - I implemented it still in alpha as per our agreement that
> >>> this
> >>>>>> will
> >>>>>>>>> give
> >>>>>>>>>>> us enough time for testing. Probably Dinesh as a reviewer can
> >>>> give
> >>>>>>> some
> >>>>>>>>>>> valuable feedback/opinion on the patch.
> >>>>>>>>>>> - It definitely touches around important places but the
> >>>> important
> >>>>>>> thing
> >>>>>>>>> is
> >>>>>>>>>>> to see how exactly it touches, I think
> >>>>>>>>>>> - Considering it for alpha before the major testing in beta
> >>>> sounds
> >>>>>>>>>>> reasonable to me but I guess it also depends on people
> >>>>> availability
> >>>>>> to
> >>>>>>>>>>> review it in detail and the exact test plans afterwards
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, 27 May 2020 at 7:14, Benedict Elliott Smith <
> >>>>>>>>> benedict@apache.org>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I think our "pre-beta" criteria should also be our "not in a
> >>>>> major"
> >>>>>>>>>>>> criteria.
> >>>>>>>>>>>>
> >>>>>>>>>>>> If work is prohibited because it invalidates our pre-release
> >>>>>>>>>>> verification,
> >>>>>>>>>>>> then it should not land until we next perform pre-release
> >>>>>>> verification,
> >>>>>>>>>>>> which only currently happens once per major.
> >>>>>>>>>>>>
> >>>>>>>>>>>> This could mean either landing less in a major, or
> >>> permitting
> >>>>> more
> >>>>>> in
> >>>>>>>>>>> beta
> >>>>>>>>>>>> etc.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 26/05/2020, 19:24, "Joshua McKenzie" <
> >>> jmckenzie@apache.org
> >>>>>
> >>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>  I think an interesting question that informs when to stop
> >>>>>> accepting
> >>>>>>>>>>>>  specific changes in a release is when we expect any
> >>> extensive
> >>>>>>>>>>>> pre-release
> >>>>>>>>>>>>  testing to take place.
> >>>>>>>>>>>>
> >>>>>>>>>>>>  If we go by our release lifecycle, gutting deprecated code
> >>>>> seems
> >>>>>>>>>>>> compatible
> >>>>>>>>>>>>  w/Alpha but I wouldn't endorse merging it into Beta:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
> >>>>>>> .
> >>>>>>>>>>>>  Since almost all of the 40_quality_testing epic stuff is
> >>> also
> >>>>>> beta
> >>>>>>>>>>>> phase
> >>>>>>>>>>>>  and hasn't really taken off yet, it also seems like there
> >>>> will
> >>>>> be
> >>>>>>>>>>>> extensive
> >>>>>>>>>>>>  testing after this phase transition.
> >>>>>>>>>>>>
> >>>>>>>>>>>>  All that being said, I'd advocate for marking FixVer 4.x
> >>> to
> >>>>>>> indicate
> >>>>>>>>>>>>  optionality and disallow merge of tickets like this after
> >>>> we're
> >>>>>>> done
> >>>>>>>>>>>>  w/alpha phase in keeping w/our lifecycle doc in general.
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Does that make sense? Should we consider revisiting and
> >>>>> revising
> >>>>>>> the
> >>>>>>>>>>>>  lifecycle doc re: larger deprecation / changes and cycle
> >>>>> stages?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>  On Tue, May 26, 2020 at 12:53 PM Oleksandr Petrov <
> >>>>>>>>>>>>  oleksandr.petrov@gmail.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> 1) Would you block the release over this ticket?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I would definitely not block the release on this ticket.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2) Would you prioritize this ticket over testing?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Same here, I would prioritise testing.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> 3) Does fixing this ticket make 4.0 a more stable release?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I wanted to give some context: I wrote that in August 2018.
> >>>>> While
> >>>>>> I
> >>>>>>>>>>>> still
> >>>>>>>>>>>>> believe it is important to get rid of this code, I'm
> >>>> disinclined
> >>>>>> to
> >>>>>>>>>>>> merge
> >>>>>>>>>>>>> it into 4.0.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Given that the patch is rather big (421 additions and 1,480
> >>>>>>>>>>>> deletions) and
> >>>>>>>>>>>>> touches many important places, including parser, I would be
> >>>>>>>>>>> extremely
> >>>>>>>>>>>>> cautious to merge it that late in release cycle. It would
> >>> be
> >>>>> great
> >>>>>>>>>>>> to also
> >>>>>>>>>>>>> hear arguments that would justify the risk.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you for starting this discussion,
> >>>>>>>>>>>>> -- Alex
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, May 26, 2020 at 5:20 PM Ekaterina Dimitrova <
> >>>>>>>>>>>>> ekaterina.dimitrova@datastax.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Dear all,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Following the ticket review sent on 12th May I wanted to
> >>>> bring
> >>>>> up
> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-13994:
> >>>> Remove
> >>>>>>>>>>>> COMPACT
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> STORAGE internals before 4.0 release.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> It is already under review by Dinesh Joshi and Alex
> >>> Petrov.
> >>>>> Not a
> >>>>>>>>>>>>>> blocker but already under review.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Below are my responses to the questions brought up.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 1) Would you block the release over this
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ticket? - probably not
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2) Would you prioritize this ticket over testing? -
> >>> already
> >>>>>>>>>>>>>> implemented but if there are some big changes needed after
> >>>> the
> >>>>>>>>>>>> review,
> >>>>>>>>>>>>>> I doubt it we will want to prioritize over the testing
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 3) Does fixing
> >>>>>>>>>>>>>> this ticket make 4.0 a more stable release? - I will just
> >>>> cite
> >>>>>>>>>>> Alex
> >>>>>>>>>>>>>> Petrov who reported this Jira and I think the rest of us
> >>>> would
> >>>>>>>>>>>> agree
> >>>>>>>>>>>>>> with him here.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> "I would say it's quite important to clean up compact
> >>> storage
> >>>>>>>>>>>>>> internals in 4.0 before the release. It should have no
> >>>> visible
> >>>>>>>>>>>>>> side-effects, but it'd be very good to have as it
> >>> simplifies
> >>>>>>>>>>>> multiple
> >>>>>>>>>>>>>> code paths."
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ekaterina Dimitrova
> >>>>>>>>>>>>>> e. ekaterina.dimitrova@datastax.com
> >>>>>>>>>>>>>> w. www.datastax.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> alex p
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>>>>>>> To unsubscribe, e-mail:
> >>> dev-unsubscribe@cassandra.apache.org
> >>>>>>>>>>>> For additional commands, e-mail:
> >>> dev-help@cassandra.apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>>>>>  To unsubscribe, e-mail:
> >>> dev-unsubscribe@cassandra.apache.org
> >>>>>>>>>>  For additional commands, e-mail:
> >>> dev-help@cassandra.apache.org
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>>>>>>>> For additional commands, e-mail:
> >>> dev-help@cassandra.apache.org
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>

Re: [DISCUSS] CASSANDRA-13994

[Joshua McKenzie <jm...@apache.org>]

The new release lifecycle guarantees changes these dynamics. Given our
commitment to API stability between beta 1 and GA, this has introduced a
time where changes being deferred to the next major due to API modification
will be unable to be merged unless we branch upon beta.

This warrants another discuss thread I think rather than nesting it here.
I'll take that on.

On Fri, Jun 26, 2020 at 4:38 AM Mick Semb Wever <mc...@apache.org> wrote:

> >
> >  Also, what is the plan for cutting beta branch? I am still learning the
> > Apache/C* way so excuse me if I missed something. Looking at the
> Lifecycle
> > document, I see a point only about GA major version branch.
> > My understanding is that we are already in freeze for big changes. When
> > would be the right time for creating a beta branch? Something more than
> > closing the last beta blockers tickets?
>
>
>
> My understanding from the doc is that creating the release branch
> `cassandra-4.0` happens with GA. That means we're working on trunk until
> GA. Previously C* hasn't had any branches but these release branches, i.e.
> no beta or rc branches…  (afaik)
>
> I would have thought it better to branch on RC? if at that point much of
> the testing efforts are subsiding, and folk are ready for other things.
>

Re: [DISCUSS] CASSANDRA-13994

[Mick Semb Wever <mc...@apache.org>]

>
>  Also, what is the plan for cutting beta branch? I am still learning the
> Apache/C* way so excuse me if I missed something. Looking at the Lifecycle
> document, I see a point only about GA major version branch.
> My understanding is that we are already in freeze for big changes. When
> would be the right time for creating a beta branch? Something more than
> closing the last beta blockers tickets?



My understanding from the doc is that creating the release branch
`cassandra-4.0` happens with GA. That means we're working on trunk until
GA. Previously C* hasn't had any branches but these release branches, i.e.
no beta or rc branches…  (afaik)

I would have thought it better to branch on RC? if at that point much of
the testing efforts are subsiding, and folk are ready for other things.

Re: [DISCUSS] CASSANDRA-13994

[Ekaterina Dimitrova <e....@gmail.com>]

Considering the latest comments  on the ticket from today, I don’t see this
ticket anymore as a beta blocker.
 Also, what is the plan for cutting beta branch? I am still learning the
Apache/C* way so excuse me if I missed something. Looking at the Lifecycle
document, I see a point only about GA major version branch.
My understanding is that we are already in freeze for big changes. When
would be the right time for creating a beta branch? Something more than
closing the last beta blockers tickets?
Thanks
Ekaterina

On Thu, 25 Jun 2020 at 14:25, Joshua McKenzie <jm...@apache.org> wrote:

> I don't think we really reached a consensus on this one. Some folks
> certainly vocalized a preference for not removing in 4.0, but I read it as
> debate still ongoing then losing momentum. Specifically thinking of
> Sylvain's comment here that kind of dead-ended:
>
> https://issues.apache.org/jira/browse/CASSANDRA-13994?focusedCommentId=17138278&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17138278
>
> fwiw I'm personally fine w/us deferring this until after 4.0 assuming we
> cut a branch for beta so changes like this don't bitrot.
>
> On Wed, Jun 24, 2020 at 3:03 PM Dinesh Joshi <dj...@apache.org> wrote:
>
> > I might be missing some context here but I thought we did not intend to
> > remove compact storage for 4.0 as it was deemed to be too invasive at
> this
> > point. Did that decision change?
> >
> > Dinesh
> >
> > > On Jun 24, 2020, at 10:48 AM, Ekaterina Dimitrova <
> > ekaterina.dimitrova@datastax.com> wrote:
> > >
> > > Dear all,
> > > As we talked yesterday during our meeting, I am bringing back to the
> > table
> > > CASSANDRA-13994 <https://issues.apache.org/jira/browse/CASSANDRA-13994
> >.
> > > Description
> > >
> > > 4.0 comes without thrift (after CASSANDRA-11115
> > > <https://issues.apache.org/jira/browse/CASSANDRA-11115>) and COMPACT
> > > STORAGE (after CASSANDRA-10857
> > > <https://issues.apache.org/jira/browse/CASSANDRA-10857>), and since
> > Compact
> > > Storage flags are now disabled, all of the related functionality is
> > useless.
> > >
> > > There are still some things to consider:
> > > 1. One of the system tables (built indexes) was compact. For now, we
> just
> > > added value column to it to make sure it's backwards-compatible, but we
> > > might want to make sure it's just a "normal" table and doesn't have
> > > redundant columns.
> > > 2. Compact Tables were building indexes in KEYS mode. Removing it is
> > > trivial, but this would mean that all built indexes will be defunct. We
> > > could log a warning for now and ask users to migrate off those for now
> > and
> > > completely remove it from future releases. It's just a couple of
> classes
> > > though.
> > >
> > >
> > > Thank you Sylvain for the first pass of review.
> > >
> > > An option, as also Sylvain suggested, is to split the ticket in two
> > tickets
> > > - clean the Compact storage dead code in this one and open a new ticket
> > to
> > > handle the indexes.
> > >
> > > I am pasting here the latest comment, hopefully this makes it easier
> for
> > > the audience:
> > >
> > > I have made a first pass of review and offered a few remarks above.
> > >
> > > But I think this ticket is hanging up on us deciding whether removing
> the
> > > KEYS 2ndary index code is ok or not. And this yields, to me, the
> question
> > > of what is the upgrade path to 4.0 for users that still have KEYS index
> > > (which, reminder, could only be created with Thrift, but could *used*
> > with
> > > CQL and thus still be around).
> > >
> > > Because, while I haven't tested this myself, I suspect we have a hole
> > here.
> > >
> > > Namely, KEYS index were compact tables, and 4.0 does not start if there
> > is
> > > still compact tables. And while for user tables, user are asked to use
> > DROP
> > > COMPACT STORAGE before upgrading, this cannot be done on KEYS index
> > (there
> > > is just no syntax to do it), so unless there is code I'm not aware of
> > (and
> > > please, someone correct me if I'm wrong), I don't think user can
> upgrade
> > to
> > > 4.0 at all if they still have KEYS index. They'd have to drop those
> index
> > > first.
> > >
> > > So If I'm right here, this technically mean removing the KEYS index
> code
> > in
> > > 4.0 is fine, since you cannot upgrade in the first place if you have
> KEYS
> > > index. But the more important question for 4.0 imo is what is the
> upgrade
> > > path for users if they have a KEYS index in 3.X?
> > >
> > > Currently (without code changes), the only available option I can think
> > of
> > > is that before upgrade to 4.0, users would have to 1) drop their KEYS
> > index
> > > and then 2) re-create a "normal" (non-KEYS) equivalent index.
> > >
> > > Are we comfortable with that being the upgrade path for KEYS index?
> > >
> > > Personally, I'm not sure I am because this is not a seamless upgrade,
> as
> > > between the 1) and 2) above, there is a window where there is no
> > accessible
> > > index, so if the user application rely on it, it means a period of
> > downtime
> > > for the application to perform the upgrade. However, if we want a more
> > > seamless upgrade, we need to figure something out, and that probably
> > > involve non trivial amounts of code and testing. And, playing devil's
> > > advocate, KEYS index being so old, maybe nobody that plans to upgrade
> to
> > > 4.0 have them anymore, and maybe it's not worth bothering?
> > >
> > > So I could use others opinions here.
> > >
> > > Tl;dr, this ticket raises the point that "Oops, I'm not sure we have
> > though
> > > through the question of upgrade to 4.0 for KEYS indexes". And tbc, it's
> > not
> > > directly related to this ticket, only indirectly, but it is still
> > something
> > > we need to figure out. And I'd say, before 4.0-alpha. But I'm happy to
> > > create a separate ticket specific to that question if that helps.
> > >
> > >
> > > Ekaterina
> > >
> > >>
> > >>
> > >> It was moved to alpha following the lifecycle document and the latest
> > >> discussion on the ML. Sylvain made a review yesterday. To me the
> biggest
> > >> question is about the indexing. (Posted questions on the ticket
> itself,
> > >> waiting for opinion and agreement) For the rest of the ticket I
> support
> > >> Sylvain that it is primarily removal of dead code.
> > >>
> > >> My understanding is either it lands in alpha before the big testing in
> > >> beta or it is delayed for v.5
> > >>
> > >> Ekaterina
> > >>
> > >> On Wed, 10 Jun 2020 at 12:59, Joshua McKenzie <jm...@apache.org>
> > >> wrote:
> > >>
> > >>> Where did we land on inclusion of this ticket in 4.0? Since it's now
> > >>> flagged as blocking beta rel, I am now very interested in this
> > question.
> > >>>
> > >>> :)
> > >>>
> > >>>
> > >>> On Wed, May 27, 2020 at 7:52 PM Jordan West <jo...@gmail.com>
> > wrote:
> > >>>
> > >>>> On Wed, May 27, 2020 at 1:23 PM Joshua McKenzie <
> jmckenzie@apache.org
> > >
> > >>>> wrote:
> > >>>>
> > >>>>> Maybe. Do we just time box, say we're going to cut an RC and give
> it
> > 4
> > >>>>> weeks, if nothing awful surfaces we GA?
> > >>>>>
> > >>>>
> > >>>> I've seen that work well in the past on other projects. I agree with
> > the
> > >>>> notion that RCs are real candidates for release if no one finds
> issues
> > >>> with
> > >>>> them. Ideally we would have as little RCs as possible and have more
> > >>>> alphas/betas.
> > >>>>
> > >>>>>
> > >>>>> On Wed, May 27, 2020 at 4:12 PM Brandon Williams <driftx@gmail.com
> >
> > >>>> wrote:
> > >>>>>
> > >>>>>> Absolutely my understanding.
> > >>>>>>
> > >>>>>> On Wed, May 27, 2020, 2:49 PM Jeremiah D Jordan <
> > >>>>> jeremiah.jordan@gmail.com
> > >>>>>>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>>> A clear point to cut RC's doesn't surface from the above for me.
> > >>>>>>> Releasing
> > >>>>>>>> an RC before broad verification seems wrong, and cutting an RC
> > >>>> after
> > >>>>>> the
> > >>>>>>> 4
> > >>>>>>>> points above may as well be GA because it's all known scope.
> > >>>>>>>
> > >>>>>>> Isn’t the whole point of an RC is that it could be the GA?  It is
> > >>> a
> > >>>>>>> “release candidate”, meaning if no one finds any issues with it,
> > >>> that
> > >>>>> can
> > >>>>>>> them become the release?  So that seems like exactly the right
> > >>> time
> > >>>> to
> > >>>>>> make
> > >>>>>>> RC releases?
> > >>>>>>>
> > >>>>>>>> On May 27, 2020, at 2:45 PM, Joshua McKenzie <
> > >>> jmckenzie@apache.org
> > >>>>>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>> I think we're all on the same page here; I was focusing more on
> > >>> the
> > >>>>>>> release
> > >>>>>>>> lifecycles and sequencing than the entire version cycle. Good to
> > >>>>>> broaden
> > >>>>>>>> scope I think.
> > >>>>>>>>
> > >>>>>>>> One thing we're not considering is the separation of API changes
> > >>>> from
> > >>>>>>> major
> > >>>>>>>> changes and how that intersects with release milestones.
> > >>>>>>>>
> > >>>>>>>> Meaning:
> > >>>>>>>> 1. alpha phase
> > >>>>>>>> 2. Milestone: API freeze (all API changes pushed to next major)
> > >>>>>>>> 3. beta phase
> > >>>>>>>> 4. Verification phase (all major disruptive pushed to next
> > >>> major)
> > >>>>>>>>
> > >>>>>>>> A clear point to cut RC's doesn't surface from the above for me.
> > >>>>>>> Releasing
> > >>>>>>>> an RC before broad verification seems wrong, and cutting an RC
> > >>>> after
> > >>>>>> the
> > >>>>>>> 4
> > >>>>>>>> points above may as well be GA because it's all known scope.
> > >>>>>>>>
> > >>>>>>>> Thoughts?
> > >>>>>>>>
> > >>>>>>>> On Wed, May 27, 2020 at 3:28 PM Scott Andreas <
> > >>>> scott@paradoxica.net>
> > >>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> That makes sense to me, yep.
> > >>>>>>>>>
> > >>>>>>>>> My hope and expectation is that the time required for
> > >>>> "verification
> > >>>>>>> work"
> > >>>>>>>>> will shrink dramatically in the not too distant future -
> > >>> ideally
> > >>>> to
> > >>>>> a
> > >>>>>>>>> period of less than a month. In this world, the cost of missing
> > >>>> one
> > >>>>>>> train
> > >>>>>>>>> is reduced to catching the next one.
> > >>>>>>>>>
> > >>>>>>>>> One of the main goals in shifting focus from "testing" and
> > >>> "test
> > >>>>>> plans"
> > >>>>>>> to
> > >>>>>>>>> "test engineering" is automating as many aspects of release
> > >>>>>>> qualification
> > >>>>>>>>> as possible, with an asymptotic ideal as a function of compute
> > >>>>>> capacity
> > >>>>>>> and
> > >>>>>>>>> time. While such automation will never be complete (it's likely
> > >>>> that
> > >>>>>>>>> development of new features will/must include qualification
> > >>> infra
> > >>>>>>> changes
> > >>>>>>>>> to exercise them), if we're able to apply the same rigor to
> > >>> major
> > >>>>>>> releases
> > >>>>>>>>> as we are to patchlevel builds with little incremental effort,
> > >>> I'd
> > >>>>> be
> > >>>>>>>>> thrilled.
> > >>>>>>>>>
> > >>>>>>>>> This is mostly a way of saying:
> > >>>>>>>>> – I like the cadence/sequencing Benedict proposes below.
> > >>>>>>>>> – I think improvements in test engineering can reduce/eliminate
> > >>>>>>>>> invalidation and may increase the scope of what can be a
> > >>> candidate
> > >>>>> for
> > >>>>>>>>> merge on a given branch
> > >>>>>>>>> – And if not, the cost of missing the train is lower because
> > >>> we'll
> > >>>>> be
> > >>>>>>> able
> > >>>>>>>>> to deliver major releases more often.
> > >>>>>>>>>
> > >>>>>>>>> Scott
> > >>>>>>>>>
> > >>>>>>>>> ________________________________________
> > >>>>>>>>> From: Jeremiah D Jordan <je...@gmail.com>
> > >>>>>>>>> Sent: Wednesday, May 27, 2020 11:54 AM
> > >>>>>>>>> To: Cassandra DEV
> > >>>>>>>>> Subject: Re: [DISCUSS] CASSANDRA-13994
> > >>>>>>>>>
> > >>>>>>>>> +1 strongly agree.  If we aren’t going to let something go into
> > >>>>> 4.0.0
> > >>>>>>>>> because it would "invalidate testing” then we can not let such
> > >>> a
> > >>>>> thing
> > >>>>>>> go
> > >>>>>>>>> into 4.0.1 unless we plan to re-do said testing for the patch
> > >>>>> release.
> > >>>>>>>>>
> > >>>>>>>>>> On May 27, 2020, at 1:31 PM, Benedict Elliott Smith <
> > >>>>>>> benedict@apache.org>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> I'm being told this still isn't clear, so let me try in a
> > >>>>>> bullet-point
> > >>>>>>>>> timeline:
> > >>>>>>>>>>
> > >>>>>>>>>> * 4.0 Beta
> > >>>>>>>>>> * 4.0 Verification Work
> > >>>>>>>>>> * [Merge Window]
> > >>>>>>>>>> * 4.0 GA
> > >>>>>>>>>> * 4.0 Minor Releases
> > >>>>>>>>>> * ...
> > >>>>>>>>>> * 5.0 Dev
> > >>>>>>>>>> * ...
> > >>>>>>>>>> * 5.0 Verification Work
> > >>>>>>>>>> * GA 5.0
> > >>>>>>>>>>
> > >>>>>>>>>> I think that anything that is prohibited from "[Merge Window]"
> > >>>>>> because
> > >>>>>>>>> it invalidates "4.0 Verification Work" must also be prohibited
> > >>>> until
> > >>>>>>> "5.0
> > >>>>>>>>> Dev" because the next equivalent work that can now validate it
> > >>>>> occurs
> > >>>>>>> only
> > >>>>>>>>> at "5.0 Verification Work"
> > >>>>>>>>>>
> > >>>>>>>>>> On 27/05/2020, 19:05, "Benedict Elliott Smith" <
> > >>>>> benedict@apache.org
> > >>>>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>  I'm not sure if I communicated my point very well.  I mean
> > >>> to
> > >>>> say
> > >>>>>>>>> that if the reason we are prohibiting a patch to land
> > >>> post-beta is
> > >>>>>>> because
> > >>>>>>>>> it invalidates work we only perform pre-ga, then it probably
> > >>>> should
> > >>>>>> not
> > >>>>>>> be
> > >>>>>>>>> permitted to land post-ga either, since it must also invalidate
> > >>>> the
> > >>>>>> same
> > >>>>>>>>> work?
> > >>>>>>>>>>
> > >>>>>>>>>>  That is to say, if we're comfortable with work landing
> > >>> post-ga
> > >>>>>>>>> because we believe it to be safe to release without our
> > >>>>>>> pre-major-release
> > >>>>>>>>> verification, we should be comfortable with it landing at any
> > >>> time
> > >>>>>>> pre-ga
> > >>>>>>>>> too.  Anything else seems inconsistent to me, and we should
> > >>>> examine
> > >>>>>> what
> > >>>>>>>>> assumptions we're making that permit this inconsistency to
> > >>> arise.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>  On 27/05/2020, 18:49, "Joshua McKenzie" <
> > >>> jmckenzie@apache.org
> > >>>>>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> because it invalidates our pre-release verification, then it
> > >>>>> should
> > >>>>>>> not
> > >>>>>>>>>>> land
> > >>>>>>>>>>
> > >>>>>>>>>>      until we next perform pre-release verification
> > >>>>>>>>>>
> > >>>>>>>>>>      At least for me there's a little softness around our
> > >>>>> collective
> > >>>>>>>>> alignment
> > >>>>>>>>>>      on when pre-release verification takes place. If it's
> > >>>> between
> > >>>>>>>>> alpha-1 and
> > >>>>>>>>>>      ga we don't want changes that would invalidate those
> > >>>> changes
> > >>>>> to
> > >>>>>>>>> land during
> > >>>>>>>>>>      that time frame. Different for beta-1 to ga. We also
> > >>> risk
> > >>>>>>>>> invalidating
> > >>>>>>>>>>      testing if we do any of that testing before wherever
> > >>> that
> > >>>>>> cutoff
> > >>>>>>>>> is, and a
> > >>>>>>>>>>      lack of clarity on that cutoff further muddies those
> > >>>> waters.
> > >>>>>>>>>>
> > >>>>>>>>>>      My very loosely held perspective is that beta-1 to ga is
> > >>>> the
> > >>>>>>>>> window in
> > >>>>>>>>>>      which we apply the "don't do things that will invalidate
> > >>>>>>>>> verification", and
> > >>>>>>>>>>      we plan to do that verification during the beta phase. I
> > >>>>>> *think*
> > >>>>>>>>> this is
> > >>>>>>>>>>      consistent w/the current framing of the lifecycle doc.
> > >>> That
> > >>>>>> being
> > >>>>>>>>> said, I
> > >>>>>>>>>>      don't have strong religion on this so if we collectively
> > >>>> want
> > >>>>>> to
> > >>>>>>>>> call it
> > >>>>>>>>>>      "don't majorly disrupt from alpha-1 to ga", we can
> > >>>> formalize
> > >>>>>> that
> > >>>>>>>>> in the
> > >>>>>>>>>>      docs and go ahead and triage current open scope for 4.0
> > >>> and
> > >>>>>> move
> > >>>>>>>>> things out.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>      On Wed, May 27, 2020 at 12:59 PM Ekaterina Dimitrova <
> > >>>>>>>>>>      ekaterina.dimitrova@datastax.com> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> Thank you all for your input.
> > >>>>>>>>>>> I think an important topic is again to revise the lifecycle
> > >>> and
> > >>>>>> ensure
> > >>>>>>>>> we
> > >>>>>>>>>>> really have the vision on what is left until beta. I will
> > >>> start
> > >>>> a
> > >>>>>>>>> separate
> > >>>>>>>>>>> thread on the flaky tests situation soon.
> > >>>>>>>>>>>
> > >>>>>>>>>>> For this particular ticket I see a couple of things:
> > >>>>>>>>>>> - There are a lot of deletions of already not used code
> > >>>>>>>>>>> - I implemented it still in alpha as per our agreement that
> > >>> this
> > >>>>>> will
> > >>>>>>>>> give
> > >>>>>>>>>>> us enough time for testing. Probably Dinesh as a reviewer can
> > >>>> give
> > >>>>>>> some
> > >>>>>>>>>>> valuable feedback/opinion on the patch.
> > >>>>>>>>>>> - It definitely touches around important places but the
> > >>>> important
> > >>>>>>> thing
> > >>>>>>>>> is
> > >>>>>>>>>>> to see how exactly it touches, I think
> > >>>>>>>>>>> - Considering it for alpha before the major testing in beta
> > >>>> sounds
> > >>>>>>>>>>> reasonable to me but I guess it also depends on people
> > >>>>> availability
> > >>>>>> to
> > >>>>>>>>>>> review it in detail and the exact test plans afterwards
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Wed, 27 May 2020 at 7:14, Benedict Elliott Smith <
> > >>>>>>>>> benedict@apache.org>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> I think our "pre-beta" criteria should also be our "not in a
> > >>>>> major"
> > >>>>>>>>>>>> criteria.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> If work is prohibited because it invalidates our pre-release
> > >>>>>>>>>>> verification,
> > >>>>>>>>>>>> then it should not land until we next perform pre-release
> > >>>>>>> verification,
> > >>>>>>>>>>>> which only currently happens once per major.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This could mean either landing less in a major, or
> > >>> permitting
> > >>>>> more
> > >>>>>> in
> > >>>>>>>>>>> beta
> > >>>>>>>>>>>> etc.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> On 26/05/2020, 19:24, "Joshua McKenzie" <
> > >>> jmckenzie@apache.org
> > >>>>>
> > >>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>  I think an interesting question that informs when to stop
> > >>>>>> accepting
> > >>>>>>>>>>>>  specific changes in a release is when we expect any
> > >>> extensive
> > >>>>>>>>>>>> pre-release
> > >>>>>>>>>>>>  testing to take place.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>  If we go by our release lifecycle, gutting deprecated code
> > >>>>> seems
> > >>>>>>>>>>>> compatible
> > >>>>>>>>>>>>  w/Alpha but I wouldn't endorse merging it into Beta:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>
> > >>>>
> > https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
> > >>>>>>> .
> > >>>>>>>>>>>>  Since almost all of the 40_quality_testing epic stuff is
> > >>> also
> > >>>>>> beta
> > >>>>>>>>>>>> phase
> > >>>>>>>>>>>>  and hasn't really taken off yet, it also seems like there
> > >>>> will
> > >>>>> be
> > >>>>>>>>>>>> extensive
> > >>>>>>>>>>>>  testing after this phase transition.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>  All that being said, I'd advocate for marking FixVer 4.x
> > >>> to
> > >>>>>>> indicate
> > >>>>>>>>>>>>  optionality and disallow merge of tickets like this after
> > >>>> we're
> > >>>>>>> done
> > >>>>>>>>>>>>  w/alpha phase in keeping w/our lifecycle doc in general.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>  Does that make sense? Should we consider revisiting and
> > >>>>> revising
> > >>>>>>> the
> > >>>>>>>>>>>>  lifecycle doc re: larger deprecation / changes and cycle
> > >>>>> stages?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>  On Tue, May 26, 2020 at 12:53 PM Oleksandr Petrov <
> > >>>>>>>>>>>>  oleksandr.petrov@gmail.com> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>>> 1) Would you block the release over this ticket?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I would definitely not block the release on this ticket.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 2) Would you prioritize this ticket over testing?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Same here, I would prioritise testing.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 3) Does fixing this ticket make 4.0 a more stable release?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> I wanted to give some context: I wrote that in August 2018.
> > >>>>> While
> > >>>>>> I
> > >>>>>>>>>>>> still
> > >>>>>>>>>>>>> believe it is important to get rid of this code, I'm
> > >>>> disinclined
> > >>>>>> to
> > >>>>>>>>>>>> merge
> > >>>>>>>>>>>>> it into 4.0.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Given that the patch is rather big (421 additions and 1,480
> > >>>>>>>>>>>> deletions) and
> > >>>>>>>>>>>>> touches many important places, including parser, I would be
> > >>>>>>>>>>> extremely
> > >>>>>>>>>>>>> cautious to merge it that late in release cycle. It would
> > >>> be
> > >>>>> great
> > >>>>>>>>>>>> to also
> > >>>>>>>>>>>>> hear arguments that would justify the risk.
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> Thank you for starting this discussion,
> > >>>>>>>>>>>>> -- Alex
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> On Tue, May 26, 2020 at 5:20 PM Ekaterina Dimitrova <
> > >>>>>>>>>>>>> ekaterina.dimitrova@datastax.com> wrote:
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Dear all,
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Following the ticket review sent on 12th May I wanted to
> > >>>> bring
> > >>>>> up
> > >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-13994:
> > >>>> Remove
> > >>>>>>>>>>>> COMPACT
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> STORAGE internals before 4.0 release.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> It is already under review by Dinesh Joshi and Alex
> > >>> Petrov.
> > >>>>> Not a
> > >>>>>>>>>>>>>> blocker but already under review.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Below are my responses to the questions brought up.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 1) Would you block the release over this
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> ticket? - probably not
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 2) Would you prioritize this ticket over testing? -
> > >>> already
> > >>>>>>>>>>>>>> implemented but if there are some big changes needed after
> > >>>> the
> > >>>>>>>>>>>> review,
> > >>>>>>>>>>>>>> I doubt it we will want to prioritize over the testing
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> 3) Does fixing
> > >>>>>>>>>>>>>> this ticket make 4.0 a more stable release? - I will just
> > >>>> cite
> > >>>>>>>>>>> Alex
> > >>>>>>>>>>>>>> Petrov who reported this Jira and I think the rest of us
> > >>>> would
> > >>>>>>>>>>>> agree
> > >>>>>>>>>>>>>> with him here.
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> "I would say it's quite important to clean up compact
> > >>> storage
> > >>>>>>>>>>>>>> internals in 4.0 before the release. It should have no
> > >>>> visible
> > >>>>>>>>>>>>>> side-effects, but it'd be very good to have as it
> > >>> simplifies
> > >>>>>>>>>>>> multiple
> > >>>>>>>>>>>>>> code paths."
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>> Ekaterina Dimitrova
> > >>>>>>>>>>>>>> e. ekaterina.dimitrova@datastax.com
> > >>>>>>>>>>>>>> w. www.datastax.com
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>> --
> > >>>>>>>>>>>>> alex p
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>
> > >>> ---------------------------------------------------------------------
> > >>>>>>>>>>>> To unsubscribe, e-mail:
> > >>> dev-unsubscribe@cassandra.apache.org
> > >>>>>>>>>>>> For additional commands, e-mail:
> > >>> dev-help@cassandra.apache.org
> > >>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>
> > >>> ---------------------------------------------------------------------
> > >>>>>>>>>>  To unsubscribe, e-mail:
> > >>> dev-unsubscribe@cassandra.apache.org
> > >>>>>>>>>>  For additional commands, e-mail:
> > >>> dev-help@cassandra.apache.org
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>
> ---------------------------------------------------------------------
> > >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > >>>>>>>>>> For additional commands, e-mail:
> > >>> dev-help@cassandra.apache.org
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>
> ---------------------------------------------------------------------
> > >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > >>>>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>
> ---------------------------------------------------------------------
> > >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > >>>>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>>>>>>
> > >>> ---------------------------------------------------------------------
> > >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > >>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> > >>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >
> >
>

Re: [DISCUSS] CASSANDRA-13994

[Joshua McKenzie <jm...@apache.org>]

I don't think we really reached a consensus on this one. Some folks
certainly vocalized a preference for not removing in 4.0, but I read it as
debate still ongoing then losing momentum. Specifically thinking of
Sylvain's comment here that kind of dead-ended:
https://issues.apache.org/jira/browse/CASSANDRA-13994?focusedCommentId=17138278&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17138278

fwiw I'm personally fine w/us deferring this until after 4.0 assuming we
cut a branch for beta so changes like this don't bitrot.

On Wed, Jun 24, 2020 at 3:03 PM Dinesh Joshi <dj...@apache.org> wrote:

> I might be missing some context here but I thought we did not intend to
> remove compact storage for 4.0 as it was deemed to be too invasive at this
> point. Did that decision change?
>
> Dinesh
>
> > On Jun 24, 2020, at 10:48 AM, Ekaterina Dimitrova <
> ekaterina.dimitrova@datastax.com> wrote:
> >
> > Dear all,
> > As we talked yesterday during our meeting, I am bringing back to the
> table
> > CASSANDRA-13994 <https://issues.apache.org/jira/browse/CASSANDRA-13994>.
> > Description
> >
> > 4.0 comes without thrift (after CASSANDRA-11115
> > <https://issues.apache.org/jira/browse/CASSANDRA-11115>) and COMPACT
> > STORAGE (after CASSANDRA-10857
> > <https://issues.apache.org/jira/browse/CASSANDRA-10857>), and since
> Compact
> > Storage flags are now disabled, all of the related functionality is
> useless.
> >
> > There are still some things to consider:
> > 1. One of the system tables (built indexes) was compact. For now, we just
> > added value column to it to make sure it's backwards-compatible, but we
> > might want to make sure it's just a "normal" table and doesn't have
> > redundant columns.
> > 2. Compact Tables were building indexes in KEYS mode. Removing it is
> > trivial, but this would mean that all built indexes will be defunct. We
> > could log a warning for now and ask users to migrate off those for now
> and
> > completely remove it from future releases. It's just a couple of classes
> > though.
> >
> >
> > Thank you Sylvain for the first pass of review.
> >
> > An option, as also Sylvain suggested, is to split the ticket in two
> tickets
> > - clean the Compact storage dead code in this one and open a new ticket
> to
> > handle the indexes.
> >
> > I am pasting here the latest comment, hopefully this makes it easier for
> > the audience:
> >
> > I have made a first pass of review and offered a few remarks above.
> >
> > But I think this ticket is hanging up on us deciding whether removing the
> > KEYS 2ndary index code is ok or not. And this yields, to me, the question
> > of what is the upgrade path to 4.0 for users that still have KEYS index
> > (which, reminder, could only be created with Thrift, but could *used*
> with
> > CQL and thus still be around).
> >
> > Because, while I haven't tested this myself, I suspect we have a hole
> here.
> >
> > Namely, KEYS index were compact tables, and 4.0 does not start if there
> is
> > still compact tables. And while for user tables, user are asked to use
> DROP
> > COMPACT STORAGE before upgrading, this cannot be done on KEYS index
> (there
> > is just no syntax to do it), so unless there is code I'm not aware of
> (and
> > please, someone correct me if I'm wrong), I don't think user can upgrade
> to
> > 4.0 at all if they still have KEYS index. They'd have to drop those index
> > first.
> >
> > So If I'm right here, this technically mean removing the KEYS index code
> in
> > 4.0 is fine, since you cannot upgrade in the first place if you have KEYS
> > index. But the more important question for 4.0 imo is what is the upgrade
> > path for users if they have a KEYS index in 3.X?
> >
> > Currently (without code changes), the only available option I can think
> of
> > is that before upgrade to 4.0, users would have to 1) drop their KEYS
> index
> > and then 2) re-create a "normal" (non-KEYS) equivalent index.
> >
> > Are we comfortable with that being the upgrade path for KEYS index?
> >
> > Personally, I'm not sure I am because this is not a seamless upgrade, as
> > between the 1) and 2) above, there is a window where there is no
> accessible
> > index, so if the user application rely on it, it means a period of
> downtime
> > for the application to perform the upgrade. However, if we want a more
> > seamless upgrade, we need to figure something out, and that probably
> > involve non trivial amounts of code and testing. And, playing devil's
> > advocate, KEYS index being so old, maybe nobody that plans to upgrade to
> > 4.0 have them anymore, and maybe it's not worth bothering?
> >
> > So I could use others opinions here.
> >
> > Tl;dr, this ticket raises the point that "Oops, I'm not sure we have
> though
> > through the question of upgrade to 4.0 for KEYS indexes". And tbc, it's
> not
> > directly related to this ticket, only indirectly, but it is still
> something
> > we need to figure out. And I'd say, before 4.0-alpha. But I'm happy to
> > create a separate ticket specific to that question if that helps.
> >
> >
> > Ekaterina
> >
> >>
> >>
> >> It was moved to alpha following the lifecycle document and the latest
> >> discussion on the ML. Sylvain made a review yesterday. To me the biggest
> >> question is about the indexing. (Posted questions on the ticket itself,
> >> waiting for opinion and agreement) For the rest of the ticket I support
> >> Sylvain that it is primarily removal of dead code.
> >>
> >> My understanding is either it lands in alpha before the big testing in
> >> beta or it is delayed for v.5
> >>
> >> Ekaterina
> >>
> >> On Wed, 10 Jun 2020 at 12:59, Joshua McKenzie <jm...@apache.org>
> >> wrote:
> >>
> >>> Where did we land on inclusion of this ticket in 4.0? Since it's now
> >>> flagged as blocking beta rel, I am now very interested in this
> question.
> >>>
> >>> :)
> >>>
> >>>
> >>> On Wed, May 27, 2020 at 7:52 PM Jordan West <jo...@gmail.com>
> wrote:
> >>>
> >>>> On Wed, May 27, 2020 at 1:23 PM Joshua McKenzie <jmckenzie@apache.org
> >
> >>>> wrote:
> >>>>
> >>>>> Maybe. Do we just time box, say we're going to cut an RC and give it
> 4
> >>>>> weeks, if nothing awful surfaces we GA?
> >>>>>
> >>>>
> >>>> I've seen that work well in the past on other projects. I agree with
> the
> >>>> notion that RCs are real candidates for release if no one finds issues
> >>> with
> >>>> them. Ideally we would have as little RCs as possible and have more
> >>>> alphas/betas.
> >>>>
> >>>>>
> >>>>> On Wed, May 27, 2020 at 4:12 PM Brandon Williams <dr...@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>>> Absolutely my understanding.
> >>>>>>
> >>>>>> On Wed, May 27, 2020, 2:49 PM Jeremiah D Jordan <
> >>>>> jeremiah.jordan@gmail.com
> >>>>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>>> A clear point to cut RC's doesn't surface from the above for me.
> >>>>>>> Releasing
> >>>>>>>> an RC before broad verification seems wrong, and cutting an RC
> >>>> after
> >>>>>> the
> >>>>>>> 4
> >>>>>>>> points above may as well be GA because it's all known scope.
> >>>>>>>
> >>>>>>> Isn’t the whole point of an RC is that it could be the GA?  It is
> >>> a
> >>>>>>> “release candidate”, meaning if no one finds any issues with it,
> >>> that
> >>>>> can
> >>>>>>> them become the release?  So that seems like exactly the right
> >>> time
> >>>> to
> >>>>>> make
> >>>>>>> RC releases?
> >>>>>>>
> >>>>>>>> On May 27, 2020, at 2:45 PM, Joshua McKenzie <
> >>> jmckenzie@apache.org
> >>>>>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> I think we're all on the same page here; I was focusing more on
> >>> the
> >>>>>>> release
> >>>>>>>> lifecycles and sequencing than the entire version cycle. Good to
> >>>>>> broaden
> >>>>>>>> scope I think.
> >>>>>>>>
> >>>>>>>> One thing we're not considering is the separation of API changes
> >>>> from
> >>>>>>> major
> >>>>>>>> changes and how that intersects with release milestones.
> >>>>>>>>
> >>>>>>>> Meaning:
> >>>>>>>> 1. alpha phase
> >>>>>>>> 2. Milestone: API freeze (all API changes pushed to next major)
> >>>>>>>> 3. beta phase
> >>>>>>>> 4. Verification phase (all major disruptive pushed to next
> >>> major)
> >>>>>>>>
> >>>>>>>> A clear point to cut RC's doesn't surface from the above for me.
> >>>>>>> Releasing
> >>>>>>>> an RC before broad verification seems wrong, and cutting an RC
> >>>> after
> >>>>>> the
> >>>>>>> 4
> >>>>>>>> points above may as well be GA because it's all known scope.
> >>>>>>>>
> >>>>>>>> Thoughts?
> >>>>>>>>
> >>>>>>>> On Wed, May 27, 2020 at 3:28 PM Scott Andreas <
> >>>> scott@paradoxica.net>
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> That makes sense to me, yep.
> >>>>>>>>>
> >>>>>>>>> My hope and expectation is that the time required for
> >>>> "verification
> >>>>>>> work"
> >>>>>>>>> will shrink dramatically in the not too distant future -
> >>> ideally
> >>>> to
> >>>>> a
> >>>>>>>>> period of less than a month. In this world, the cost of missing
> >>>> one
> >>>>>>> train
> >>>>>>>>> is reduced to catching the next one.
> >>>>>>>>>
> >>>>>>>>> One of the main goals in shifting focus from "testing" and
> >>> "test
> >>>>>> plans"
> >>>>>>> to
> >>>>>>>>> "test engineering" is automating as many aspects of release
> >>>>>>> qualification
> >>>>>>>>> as possible, with an asymptotic ideal as a function of compute
> >>>>>> capacity
> >>>>>>> and
> >>>>>>>>> time. While such automation will never be complete (it's likely
> >>>> that
> >>>>>>>>> development of new features will/must include qualification
> >>> infra
> >>>>>>> changes
> >>>>>>>>> to exercise them), if we're able to apply the same rigor to
> >>> major
> >>>>>>> releases
> >>>>>>>>> as we are to patchlevel builds with little incremental effort,
> >>> I'd
> >>>>> be
> >>>>>>>>> thrilled.
> >>>>>>>>>
> >>>>>>>>> This is mostly a way of saying:
> >>>>>>>>> – I like the cadence/sequencing Benedict proposes below.
> >>>>>>>>> – I think improvements in test engineering can reduce/eliminate
> >>>>>>>>> invalidation and may increase the scope of what can be a
> >>> candidate
> >>>>> for
> >>>>>>>>> merge on a given branch
> >>>>>>>>> – And if not, the cost of missing the train is lower because
> >>> we'll
> >>>>> be
> >>>>>>> able
> >>>>>>>>> to deliver major releases more often.
> >>>>>>>>>
> >>>>>>>>> Scott
> >>>>>>>>>
> >>>>>>>>> ________________________________________
> >>>>>>>>> From: Jeremiah D Jordan <je...@gmail.com>
> >>>>>>>>> Sent: Wednesday, May 27, 2020 11:54 AM
> >>>>>>>>> To: Cassandra DEV
> >>>>>>>>> Subject: Re: [DISCUSS] CASSANDRA-13994
> >>>>>>>>>
> >>>>>>>>> +1 strongly agree.  If we aren’t going to let something go into
> >>>>> 4.0.0
> >>>>>>>>> because it would "invalidate testing” then we can not let such
> >>> a
> >>>>> thing
> >>>>>>> go
> >>>>>>>>> into 4.0.1 unless we plan to re-do said testing for the patch
> >>>>> release.
> >>>>>>>>>
> >>>>>>>>>> On May 27, 2020, at 1:31 PM, Benedict Elliott Smith <
> >>>>>>> benedict@apache.org>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> I'm being told this still isn't clear, so let me try in a
> >>>>>> bullet-point
> >>>>>>>>> timeline:
> >>>>>>>>>>
> >>>>>>>>>> * 4.0 Beta
> >>>>>>>>>> * 4.0 Verification Work
> >>>>>>>>>> * [Merge Window]
> >>>>>>>>>> * 4.0 GA
> >>>>>>>>>> * 4.0 Minor Releases
> >>>>>>>>>> * ...
> >>>>>>>>>> * 5.0 Dev
> >>>>>>>>>> * ...
> >>>>>>>>>> * 5.0 Verification Work
> >>>>>>>>>> * GA 5.0
> >>>>>>>>>>
> >>>>>>>>>> I think that anything that is prohibited from "[Merge Window]"
> >>>>>> because
> >>>>>>>>> it invalidates "4.0 Verification Work" must also be prohibited
> >>>> until
> >>>>>>> "5.0
> >>>>>>>>> Dev" because the next equivalent work that can now validate it
> >>>>> occurs
> >>>>>>> only
> >>>>>>>>> at "5.0 Verification Work"
> >>>>>>>>>>
> >>>>>>>>>> On 27/05/2020, 19:05, "Benedict Elliott Smith" <
> >>>>> benedict@apache.org
> >>>>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>  I'm not sure if I communicated my point very well.  I mean
> >>> to
> >>>> say
> >>>>>>>>> that if the reason we are prohibiting a patch to land
> >>> post-beta is
> >>>>>>> because
> >>>>>>>>> it invalidates work we only perform pre-ga, then it probably
> >>>> should
> >>>>>> not
> >>>>>>> be
> >>>>>>>>> permitted to land post-ga either, since it must also invalidate
> >>>> the
> >>>>>> same
> >>>>>>>>> work?
> >>>>>>>>>>
> >>>>>>>>>>  That is to say, if we're comfortable with work landing
> >>> post-ga
> >>>>>>>>> because we believe it to be safe to release without our
> >>>>>>> pre-major-release
> >>>>>>>>> verification, we should be comfortable with it landing at any
> >>> time
> >>>>>>> pre-ga
> >>>>>>>>> too.  Anything else seems inconsistent to me, and we should
> >>>> examine
> >>>>>> what
> >>>>>>>>> assumptions we're making that permit this inconsistency to
> >>> arise.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>  On 27/05/2020, 18:49, "Joshua McKenzie" <
> >>> jmckenzie@apache.org
> >>>>>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> because it invalidates our pre-release verification, then it
> >>>>> should
> >>>>>>> not
> >>>>>>>>>>> land
> >>>>>>>>>>
> >>>>>>>>>>      until we next perform pre-release verification
> >>>>>>>>>>
> >>>>>>>>>>      At least for me there's a little softness around our
> >>>>> collective
> >>>>>>>>> alignment
> >>>>>>>>>>      on when pre-release verification takes place. If it's
> >>>> between
> >>>>>>>>> alpha-1 and
> >>>>>>>>>>      ga we don't want changes that would invalidate those
> >>>> changes
> >>>>> to
> >>>>>>>>> land during
> >>>>>>>>>>      that time frame. Different for beta-1 to ga. We also
> >>> risk
> >>>>>>>>> invalidating
> >>>>>>>>>>      testing if we do any of that testing before wherever
> >>> that
> >>>>>> cutoff
> >>>>>>>>> is, and a
> >>>>>>>>>>      lack of clarity on that cutoff further muddies those
> >>>> waters.
> >>>>>>>>>>
> >>>>>>>>>>      My very loosely held perspective is that beta-1 to ga is
> >>>> the
> >>>>>>>>> window in
> >>>>>>>>>>      which we apply the "don't do things that will invalidate
> >>>>>>>>> verification", and
> >>>>>>>>>>      we plan to do that verification during the beta phase. I
> >>>>>> *think*
> >>>>>>>>> this is
> >>>>>>>>>>      consistent w/the current framing of the lifecycle doc.
> >>> That
> >>>>>> being
> >>>>>>>>> said, I
> >>>>>>>>>>      don't have strong religion on this so if we collectively
> >>>> want
> >>>>>> to
> >>>>>>>>> call it
> >>>>>>>>>>      "don't majorly disrupt from alpha-1 to ga", we can
> >>>> formalize
> >>>>>> that
> >>>>>>>>> in the
> >>>>>>>>>>      docs and go ahead and triage current open scope for 4.0
> >>> and
> >>>>>> move
> >>>>>>>>> things out.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>      On Wed, May 27, 2020 at 12:59 PM Ekaterina Dimitrova <
> >>>>>>>>>>      ekaterina.dimitrova@datastax.com> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Thank you all for your input.
> >>>>>>>>>>> I think an important topic is again to revise the lifecycle
> >>> and
> >>>>>> ensure
> >>>>>>>>> we
> >>>>>>>>>>> really have the vision on what is left until beta. I will
> >>> start
> >>>> a
> >>>>>>>>> separate
> >>>>>>>>>>> thread on the flaky tests situation soon.
> >>>>>>>>>>>
> >>>>>>>>>>> For this particular ticket I see a couple of things:
> >>>>>>>>>>> - There are a lot of deletions of already not used code
> >>>>>>>>>>> - I implemented it still in alpha as per our agreement that
> >>> this
> >>>>>> will
> >>>>>>>>> give
> >>>>>>>>>>> us enough time for testing. Probably Dinesh as a reviewer can
> >>>> give
> >>>>>>> some
> >>>>>>>>>>> valuable feedback/opinion on the patch.
> >>>>>>>>>>> - It definitely touches around important places but the
> >>>> important
> >>>>>>> thing
> >>>>>>>>> is
> >>>>>>>>>>> to see how exactly it touches, I think
> >>>>>>>>>>> - Considering it for alpha before the major testing in beta
> >>>> sounds
> >>>>>>>>>>> reasonable to me but I guess it also depends on people
> >>>>> availability
> >>>>>> to
> >>>>>>>>>>> review it in detail and the exact test plans afterwards
> >>>>>>>>>>>
> >>>>>>>>>>> On Wed, 27 May 2020 at 7:14, Benedict Elliott Smith <
> >>>>>>>>> benedict@apache.org>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I think our "pre-beta" criteria should also be our "not in a
> >>>>> major"
> >>>>>>>>>>>> criteria.
> >>>>>>>>>>>>
> >>>>>>>>>>>> If work is prohibited because it invalidates our pre-release
> >>>>>>>>>>> verification,
> >>>>>>>>>>>> then it should not land until we next perform pre-release
> >>>>>>> verification,
> >>>>>>>>>>>> which only currently happens once per major.
> >>>>>>>>>>>>
> >>>>>>>>>>>> This could mean either landing less in a major, or
> >>> permitting
> >>>>> more
> >>>>>> in
> >>>>>>>>>>> beta
> >>>>>>>>>>>> etc.
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 26/05/2020, 19:24, "Joshua McKenzie" <
> >>> jmckenzie@apache.org
> >>>>>
> >>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>  I think an interesting question that informs when to stop
> >>>>>> accepting
> >>>>>>>>>>>>  specific changes in a release is when we expect any
> >>> extensive
> >>>>>>>>>>>> pre-release
> >>>>>>>>>>>>  testing to take place.
> >>>>>>>>>>>>
> >>>>>>>>>>>>  If we go by our release lifecycle, gutting deprecated code
> >>>>> seems
> >>>>>>>>>>>> compatible
> >>>>>>>>>>>>  w/Alpha but I wouldn't endorse merging it into Beta:
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>
> >>>>
> https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
> >>>>>>> .
> >>>>>>>>>>>>  Since almost all of the 40_quality_testing epic stuff is
> >>> also
> >>>>>> beta
> >>>>>>>>>>>> phase
> >>>>>>>>>>>>  and hasn't really taken off yet, it also seems like there
> >>>> will
> >>>>> be
> >>>>>>>>>>>> extensive
> >>>>>>>>>>>>  testing after this phase transition.
> >>>>>>>>>>>>
> >>>>>>>>>>>>  All that being said, I'd advocate for marking FixVer 4.x
> >>> to
> >>>>>>> indicate
> >>>>>>>>>>>>  optionality and disallow merge of tickets like this after
> >>>> we're
> >>>>>>> done
> >>>>>>>>>>>>  w/alpha phase in keeping w/our lifecycle doc in general.
> >>>>>>>>>>>>
> >>>>>>>>>>>>  Does that make sense? Should we consider revisiting and
> >>>>> revising
> >>>>>>> the
> >>>>>>>>>>>>  lifecycle doc re: larger deprecation / changes and cycle
> >>>>> stages?
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>  On Tue, May 26, 2020 at 12:53 PM Oleksandr Petrov <
> >>>>>>>>>>>>  oleksandr.petrov@gmail.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>>>> 1) Would you block the release over this ticket?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I would definitely not block the release on this ticket.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2) Would you prioritize this ticket over testing?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Same here, I would prioritise testing.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> 3) Does fixing this ticket make 4.0 a more stable release?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I wanted to give some context: I wrote that in August 2018.
> >>>>> While
> >>>>>> I
> >>>>>>>>>>>> still
> >>>>>>>>>>>>> believe it is important to get rid of this code, I'm
> >>>> disinclined
> >>>>>> to
> >>>>>>>>>>>> merge
> >>>>>>>>>>>>> it into 4.0.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Given that the patch is rather big (421 additions and 1,480
> >>>>>>>>>>>> deletions) and
> >>>>>>>>>>>>> touches many important places, including parser, I would be
> >>>>>>>>>>> extremely
> >>>>>>>>>>>>> cautious to merge it that late in release cycle. It would
> >>> be
> >>>>> great
> >>>>>>>>>>>> to also
> >>>>>>>>>>>>> hear arguments that would justify the risk.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Thank you for starting this discussion,
> >>>>>>>>>>>>> -- Alex
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Tue, May 26, 2020 at 5:20 PM Ekaterina Dimitrova <
> >>>>>>>>>>>>> ekaterina.dimitrova@datastax.com> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> Dear all,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Following the ticket review sent on 12th May I wanted to
> >>>> bring
> >>>>> up
> >>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-13994:
> >>>> Remove
> >>>>>>>>>>>> COMPACT
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> STORAGE internals before 4.0 release.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> It is already under review by Dinesh Joshi and Alex
> >>> Petrov.
> >>>>> Not a
> >>>>>>>>>>>>>> blocker but already under review.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Below are my responses to the questions brought up.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 1) Would you block the release over this
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> ticket? - probably not
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 2) Would you prioritize this ticket over testing? -
> >>> already
> >>>>>>>>>>>>>> implemented but if there are some big changes needed after
> >>>> the
> >>>>>>>>>>>> review,
> >>>>>>>>>>>>>> I doubt it we will want to prioritize over the testing
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> 3) Does fixing
> >>>>>>>>>>>>>> this ticket make 4.0 a more stable release? - I will just
> >>>> cite
> >>>>>>>>>>> Alex
> >>>>>>>>>>>>>> Petrov who reported this Jira and I think the rest of us
> >>>> would
> >>>>>>>>>>>> agree
> >>>>>>>>>>>>>> with him here.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> "I would say it's quite important to clean up compact
> >>> storage
> >>>>>>>>>>>>>> internals in 4.0 before the release. It should have no
> >>>> visible
> >>>>>>>>>>>>>> side-effects, but it'd be very good to have as it
> >>> simplifies
> >>>>>>>>>>>> multiple
> >>>>>>>>>>>>>> code paths."
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Ekaterina Dimitrova
> >>>>>>>>>>>>>> e. ekaterina.dimitrova@datastax.com
> >>>>>>>>>>>>>> w. www.datastax.com
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> alex p
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>>>>>>> To unsubscribe, e-mail:
> >>> dev-unsubscribe@cassandra.apache.org
> >>>>>>>>>>>> For additional commands, e-mail:
> >>> dev-help@cassandra.apache.org
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>>>>>  To unsubscribe, e-mail:
> >>> dev-unsubscribe@cassandra.apache.org
> >>>>>>>>>>  For additional commands, e-mail:
> >>> dev-help@cassandra.apache.org
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>>>>>>>> For additional commands, e-mail:
> >>> dev-help@cassandra.apache.org
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>> ---------------------------------------------------------------------
> >>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>> ---------------------------------------------------------------------
> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>

Re: [DISCUSS] CASSANDRA-13994

[Dinesh Joshi <dj...@apache.org>]

I might be missing some context here but I thought we did not intend to remove compact storage for 4.0 as it was deemed to be too invasive at this point. Did that decision change?

Dinesh

> On Jun 24, 2020, at 10:48 AM, Ekaterina Dimitrova <ek...@datastax.com> wrote:
> 
> Dear all,
> As we talked yesterday during our meeting, I am bringing back to the table
> CASSANDRA-13994 <https://issues.apache.org/jira/browse/CASSANDRA-13994>.
> Description
> 
> 4.0 comes without thrift (after CASSANDRA-11115
> <https://issues.apache.org/jira/browse/CASSANDRA-11115>) and COMPACT
> STORAGE (after CASSANDRA-10857
> <https://issues.apache.org/jira/browse/CASSANDRA-10857>), and since Compact
> Storage flags are now disabled, all of the related functionality is useless.
> 
> There are still some things to consider:
> 1. One of the system tables (built indexes) was compact. For now, we just
> added value column to it to make sure it's backwards-compatible, but we
> might want to make sure it's just a "normal" table and doesn't have
> redundant columns.
> 2. Compact Tables were building indexes in KEYS mode. Removing it is
> trivial, but this would mean that all built indexes will be defunct. We
> could log a warning for now and ask users to migrate off those for now and
> completely remove it from future releases. It's just a couple of classes
> though.
> 
> 
> Thank you Sylvain for the first pass of review.
> 
> An option, as also Sylvain suggested, is to split the ticket in two tickets
> - clean the Compact storage dead code in this one and open a new ticket to
> handle the indexes.
> 
> I am pasting here the latest comment, hopefully this makes it easier for
> the audience:
> 
> I have made a first pass of review and offered a few remarks above.
> 
> But I think this ticket is hanging up on us deciding whether removing the
> KEYS 2ndary index code is ok or not. And this yields, to me, the question
> of what is the upgrade path to 4.0 for users that still have KEYS index
> (which, reminder, could only be created with Thrift, but could *used* with
> CQL and thus still be around).
> 
> Because, while I haven't tested this myself, I suspect we have a hole here.
> 
> Namely, KEYS index were compact tables, and 4.0 does not start if there is
> still compact tables. And while for user tables, user are asked to use DROP
> COMPACT STORAGE before upgrading, this cannot be done on KEYS index (there
> is just no syntax to do it), so unless there is code I'm not aware of (and
> please, someone correct me if I'm wrong), I don't think user can upgrade to
> 4.0 at all if they still have KEYS index. They'd have to drop those index
> first.
> 
> So If I'm right here, this technically mean removing the KEYS index code in
> 4.0 is fine, since you cannot upgrade in the first place if you have KEYS
> index. But the more important question for 4.0 imo is what is the upgrade
> path for users if they have a KEYS index in 3.X?
> 
> Currently (without code changes), the only available option I can think of
> is that before upgrade to 4.0, users would have to 1) drop their KEYS index
> and then 2) re-create a "normal" (non-KEYS) equivalent index.
> 
> Are we comfortable with that being the upgrade path for KEYS index?
> 
> Personally, I'm not sure I am because this is not a seamless upgrade, as
> between the 1) and 2) above, there is a window where there is no accessible
> index, so if the user application rely on it, it means a period of downtime
> for the application to perform the upgrade. However, if we want a more
> seamless upgrade, we need to figure something out, and that probably
> involve non trivial amounts of code and testing. And, playing devil's
> advocate, KEYS index being so old, maybe nobody that plans to upgrade to
> 4.0 have them anymore, and maybe it's not worth bothering?
> 
> So I could use others opinions here.
> 
> Tl;dr, this ticket raises the point that "Oops, I'm not sure we have though
> through the question of upgrade to 4.0 for KEYS indexes". And tbc, it's not
> directly related to this ticket, only indirectly, but it is still something
> we need to figure out. And I'd say, before 4.0-alpha. But I'm happy to
> create a separate ticket specific to that question if that helps.
> 
> 
> Ekaterina
> 
>> 
>> 
>> It was moved to alpha following the lifecycle document and the latest
>> discussion on the ML. Sylvain made a review yesterday. To me the biggest
>> question is about the indexing. (Posted questions on the ticket itself,
>> waiting for opinion and agreement) For the rest of the ticket I support
>> Sylvain that it is primarily removal of dead code.
>> 
>> My understanding is either it lands in alpha before the big testing in
>> beta or it is delayed for v.5
>> 
>> Ekaterina
>> 
>> On Wed, 10 Jun 2020 at 12:59, Joshua McKenzie <jm...@apache.org>
>> wrote:
>> 
>>> Where did we land on inclusion of this ticket in 4.0? Since it's now
>>> flagged as blocking beta rel, I am now very interested in this question.
>>> 
>>> :)
>>> 
>>> 
>>> On Wed, May 27, 2020 at 7:52 PM Jordan West <jo...@gmail.com> wrote:
>>> 
>>>> On Wed, May 27, 2020 at 1:23 PM Joshua McKenzie <jm...@apache.org>
>>>> wrote:
>>>> 
>>>>> Maybe. Do we just time box, say we're going to cut an RC and give it 4
>>>>> weeks, if nothing awful surfaces we GA?
>>>>> 
>>>> 
>>>> I've seen that work well in the past on other projects. I agree with the
>>>> notion that RCs are real candidates for release if no one finds issues
>>> with
>>>> them. Ideally we would have as little RCs as possible and have more
>>>> alphas/betas.
>>>> 
>>>>> 
>>>>> On Wed, May 27, 2020 at 4:12 PM Brandon Williams <dr...@gmail.com>
>>>> wrote:
>>>>> 
>>>>>> Absolutely my understanding.
>>>>>> 
>>>>>> On Wed, May 27, 2020, 2:49 PM Jeremiah D Jordan <
>>>>> jeremiah.jordan@gmail.com
>>>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>>> A clear point to cut RC's doesn't surface from the above for me.
>>>>>>> Releasing
>>>>>>>> an RC before broad verification seems wrong, and cutting an RC
>>>> after
>>>>>> the
>>>>>>> 4
>>>>>>>> points above may as well be GA because it's all known scope.
>>>>>>> 
>>>>>>> Isn’t the whole point of an RC is that it could be the GA?  It is
>>> a
>>>>>>> “release candidate”, meaning if no one finds any issues with it,
>>> that
>>>>> can
>>>>>>> them become the release?  So that seems like exactly the right
>>> time
>>>> to
>>>>>> make
>>>>>>> RC releases?
>>>>>>> 
>>>>>>>> On May 27, 2020, at 2:45 PM, Joshua McKenzie <
>>> jmckenzie@apache.org
>>>>> 
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> I think we're all on the same page here; I was focusing more on
>>> the
>>>>>>> release
>>>>>>>> lifecycles and sequencing than the entire version cycle. Good to
>>>>>> broaden
>>>>>>>> scope I think.
>>>>>>>> 
>>>>>>>> One thing we're not considering is the separation of API changes
>>>> from
>>>>>>> major
>>>>>>>> changes and how that intersects with release milestones.
>>>>>>>> 
>>>>>>>> Meaning:
>>>>>>>> 1. alpha phase
>>>>>>>> 2. Milestone: API freeze (all API changes pushed to next major)
>>>>>>>> 3. beta phase
>>>>>>>> 4. Verification phase (all major disruptive pushed to next
>>> major)
>>>>>>>> 
>>>>>>>> A clear point to cut RC's doesn't surface from the above for me.
>>>>>>> Releasing
>>>>>>>> an RC before broad verification seems wrong, and cutting an RC
>>>> after
>>>>>> the
>>>>>>> 4
>>>>>>>> points above may as well be GA because it's all known scope.
>>>>>>>> 
>>>>>>>> Thoughts?
>>>>>>>> 
>>>>>>>> On Wed, May 27, 2020 at 3:28 PM Scott Andreas <
>>>> scott@paradoxica.net>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> That makes sense to me, yep.
>>>>>>>>> 
>>>>>>>>> My hope and expectation is that the time required for
>>>> "verification
>>>>>>> work"
>>>>>>>>> will shrink dramatically in the not too distant future -
>>> ideally
>>>> to
>>>>> a
>>>>>>>>> period of less than a month. In this world, the cost of missing
>>>> one
>>>>>>> train
>>>>>>>>> is reduced to catching the next one.
>>>>>>>>> 
>>>>>>>>> One of the main goals in shifting focus from "testing" and
>>> "test
>>>>>> plans"
>>>>>>> to
>>>>>>>>> "test engineering" is automating as many aspects of release
>>>>>>> qualification
>>>>>>>>> as possible, with an asymptotic ideal as a function of compute
>>>>>> capacity
>>>>>>> and
>>>>>>>>> time. While such automation will never be complete (it's likely
>>>> that
>>>>>>>>> development of new features will/must include qualification
>>> infra
>>>>>>> changes
>>>>>>>>> to exercise them), if we're able to apply the same rigor to
>>> major
>>>>>>> releases
>>>>>>>>> as we are to patchlevel builds with little incremental effort,
>>> I'd
>>>>> be
>>>>>>>>> thrilled.
>>>>>>>>> 
>>>>>>>>> This is mostly a way of saying:
>>>>>>>>> – I like the cadence/sequencing Benedict proposes below.
>>>>>>>>> – I think improvements in test engineering can reduce/eliminate
>>>>>>>>> invalidation and may increase the scope of what can be a
>>> candidate
>>>>> for
>>>>>>>>> merge on a given branch
>>>>>>>>> – And if not, the cost of missing the train is lower because
>>> we'll
>>>>> be
>>>>>>> able
>>>>>>>>> to deliver major releases more often.
>>>>>>>>> 
>>>>>>>>> Scott
>>>>>>>>> 
>>>>>>>>> ________________________________________
>>>>>>>>> From: Jeremiah D Jordan <je...@gmail.com>
>>>>>>>>> Sent: Wednesday, May 27, 2020 11:54 AM
>>>>>>>>> To: Cassandra DEV
>>>>>>>>> Subject: Re: [DISCUSS] CASSANDRA-13994
>>>>>>>>> 
>>>>>>>>> +1 strongly agree.  If we aren’t going to let something go into
>>>>> 4.0.0
>>>>>>>>> because it would "invalidate testing” then we can not let such
>>> a
>>>>> thing
>>>>>>> go
>>>>>>>>> into 4.0.1 unless we plan to re-do said testing for the patch
>>>>> release.
>>>>>>>>> 
>>>>>>>>>> On May 27, 2020, at 1:31 PM, Benedict Elliott Smith <
>>>>>>> benedict@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> I'm being told this still isn't clear, so let me try in a
>>>>>> bullet-point
>>>>>>>>> timeline:
>>>>>>>>>> 
>>>>>>>>>> * 4.0 Beta
>>>>>>>>>> * 4.0 Verification Work
>>>>>>>>>> * [Merge Window]
>>>>>>>>>> * 4.0 GA
>>>>>>>>>> * 4.0 Minor Releases
>>>>>>>>>> * ...
>>>>>>>>>> * 5.0 Dev
>>>>>>>>>> * ...
>>>>>>>>>> * 5.0 Verification Work
>>>>>>>>>> * GA 5.0
>>>>>>>>>> 
>>>>>>>>>> I think that anything that is prohibited from "[Merge Window]"
>>>>>> because
>>>>>>>>> it invalidates "4.0 Verification Work" must also be prohibited
>>>> until
>>>>>>> "5.0
>>>>>>>>> Dev" because the next equivalent work that can now validate it
>>>>> occurs
>>>>>>> only
>>>>>>>>> at "5.0 Verification Work"
>>>>>>>>>> 
>>>>>>>>>> On 27/05/2020, 19:05, "Benedict Elliott Smith" <
>>>>> benedict@apache.org
>>>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>  I'm not sure if I communicated my point very well.  I mean
>>> to
>>>> say
>>>>>>>>> that if the reason we are prohibiting a patch to land
>>> post-beta is
>>>>>>> because
>>>>>>>>> it invalidates work we only perform pre-ga, then it probably
>>>> should
>>>>>> not
>>>>>>> be
>>>>>>>>> permitted to land post-ga either, since it must also invalidate
>>>> the
>>>>>> same
>>>>>>>>> work?
>>>>>>>>>> 
>>>>>>>>>>  That is to say, if we're comfortable with work landing
>>> post-ga
>>>>>>>>> because we believe it to be safe to release without our
>>>>>>> pre-major-release
>>>>>>>>> verification, we should be comfortable with it landing at any
>>> time
>>>>>>> pre-ga
>>>>>>>>> too.  Anything else seems inconsistent to me, and we should
>>>> examine
>>>>>> what
>>>>>>>>> assumptions we're making that permit this inconsistency to
>>> arise.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>  On 27/05/2020, 18:49, "Joshua McKenzie" <
>>> jmckenzie@apache.org
>>>>> 
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> because it invalidates our pre-release verification, then it
>>>>> should
>>>>>>> not
>>>>>>>>>>> land
>>>>>>>>>> 
>>>>>>>>>>      until we next perform pre-release verification
>>>>>>>>>> 
>>>>>>>>>>      At least for me there's a little softness around our
>>>>> collective
>>>>>>>>> alignment
>>>>>>>>>>      on when pre-release verification takes place. If it's
>>>> between
>>>>>>>>> alpha-1 and
>>>>>>>>>>      ga we don't want changes that would invalidate those
>>>> changes
>>>>> to
>>>>>>>>> land during
>>>>>>>>>>      that time frame. Different for beta-1 to ga. We also
>>> risk
>>>>>>>>> invalidating
>>>>>>>>>>      testing if we do any of that testing before wherever
>>> that
>>>>>> cutoff
>>>>>>>>> is, and a
>>>>>>>>>>      lack of clarity on that cutoff further muddies those
>>>> waters.
>>>>>>>>>> 
>>>>>>>>>>      My very loosely held perspective is that beta-1 to ga is
>>>> the
>>>>>>>>> window in
>>>>>>>>>>      which we apply the "don't do things that will invalidate
>>>>>>>>> verification", and
>>>>>>>>>>      we plan to do that verification during the beta phase. I
>>>>>> *think*
>>>>>>>>> this is
>>>>>>>>>>      consistent w/the current framing of the lifecycle doc.
>>> That
>>>>>> being
>>>>>>>>> said, I
>>>>>>>>>>      don't have strong religion on this so if we collectively
>>>> want
>>>>>> to
>>>>>>>>> call it
>>>>>>>>>>      "don't majorly disrupt from alpha-1 to ga", we can
>>>> formalize
>>>>>> that
>>>>>>>>> in the
>>>>>>>>>>      docs and go ahead and triage current open scope for 4.0
>>> and
>>>>>> move
>>>>>>>>> things out.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>>      On Wed, May 27, 2020 at 12:59 PM Ekaterina Dimitrova <
>>>>>>>>>>      ekaterina.dimitrova@datastax.com> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Thank you all for your input.
>>>>>>>>>>> I think an important topic is again to revise the lifecycle
>>> and
>>>>>> ensure
>>>>>>>>> we
>>>>>>>>>>> really have the vision on what is left until beta. I will
>>> start
>>>> a
>>>>>>>>> separate
>>>>>>>>>>> thread on the flaky tests situation soon.
>>>>>>>>>>> 
>>>>>>>>>>> For this particular ticket I see a couple of things:
>>>>>>>>>>> - There are a lot of deletions of already not used code
>>>>>>>>>>> - I implemented it still in alpha as per our agreement that
>>> this
>>>>>> will
>>>>>>>>> give
>>>>>>>>>>> us enough time for testing. Probably Dinesh as a reviewer can
>>>> give
>>>>>>> some
>>>>>>>>>>> valuable feedback/opinion on the patch.
>>>>>>>>>>> - It definitely touches around important places but the
>>>> important
>>>>>>> thing
>>>>>>>>> is
>>>>>>>>>>> to see how exactly it touches, I think
>>>>>>>>>>> - Considering it for alpha before the major testing in beta
>>>> sounds
>>>>>>>>>>> reasonable to me but I guess it also depends on people
>>>>> availability
>>>>>> to
>>>>>>>>>>> review it in detail and the exact test plans afterwards
>>>>>>>>>>> 
>>>>>>>>>>> On Wed, 27 May 2020 at 7:14, Benedict Elliott Smith <
>>>>>>>>> benedict@apache.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> I think our "pre-beta" criteria should also be our "not in a
>>>>> major"
>>>>>>>>>>>> criteria.
>>>>>>>>>>>> 
>>>>>>>>>>>> If work is prohibited because it invalidates our pre-release
>>>>>>>>>>> verification,
>>>>>>>>>>>> then it should not land until we next perform pre-release
>>>>>>> verification,
>>>>>>>>>>>> which only currently happens once per major.
>>>>>>>>>>>> 
>>>>>>>>>>>> This could mean either landing less in a major, or
>>> permitting
>>>>> more
>>>>>> in
>>>>>>>>>>> beta
>>>>>>>>>>>> etc.
>>>>>>>>>>>> 
>>>>>>>>>>>> On 26/05/2020, 19:24, "Joshua McKenzie" <
>>> jmckenzie@apache.org
>>>>> 
>>>>>>> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>  I think an interesting question that informs when to stop
>>>>>> accepting
>>>>>>>>>>>>  specific changes in a release is when we expect any
>>> extensive
>>>>>>>>>>>> pre-release
>>>>>>>>>>>>  testing to take place.
>>>>>>>>>>>> 
>>>>>>>>>>>>  If we go by our release lifecycle, gutting deprecated code
>>>>> seems
>>>>>>>>>>>> compatible
>>>>>>>>>>>>  w/Alpha but I wouldn't endorse merging it into Beta:
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>> 
>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
>>>>>>> .
>>>>>>>>>>>>  Since almost all of the 40_quality_testing epic stuff is
>>> also
>>>>>> beta
>>>>>>>>>>>> phase
>>>>>>>>>>>>  and hasn't really taken off yet, it also seems like there
>>>> will
>>>>> be
>>>>>>>>>>>> extensive
>>>>>>>>>>>>  testing after this phase transition.
>>>>>>>>>>>> 
>>>>>>>>>>>>  All that being said, I'd advocate for marking FixVer 4.x
>>> to
>>>>>>> indicate
>>>>>>>>>>>>  optionality and disallow merge of tickets like this after
>>>> we're
>>>>>>> done
>>>>>>>>>>>>  w/alpha phase in keeping w/our lifecycle doc in general.
>>>>>>>>>>>> 
>>>>>>>>>>>>  Does that make sense? Should we consider revisiting and
>>>>> revising
>>>>>>> the
>>>>>>>>>>>>  lifecycle doc re: larger deprecation / changes and cycle
>>>>> stages?
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>>  On Tue, May 26, 2020 at 12:53 PM Oleksandr Petrov <
>>>>>>>>>>>>  oleksandr.petrov@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>>> 1) Would you block the release over this ticket?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would definitely not block the release on this ticket.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 2) Would you prioritize this ticket over testing?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Same here, I would prioritise testing.
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 3) Does fixing this ticket make 4.0 a more stable release?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I wanted to give some context: I wrote that in August 2018.
>>>>> While
>>>>>> I
>>>>>>>>>>>> still
>>>>>>>>>>>>> believe it is important to get rid of this code, I'm
>>>> disinclined
>>>>>> to
>>>>>>>>>>>> merge
>>>>>>>>>>>>> it into 4.0.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Given that the patch is rather big (421 additions and 1,480
>>>>>>>>>>>> deletions) and
>>>>>>>>>>>>> touches many important places, including parser, I would be
>>>>>>>>>>> extremely
>>>>>>>>>>>>> cautious to merge it that late in release cycle. It would
>>> be
>>>>> great
>>>>>>>>>>>> to also
>>>>>>>>>>>>> hear arguments that would justify the risk.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Thank you for starting this discussion,
>>>>>>>>>>>>> -- Alex
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On Tue, May 26, 2020 at 5:20 PM Ekaterina Dimitrova <
>>>>>>>>>>>>> ekaterina.dimitrova@datastax.com> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Following the ticket review sent on 12th May I wanted to
>>>> bring
>>>>> up
>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-13994:
>>>> Remove
>>>>>>>>>>>> COMPACT
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> STORAGE internals before 4.0 release.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> It is already under review by Dinesh Joshi and Alex
>>> Petrov.
>>>>> Not a
>>>>>>>>>>>>>> blocker but already under review.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Below are my responses to the questions brought up.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 1) Would you block the release over this
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> ticket? - probably not
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 2) Would you prioritize this ticket over testing? -
>>> already
>>>>>>>>>>>>>> implemented but if there are some big changes needed after
>>>> the
>>>>>>>>>>>> review,
>>>>>>>>>>>>>> I doubt it we will want to prioritize over the testing
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 3) Does fixing
>>>>>>>>>>>>>> this ticket make 4.0 a more stable release? - I will just
>>>> cite
>>>>>>>>>>> Alex
>>>>>>>>>>>>>> Petrov who reported this Jira and I think the rest of us
>>>> would
>>>>>>>>>>>> agree
>>>>>>>>>>>>>> with him here.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> "I would say it's quite important to clean up compact
>>> storage
>>>>>>>>>>>>>> internals in 4.0 before the release. It should have no
>>>> visible
>>>>>>>>>>>>>> side-effects, but it'd be very good to have as it
>>> simplifies
>>>>>>>>>>>> multiple
>>>>>>>>>>>>>> code paths."
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Ekaterina Dimitrova
>>>>>>>>>>>>>> e. ekaterina.dimitrova@datastax.com
>>>>>>>>>>>>>> w. www.datastax.com
>>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> --
>>>>>>>>>>>>> alex p
>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>> 
>>> ---------------------------------------------------------------------
>>>>>>>>>>>> To unsubscribe, e-mail:
>>> dev-unsubscribe@cassandra.apache.org
>>>>>>>>>>>> For additional commands, e-mail:
>>> dev-help@cassandra.apache.org
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>> 
>>> ---------------------------------------------------------------------
>>>>>>>>>>  To unsubscribe, e-mail:
>>> dev-unsubscribe@cassandra.apache.org
>>>>>>>>>>  For additional commands, e-mail:
>>> dev-help@cassandra.apache.org
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>>>>>>>>> For additional commands, e-mail:
>>> dev-help@cassandra.apache.org
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>>>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>>>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
>>>>>>>>> 
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org

Re: [DISCUSS] CASSANDRA-13994

[Ekaterina Dimitrova <ek...@datastax.com>]

Dear all,
As we talked yesterday during our meeting, I am bringing back to the table
CASSANDRA-13994 <https://issues.apache.org/jira/browse/CASSANDRA-13994>.
Description

4.0 comes without thrift (after CASSANDRA-11115
<https://issues.apache.org/jira/browse/CASSANDRA-11115>) and COMPACT
STORAGE (after CASSANDRA-10857
<https://issues.apache.org/jira/browse/CASSANDRA-10857>), and since Compact
Storage flags are now disabled, all of the related functionality is useless.

There are still some things to consider:
1. One of the system tables (built indexes) was compact. For now, we just
added value column to it to make sure it's backwards-compatible, but we
might want to make sure it's just a "normal" table and doesn't have
redundant columns.
2. Compact Tables were building indexes in KEYS mode. Removing it is
trivial, but this would mean that all built indexes will be defunct. We
could log a warning for now and ask users to migrate off those for now and
completely remove it from future releases. It's just a couple of classes
though.


Thank you Sylvain for the first pass of review.

An option, as also Sylvain suggested, is to split the ticket in two tickets
- clean the Compact storage dead code in this one and open a new ticket to
handle the indexes.

I am pasting here the latest comment, hopefully this makes it easier for
the audience:

I have made a first pass of review and offered a few remarks above.

But I think this ticket is hanging up on us deciding whether removing the
KEYS 2ndary index code is ok or not. And this yields, to me, the question
of what is the upgrade path to 4.0 for users that still have KEYS index
(which, reminder, could only be created with Thrift, but could *used* with
CQL and thus still be around).

Because, while I haven't tested this myself, I suspect we have a hole here.

Namely, KEYS index were compact tables, and 4.0 does not start if there is
still compact tables. And while for user tables, user are asked to use DROP
COMPACT STORAGE before upgrading, this cannot be done on KEYS index (there
is just no syntax to do it), so unless there is code I'm not aware of (and
please, someone correct me if I'm wrong), I don't think user can upgrade to
4.0 at all if they still have KEYS index. They'd have to drop those index
first.

So If I'm right here, this technically mean removing the KEYS index code in
4.0 is fine, since you cannot upgrade in the first place if you have KEYS
index. But the more important question for 4.0 imo is what is the upgrade
path for users if they have a KEYS index in 3.X?

Currently (without code changes), the only available option I can think of
is that before upgrade to 4.0, users would have to 1) drop their KEYS index
and then 2) re-create a "normal" (non-KEYS) equivalent index.

Are we comfortable with that being the upgrade path for KEYS index?

Personally, I'm not sure I am because this is not a seamless upgrade, as
between the 1) and 2) above, there is a window where there is no accessible
index, so if the user application rely on it, it means a period of downtime
for the application to perform the upgrade. However, if we want a more
seamless upgrade, we need to figure something out, and that probably
involve non trivial amounts of code and testing. And, playing devil's
advocate, KEYS index being so old, maybe nobody that plans to upgrade to
4.0 have them anymore, and maybe it's not worth bothering?

So I could use others opinions here.

Tl;dr, this ticket raises the point that "Oops, I'm not sure we have though
through the question of upgrade to 4.0 for KEYS indexes". And tbc, it's not
directly related to this ticket, only indirectly, but it is still something
we need to figure out. And I'd say, before 4.0-alpha. But I'm happy to
create a separate ticket specific to that question if that helps.


Ekaterina

>
>
> It was moved to alpha following the lifecycle document and the latest
> discussion on the ML. Sylvain made a review yesterday. To me the biggest
> question is about the indexing. (Posted questions on the ticket itself,
> waiting for opinion and agreement) For the rest of the ticket I support
> Sylvain that it is primarily removal of dead code.
>
> My understanding is either it lands in alpha before the big testing in
> beta or it is delayed for v.5
>
> Ekaterina
>
> On Wed, 10 Jun 2020 at 12:59, Joshua McKenzie <jm...@apache.org>
> wrote:
>
>> Where did we land on inclusion of this ticket in 4.0? Since it's now
>> flagged as blocking beta rel, I am now very interested in this question.
>>
>> :)
>>
>>
>> On Wed, May 27, 2020 at 7:52 PM Jordan West <jo...@gmail.com> wrote:
>>
>> > On Wed, May 27, 2020 at 1:23 PM Joshua McKenzie <jm...@apache.org>
>> > wrote:
>> >
>> > > Maybe. Do we just time box, say we're going to cut an RC and give it 4
>> > > weeks, if nothing awful surfaces we GA?
>> > >
>> >
>> > I've seen that work well in the past on other projects. I agree with the
>> > notion that RCs are real candidates for release if no one finds issues
>> with
>> > them. Ideally we would have as little RCs as possible and have more
>> > alphas/betas.
>> >
>> > >
>> > > On Wed, May 27, 2020 at 4:12 PM Brandon Williams <dr...@gmail.com>
>> > wrote:
>> > >
>> > > > Absolutely my understanding.
>> > > >
>> > > > On Wed, May 27, 2020, 2:49 PM Jeremiah D Jordan <
>> > > jeremiah.jordan@gmail.com
>> > > > >
>> > > > wrote:
>> > > >
>> > > > > > A clear point to cut RC's doesn't surface from the above for me.
>> > > > > Releasing
>> > > > > > an RC before broad verification seems wrong, and cutting an RC
>> > after
>> > > > the
>> > > > > 4
>> > > > > > points above may as well be GA because it's all known scope.
>> > > > >
>> > > > > Isn’t the whole point of an RC is that it could be the GA?  It is
>> a
>> > > > > “release candidate”, meaning if no one finds any issues with it,
>> that
>> > > can
>> > > > > them become the release?  So that seems like exactly the right
>> time
>> > to
>> > > > make
>> > > > > RC releases?
>> > > > >
>> > > > > > On May 27, 2020, at 2:45 PM, Joshua McKenzie <
>> jmckenzie@apache.org
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > > I think we're all on the same page here; I was focusing more on
>> the
>> > > > > release
>> > > > > > lifecycles and sequencing than the entire version cycle. Good to
>> > > > broaden
>> > > > > > scope I think.
>> > > > > >
>> > > > > > One thing we're not considering is the separation of API changes
>> > from
>> > > > > major
>> > > > > > changes and how that intersects with release milestones.
>> > > > > >
>> > > > > > Meaning:
>> > > > > > 1. alpha phase
>> > > > > > 2. Milestone: API freeze (all API changes pushed to next major)
>> > > > > > 3. beta phase
>> > > > > > 4. Verification phase (all major disruptive pushed to next
>> major)
>> > > > > >
>> > > > > > A clear point to cut RC's doesn't surface from the above for me.
>> > > > > Releasing
>> > > > > > an RC before broad verification seems wrong, and cutting an RC
>> > after
>> > > > the
>> > > > > 4
>> > > > > > points above may as well be GA because it's all known scope.
>> > > > > >
>> > > > > > Thoughts?
>> > > > > >
>> > > > > > On Wed, May 27, 2020 at 3:28 PM Scott Andreas <
>> > scott@paradoxica.net>
>> > > > > wrote:
>> > > > > >
>> > > > > >> That makes sense to me, yep.
>> > > > > >>
>> > > > > >> My hope and expectation is that the time required for
>> > "verification
>> > > > > work"
>> > > > > >> will shrink dramatically in the not too distant future -
>> ideally
>> > to
>> > > a
>> > > > > >> period of less than a month. In this world, the cost of missing
>> > one
>> > > > > train
>> > > > > >> is reduced to catching the next one.
>> > > > > >>
>> > > > > >> One of the main goals in shifting focus from "testing" and
>> "test
>> > > > plans"
>> > > > > to
>> > > > > >> "test engineering" is automating as many aspects of release
>> > > > > qualification
>> > > > > >> as possible, with an asymptotic ideal as a function of compute
>> > > > capacity
>> > > > > and
>> > > > > >> time. While such automation will never be complete (it's likely
>> > that
>> > > > > >> development of new features will/must include qualification
>> infra
>> > > > > changes
>> > > > > >> to exercise them), if we're able to apply the same rigor to
>> major
>> > > > > releases
>> > > > > >> as we are to patchlevel builds with little incremental effort,
>> I'd
>> > > be
>> > > > > >> thrilled.
>> > > > > >>
>> > > > > >> This is mostly a way of saying:
>> > > > > >> – I like the cadence/sequencing Benedict proposes below.
>> > > > > >> – I think improvements in test engineering can reduce/eliminate
>> > > > > >> invalidation and may increase the scope of what can be a
>> candidate
>> > > for
>> > > > > >> merge on a given branch
>> > > > > >> – And if not, the cost of missing the train is lower because
>> we'll
>> > > be
>> > > > > able
>> > > > > >> to deliver major releases more often.
>> > > > > >>
>> > > > > >> Scott
>> > > > > >>
>> > > > > >> ________________________________________
>> > > > > >> From: Jeremiah D Jordan <je...@gmail.com>
>> > > > > >> Sent: Wednesday, May 27, 2020 11:54 AM
>> > > > > >> To: Cassandra DEV
>> > > > > >> Subject: Re: [DISCUSS] CASSANDRA-13994
>> > > > > >>
>> > > > > >> +1 strongly agree.  If we aren’t going to let something go into
>> > > 4.0.0
>> > > > > >> because it would "invalidate testing” then we can not let such
>> a
>> > > thing
>> > > > > go
>> > > > > >> into 4.0.1 unless we plan to re-do said testing for the patch
>> > > release.
>> > > > > >>
>> > > > > >>> On May 27, 2020, at 1:31 PM, Benedict Elliott Smith <
>> > > > > benedict@apache.org>
>> > > > > >> wrote:
>> > > > > >>>
>> > > > > >>> I'm being told this still isn't clear, so let me try in a
>> > > > bullet-point
>> > > > > >> timeline:
>> > > > > >>>
>> > > > > >>> * 4.0 Beta
>> > > > > >>> * 4.0 Verification Work
>> > > > > >>> * [Merge Window]
>> > > > > >>> * 4.0 GA
>> > > > > >>> * 4.0 Minor Releases
>> > > > > >>> * ...
>> > > > > >>> * 5.0 Dev
>> > > > > >>> * ...
>> > > > > >>> * 5.0 Verification Work
>> > > > > >>> * GA 5.0
>> > > > > >>>
>> > > > > >>> I think that anything that is prohibited from "[Merge Window]"
>> > > > because
>> > > > > >> it invalidates "4.0 Verification Work" must also be prohibited
>> > until
>> > > > > "5.0
>> > > > > >> Dev" because the next equivalent work that can now validate it
>> > > occurs
>> > > > > only
>> > > > > >> at "5.0 Verification Work"
>> > > > > >>>
>> > > > > >>> On 27/05/2020, 19:05, "Benedict Elliott Smith" <
>> > > benedict@apache.org
>> > > > >
>> > > > > >> wrote:
>> > > > > >>>
>> > > > > >>>   I'm not sure if I communicated my point very well.  I mean
>> to
>> > say
>> > > > > >> that if the reason we are prohibiting a patch to land
>> post-beta is
>> > > > > because
>> > > > > >> it invalidates work we only perform pre-ga, then it probably
>> > should
>> > > > not
>> > > > > be
>> > > > > >> permitted to land post-ga either, since it must also invalidate
>> > the
>> > > > same
>> > > > > >> work?
>> > > > > >>>
>> > > > > >>>   That is to say, if we're comfortable with work landing
>> post-ga
>> > > > > >> because we believe it to be safe to release without our
>> > > > > pre-major-release
>> > > > > >> verification, we should be comfortable with it landing at any
>> time
>> > > > > pre-ga
>> > > > > >> too.  Anything else seems inconsistent to me, and we should
>> > examine
>> > > > what
>> > > > > >> assumptions we're making that permit this inconsistency to
>> arise.
>> > > > > >>>
>> > > > > >>>
>> > > > > >>>   On 27/05/2020, 18:49, "Joshua McKenzie" <
>> jmckenzie@apache.org
>> > >
>> > > > > >> wrote:
>> > > > > >>>
>> > > > > >>>>
>> > > > > >>>> because it invalidates our pre-release verification, then it
>> > > should
>> > > > > not
>> > > > > >>>> land
>> > > > > >>>
>> > > > > >>>       until we next perform pre-release verification
>> > > > > >>>
>> > > > > >>>       At least for me there's a little softness around our
>> > > collective
>> > > > > >> alignment
>> > > > > >>>       on when pre-release verification takes place. If it's
>> > between
>> > > > > >> alpha-1 and
>> > > > > >>>       ga we don't want changes that would invalidate those
>> > changes
>> > > to
>> > > > > >> land during
>> > > > > >>>       that time frame. Different for beta-1 to ga. We also
>> risk
>> > > > > >> invalidating
>> > > > > >>>       testing if we do any of that testing before wherever
>> that
>> > > > cutoff
>> > > > > >> is, and a
>> > > > > >>>       lack of clarity on that cutoff further muddies those
>> > waters.
>> > > > > >>>
>> > > > > >>>       My very loosely held perspective is that beta-1 to ga is
>> > the
>> > > > > >> window in
>> > > > > >>>       which we apply the "don't do things that will invalidate
>> > > > > >> verification", and
>> > > > > >>>       we plan to do that verification during the beta phase. I
>> > > > *think*
>> > > > > >> this is
>> > > > > >>>       consistent w/the current framing of the lifecycle doc.
>> That
>> > > > being
>> > > > > >> said, I
>> > > > > >>>       don't have strong religion on this so if we collectively
>> > want
>> > > > to
>> > > > > >> call it
>> > > > > >>>       "don't majorly disrupt from alpha-1 to ga", we can
>> > formalize
>> > > > that
>> > > > > >> in the
>> > > > > >>>       docs and go ahead and triage current open scope for 4.0
>> and
>> > > > move
>> > > > > >> things out.
>> > > > > >>>
>> > > > > >>>
>> > > > > >>>
>> > > > > >>>       On Wed, May 27, 2020 at 12:59 PM Ekaterina Dimitrova <
>> > > > > >>>       ekaterina.dimitrova@datastax.com> wrote:
>> > > > > >>>
>> > > > > >>>> Thank you all for your input.
>> > > > > >>>> I think an important topic is again to revise the lifecycle
>> and
>> > > > ensure
>> > > > > >> we
>> > > > > >>>> really have the vision on what is left until beta. I will
>> start
>> > a
>> > > > > >> separate
>> > > > > >>>> thread on the flaky tests situation soon.
>> > > > > >>>>
>> > > > > >>>> For this particular ticket I see a couple of things:
>> > > > > >>>> - There are a lot of deletions of already not used code
>> > > > > >>>> - I implemented it still in alpha as per our agreement that
>> this
>> > > > will
>> > > > > >> give
>> > > > > >>>> us enough time for testing. Probably Dinesh as a reviewer can
>> > give
>> > > > > some
>> > > > > >>>> valuable feedback/opinion on the patch.
>> > > > > >>>> - It definitely touches around important places but the
>> > important
>> > > > > thing
>> > > > > >> is
>> > > > > >>>> to see how exactly it touches, I think
>> > > > > >>>> - Considering it for alpha before the major testing in beta
>> > sounds
>> > > > > >>>> reasonable to me but I guess it also depends on people
>> > > availability
>> > > > to
>> > > > > >>>> review it in detail and the exact test plans afterwards
>> > > > > >>>>
>> > > > > >>>> On Wed, 27 May 2020 at 7:14, Benedict Elliott Smith <
>> > > > > >> benedict@apache.org>
>> > > > > >>>> wrote:
>> > > > > >>>>
>> > > > > >>>>> I think our "pre-beta" criteria should also be our "not in a
>> > > major"
>> > > > > >>>>> criteria.
>> > > > > >>>>>
>> > > > > >>>>> If work is prohibited because it invalidates our pre-release
>> > > > > >>>> verification,
>> > > > > >>>>> then it should not land until we next perform pre-release
>> > > > > verification,
>> > > > > >>>>> which only currently happens once per major.
>> > > > > >>>>>
>> > > > > >>>>> This could mean either landing less in a major, or
>> permitting
>> > > more
>> > > > in
>> > > > > >>>> beta
>> > > > > >>>>> etc.
>> > > > > >>>>>
>> > > > > >>>>> On 26/05/2020, 19:24, "Joshua McKenzie" <
>> jmckenzie@apache.org
>> > >
>> > > > > wrote:
>> > > > > >>>>>
>> > > > > >>>>>   I think an interesting question that informs when to stop
>> > > > accepting
>> > > > > >>>>>   specific changes in a release is when we expect any
>> extensive
>> > > > > >>>>> pre-release
>> > > > > >>>>>   testing to take place.
>> > > > > >>>>>
>> > > > > >>>>>   If we go by our release lifecycle, gutting deprecated code
>> > > seems
>> > > > > >>>>> compatible
>> > > > > >>>>>   w/Alpha but I wouldn't endorse merging it into Beta:
>> > > > > >>>>>
>> > > > > >>>>>
>> > > > > >>
>> > > >
>> > https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
>> > > > > .
>> > > > > >>>>>   Since almost all of the 40_quality_testing epic stuff is
>> also
>> > > > beta
>> > > > > >>>>> phase
>> > > > > >>>>>   and hasn't really taken off yet, it also seems like there
>> > will
>> > > be
>> > > > > >>>>> extensive
>> > > > > >>>>>   testing after this phase transition.
>> > > > > >>>>>
>> > > > > >>>>>   All that being said, I'd advocate for marking FixVer 4.x
>> to
>> > > > > indicate
>> > > > > >>>>>   optionality and disallow merge of tickets like this after
>> > we're
>> > > > > done
>> > > > > >>>>>   w/alpha phase in keeping w/our lifecycle doc in general.
>> > > > > >>>>>
>> > > > > >>>>>   Does that make sense? Should we consider revisiting and
>> > > revising
>> > > > > the
>> > > > > >>>>>   lifecycle doc re: larger deprecation / changes and cycle
>> > > stages?
>> > > > > >>>>>
>> > > > > >>>>>
>> > > > > >>>>>
>> > > > > >>>>>   On Tue, May 26, 2020 at 12:53 PM Oleksandr Petrov <
>> > > > > >>>>>   oleksandr.petrov@gmail.com> wrote:
>> > > > > >>>>>
>> > > > > >>>>>>> 1) Would you block the release over this ticket?
>> > > > > >>>>>>
>> > > > > >>>>>> I would definitely not block the release on this ticket.
>> > > > > >>>>>>
>> > > > > >>>>>>> 2) Would you prioritize this ticket over testing?
>> > > > > >>>>>>
>> > > > > >>>>>> Same here, I would prioritise testing.
>> > > > > >>>>>>
>> > > > > >>>>>>> 3) Does fixing this ticket make 4.0 a more stable release?
>> > > > > >>>>>>
>> > > > > >>>>>> I wanted to give some context: I wrote that in August 2018.
>> > > While
>> > > > I
>> > > > > >>>>> still
>> > > > > >>>>>> believe it is important to get rid of this code, I'm
>> > disinclined
>> > > > to
>> > > > > >>>>> merge
>> > > > > >>>>>> it into 4.0.
>> > > > > >>>>>>
>> > > > > >>>>>> Given that the patch is rather big (421 additions and 1,480
>> > > > > >>>>> deletions) and
>> > > > > >>>>>> touches many important places, including parser, I would be
>> > > > > >>>> extremely
>> > > > > >>>>>> cautious to merge it that late in release cycle. It would
>> be
>> > > great
>> > > > > >>>>> to also
>> > > > > >>>>>> hear arguments that would justify the risk.
>> > > > > >>>>>>
>> > > > > >>>>>> Thank you for starting this discussion,
>> > > > > >>>>>> -- Alex
>> > > > > >>>>>>
>> > > > > >>>>>>
>> > > > > >>>>>>
>> > > > > >>>>>> On Tue, May 26, 2020 at 5:20 PM Ekaterina Dimitrova <
>> > > > > >>>>>> ekaterina.dimitrova@datastax.com> wrote:
>> > > > > >>>>>>
>> > > > > >>>>>>> Dear all,
>> > > > > >>>>>>>
>> > > > > >>>>>>> Following the ticket review sent on 12th May I wanted to
>> > bring
>> > > up
>> > > > > >>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-13994:
>> > Remove
>> > > > > >>>>> COMPACT
>> > > > > >>>>>>>
>> > > > > >>>>>>> STORAGE internals before 4.0 release.
>> > > > > >>>>>>>
>> > > > > >>>>>>> It is already under review by Dinesh Joshi and Alex
>> Petrov.
>> > > Not a
>> > > > > >>>>>>> blocker but already under review.
>> > > > > >>>>>>>
>> > > > > >>>>>>> Below are my responses to the questions brought up.
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>> 1) Would you block the release over this
>> > > > > >>>>>>>
>> > > > > >>>>>>> ticket? - probably not
>> > > > > >>>>>>>
>> > > > > >>>>>>> 2) Would you prioritize this ticket over testing? -
>> already
>> > > > > >>>>>>> implemented but if there are some big changes needed after
>> > the
>> > > > > >>>>> review,
>> > > > > >>>>>>> I doubt it we will want to prioritize over the testing
>> > > > > >>>>>>>
>> > > > > >>>>>>> 3) Does fixing
>> > > > > >>>>>>> this ticket make 4.0 a more stable release? - I will just
>> > cite
>> > > > > >>>> Alex
>> > > > > >>>>>>> Petrov who reported this Jira and I think the rest of us
>> > would
>> > > > > >>>>> agree
>> > > > > >>>>>>> with him here.
>> > > > > >>>>>>>
>> > > > > >>>>>>> "I would say it's quite important to clean up compact
>> storage
>> > > > > >>>>>>> internals in 4.0 before the release. It should have no
>> > visible
>> > > > > >>>>>>> side-effects, but it'd be very good to have as it
>> simplifies
>> > > > > >>>>> multiple
>> > > > > >>>>>>> code paths."
>> > > > > >>>>>>>
>> > > > > >>>>>>>
>> > > > > >>>>>>> Ekaterina Dimitrova
>> > > > > >>>>>>> e. ekaterina.dimitrova@datastax.com
>> > > > > >>>>>>> w. www.datastax.com
>> > > > > >>>>>>>
>> > > > > >>>>>>
>> > > > > >>>>>>
>> > > > > >>>>>> --
>> > > > > >>>>>> alex p
>> > > > > >>>>>>
>> > > > > >>>>>
>> > > > > >>>>>
>> > > > > >>>>>
>> > > > > >>>>>
>> > > >
>> ---------------------------------------------------------------------
>> > > > > >>>>> To unsubscribe, e-mail:
>> dev-unsubscribe@cassandra.apache.org
>> > > > > >>>>> For additional commands, e-mail:
>> dev-help@cassandra.apache.org
>> > > > > >>>>>
>> > > > > >>>>>
>> > > > > >>>>
>> > > > > >>>
>> > > > > >>>
>> > > > > >>>
>> > > > > >>>
>> > > >
>> ---------------------------------------------------------------------
>> > > > > >>>   To unsubscribe, e-mail:
>> dev-unsubscribe@cassandra.apache.org
>> > > > > >>>   For additional commands, e-mail:
>> dev-help@cassandra.apache.org
>> > > > > >>>
>> > > > > >>>
>> > > > > >>>
>> > > > > >>>
>> > > > > >>>
>> > > ---------------------------------------------------------------------
>> > > > > >>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>> > > > > >>> For additional commands, e-mail:
>> dev-help@cassandra.apache.org
>> > > > > >>>
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > ---------------------------------------------------------------------
>> > > > > >> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>> > > > > >> For additional commands, e-mail: dev-help@cassandra.apache.org
>> > > > > >>
>> > > > > >>
>> > > > > >>
>> > > ---------------------------------------------------------------------
>> > > > > >> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>> > > > > >> For additional commands, e-mail: dev-help@cassandra.apache.org
>> > > > > >>
>> > > > > >>
>> > > > >
>> > > > >
>> > > > >
>> ---------------------------------------------------------------------
>> > > > > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>> > > > > For additional commands, e-mail: dev-help@cassandra.apache.org
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] CASSANDRA-13994

[Ekaterina Dimitrova <e....@gmail.com>]

It was moved to alpha following the lifecycle document and the latest
discussion on the ML. Sylvain made a review yesterday. To me the biggest
question is about the indexing. (Posted questions on the ticket itself,
waiting for opinion and agreement) For the rest of the ticket I support
Sylvain that it is primarily removal of dead code.

My understanding is either it lands in alpha before the big testing in beta
or it is delayed for v.5

Ekaterina

On Wed, 10 Jun 2020 at 12:59, Joshua McKenzie <jm...@apache.org> wrote:

> Where did we land on inclusion of this ticket in 4.0? Since it's now
> flagged as blocking beta rel, I am now very interested in this question.
>
> :)
>
>
> On Wed, May 27, 2020 at 7:52 PM Jordan West <jo...@gmail.com> wrote:
>
> > On Wed, May 27, 2020 at 1:23 PM Joshua McKenzie <jm...@apache.org>
> > wrote:
> >
> > > Maybe. Do we just time box, say we're going to cut an RC and give it 4
> > > weeks, if nothing awful surfaces we GA?
> > >
> >
> > I've seen that work well in the past on other projects. I agree with the
> > notion that RCs are real candidates for release if no one finds issues
> with
> > them. Ideally we would have as little RCs as possible and have more
> > alphas/betas.
> >
> > >
> > > On Wed, May 27, 2020 at 4:12 PM Brandon Williams <dr...@gmail.com>
> > wrote:
> > >
> > > > Absolutely my understanding.
> > > >
> > > > On Wed, May 27, 2020, 2:49 PM Jeremiah D Jordan <
> > > jeremiah.jordan@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > > A clear point to cut RC's doesn't surface from the above for me.
> > > > > Releasing
> > > > > > an RC before broad verification seems wrong, and cutting an RC
> > after
> > > > the
> > > > > 4
> > > > > > points above may as well be GA because it's all known scope.
> > > > >
> > > > > Isn’t the whole point of an RC is that it could be the GA?  It is a
> > > > > “release candidate”, meaning if no one finds any issues with it,
> that
> > > can
> > > > > them become the release?  So that seems like exactly the right time
> > to
> > > > make
> > > > > RC releases?
> > > > >
> > > > > > On May 27, 2020, at 2:45 PM, Joshua McKenzie <
> jmckenzie@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > I think we're all on the same page here; I was focusing more on
> the
> > > > > release
> > > > > > lifecycles and sequencing than the entire version cycle. Good to
> > > > broaden
> > > > > > scope I think.
> > > > > >
> > > > > > One thing we're not considering is the separation of API changes
> > from
> > > > > major
> > > > > > changes and how that intersects with release milestones.
> > > > > >
> > > > > > Meaning:
> > > > > > 1. alpha phase
> > > > > > 2. Milestone: API freeze (all API changes pushed to next major)
> > > > > > 3. beta phase
> > > > > > 4. Verification phase (all major disruptive pushed to next major)
> > > > > >
> > > > > > A clear point to cut RC's doesn't surface from the above for me.
> > > > > Releasing
> > > > > > an RC before broad verification seems wrong, and cutting an RC
> > after
> > > > the
> > > > > 4
> > > > > > points above may as well be GA because it's all known scope.
> > > > > >
> > > > > > Thoughts?
> > > > > >
> > > > > > On Wed, May 27, 2020 at 3:28 PM Scott Andreas <
> > scott@paradoxica.net>
> > > > > wrote:
> > > > > >
> > > > > >> That makes sense to me, yep.
> > > > > >>
> > > > > >> My hope and expectation is that the time required for
> > "verification
> > > > > work"
> > > > > >> will shrink dramatically in the not too distant future - ideally
> > to
> > > a
> > > > > >> period of less than a month. In this world, the cost of missing
> > one
> > > > > train
> > > > > >> is reduced to catching the next one.
> > > > > >>
> > > > > >> One of the main goals in shifting focus from "testing" and "test
> > > > plans"
> > > > > to
> > > > > >> "test engineering" is automating as many aspects of release
> > > > > qualification
> > > > > >> as possible, with an asymptotic ideal as a function of compute
> > > > capacity
> > > > > and
> > > > > >> time. While such automation will never be complete (it's likely
> > that
> > > > > >> development of new features will/must include qualification
> infra
> > > > > changes
> > > > > >> to exercise them), if we're able to apply the same rigor to
> major
> > > > > releases
> > > > > >> as we are to patchlevel builds with little incremental effort,
> I'd
> > > be
> > > > > >> thrilled.
> > > > > >>
> > > > > >> This is mostly a way of saying:
> > > > > >> – I like the cadence/sequencing Benedict proposes below.
> > > > > >> – I think improvements in test engineering can reduce/eliminate
> > > > > >> invalidation and may increase the scope of what can be a
> candidate
> > > for
> > > > > >> merge on a given branch
> > > > > >> – And if not, the cost of missing the train is lower because
> we'll
> > > be
> > > > > able
> > > > > >> to deliver major releases more often.
> > > > > >>
> > > > > >> Scott
> > > > > >>
> > > > > >> ________________________________________
> > > > > >> From: Jeremiah D Jordan <je...@gmail.com>
> > > > > >> Sent: Wednesday, May 27, 2020 11:54 AM
> > > > > >> To: Cassandra DEV
> > > > > >> Subject: Re: [DISCUSS] CASSANDRA-13994
> > > > > >>
> > > > > >> +1 strongly agree.  If we aren’t going to let something go into
> > > 4.0.0
> > > > > >> because it would "invalidate testing” then we can not let such a
> > > thing
> > > > > go
> > > > > >> into 4.0.1 unless we plan to re-do said testing for the patch
> > > release.
> > > > > >>
> > > > > >>> On May 27, 2020, at 1:31 PM, Benedict Elliott Smith <
> > > > > benedict@apache.org>
> > > > > >> wrote:
> > > > > >>>
> > > > > >>> I'm being told this still isn't clear, so let me try in a
> > > > bullet-point
> > > > > >> timeline:
> > > > > >>>
> > > > > >>> * 4.0 Beta
> > > > > >>> * 4.0 Verification Work
> > > > > >>> * [Merge Window]
> > > > > >>> * 4.0 GA
> > > > > >>> * 4.0 Minor Releases
> > > > > >>> * ...
> > > > > >>> * 5.0 Dev
> > > > > >>> * ...
> > > > > >>> * 5.0 Verification Work
> > > > > >>> * GA 5.0
> > > > > >>>
> > > > > >>> I think that anything that is prohibited from "[Merge Window]"
> > > > because
> > > > > >> it invalidates "4.0 Verification Work" must also be prohibited
> > until
> > > > > "5.0
> > > > > >> Dev" because the next equivalent work that can now validate it
> > > occurs
> > > > > only
> > > > > >> at "5.0 Verification Work"
> > > > > >>>
> > > > > >>> On 27/05/2020, 19:05, "Benedict Elliott Smith" <
> > > benedict@apache.org
> > > > >
> > > > > >> wrote:
> > > > > >>>
> > > > > >>>   I'm not sure if I communicated my point very well.  I mean to
> > say
> > > > > >> that if the reason we are prohibiting a patch to land post-beta
> is
> > > > > because
> > > > > >> it invalidates work we only perform pre-ga, then it probably
> > should
> > > > not
> > > > > be
> > > > > >> permitted to land post-ga either, since it must also invalidate
> > the
> > > > same
> > > > > >> work?
> > > > > >>>
> > > > > >>>   That is to say, if we're comfortable with work landing
> post-ga
> > > > > >> because we believe it to be safe to release without our
> > > > > pre-major-release
> > > > > >> verification, we should be comfortable with it landing at any
> time
> > > > > pre-ga
> > > > > >> too.  Anything else seems inconsistent to me, and we should
> > examine
> > > > what
> > > > > >> assumptions we're making that permit this inconsistency to
> arise.
> > > > > >>>
> > > > > >>>
> > > > > >>>   On 27/05/2020, 18:49, "Joshua McKenzie" <
> jmckenzie@apache.org
> > >
> > > > > >> wrote:
> > > > > >>>
> > > > > >>>>
> > > > > >>>> because it invalidates our pre-release verification, then it
> > > should
> > > > > not
> > > > > >>>> land
> > > > > >>>
> > > > > >>>       until we next perform pre-release verification
> > > > > >>>
> > > > > >>>       At least for me there's a little softness around our
> > > collective
> > > > > >> alignment
> > > > > >>>       on when pre-release verification takes place. If it's
> > between
> > > > > >> alpha-1 and
> > > > > >>>       ga we don't want changes that would invalidate those
> > changes
> > > to
> > > > > >> land during
> > > > > >>>       that time frame. Different for beta-1 to ga. We also risk
> > > > > >> invalidating
> > > > > >>>       testing if we do any of that testing before wherever that
> > > > cutoff
> > > > > >> is, and a
> > > > > >>>       lack of clarity on that cutoff further muddies those
> > waters.
> > > > > >>>
> > > > > >>>       My very loosely held perspective is that beta-1 to ga is
> > the
> > > > > >> window in
> > > > > >>>       which we apply the "don't do things that will invalidate
> > > > > >> verification", and
> > > > > >>>       we plan to do that verification during the beta phase. I
> > > > *think*
> > > > > >> this is
> > > > > >>>       consistent w/the current framing of the lifecycle doc.
> That
> > > > being
> > > > > >> said, I
> > > > > >>>       don't have strong religion on this so if we collectively
> > want
> > > > to
> > > > > >> call it
> > > > > >>>       "don't majorly disrupt from alpha-1 to ga", we can
> > formalize
> > > > that
> > > > > >> in the
> > > > > >>>       docs and go ahead and triage current open scope for 4.0
> and
> > > > move
> > > > > >> things out.
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>       On Wed, May 27, 2020 at 12:59 PM Ekaterina Dimitrova <
> > > > > >>>       ekaterina.dimitrova@datastax.com> wrote:
> > > > > >>>
> > > > > >>>> Thank you all for your input.
> > > > > >>>> I think an important topic is again to revise the lifecycle
> and
> > > > ensure
> > > > > >> we
> > > > > >>>> really have the vision on what is left until beta. I will
> start
> > a
> > > > > >> separate
> > > > > >>>> thread on the flaky tests situation soon.
> > > > > >>>>
> > > > > >>>> For this particular ticket I see a couple of things:
> > > > > >>>> - There are a lot of deletions of already not used code
> > > > > >>>> - I implemented it still in alpha as per our agreement that
> this
> > > > will
> > > > > >> give
> > > > > >>>> us enough time for testing. Probably Dinesh as a reviewer can
> > give
> > > > > some
> > > > > >>>> valuable feedback/opinion on the patch.
> > > > > >>>> - It definitely touches around important places but the
> > important
> > > > > thing
> > > > > >> is
> > > > > >>>> to see how exactly it touches, I think
> > > > > >>>> - Considering it for alpha before the major testing in beta
> > sounds
> > > > > >>>> reasonable to me but I guess it also depends on people
> > > availability
> > > > to
> > > > > >>>> review it in detail and the exact test plans afterwards
> > > > > >>>>
> > > > > >>>> On Wed, 27 May 2020 at 7:14, Benedict Elliott Smith <
> > > > > >> benedict@apache.org>
> > > > > >>>> wrote:
> > > > > >>>>
> > > > > >>>>> I think our "pre-beta" criteria should also be our "not in a
> > > major"
> > > > > >>>>> criteria.
> > > > > >>>>>
> > > > > >>>>> If work is prohibited because it invalidates our pre-release
> > > > > >>>> verification,
> > > > > >>>>> then it should not land until we next perform pre-release
> > > > > verification,
> > > > > >>>>> which only currently happens once per major.
> > > > > >>>>>
> > > > > >>>>> This could mean either landing less in a major, or permitting
> > > more
> > > > in
> > > > > >>>> beta
> > > > > >>>>> etc.
> > > > > >>>>>
> > > > > >>>>> On 26/05/2020, 19:24, "Joshua McKenzie" <
> jmckenzie@apache.org
> > >
> > > > > wrote:
> > > > > >>>>>
> > > > > >>>>>   I think an interesting question that informs when to stop
> > > > accepting
> > > > > >>>>>   specific changes in a release is when we expect any
> extensive
> > > > > >>>>> pre-release
> > > > > >>>>>   testing to take place.
> > > > > >>>>>
> > > > > >>>>>   If we go by our release lifecycle, gutting deprecated code
> > > seems
> > > > > >>>>> compatible
> > > > > >>>>>   w/Alpha but I wouldn't endorse merging it into Beta:
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>
> > > >
> > https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
> > > > > .
> > > > > >>>>>   Since almost all of the 40_quality_testing epic stuff is
> also
> > > > beta
> > > > > >>>>> phase
> > > > > >>>>>   and hasn't really taken off yet, it also seems like there
> > will
> > > be
> > > > > >>>>> extensive
> > > > > >>>>>   testing after this phase transition.
> > > > > >>>>>
> > > > > >>>>>   All that being said, I'd advocate for marking FixVer 4.x to
> > > > > indicate
> > > > > >>>>>   optionality and disallow merge of tickets like this after
> > we're
> > > > > done
> > > > > >>>>>   w/alpha phase in keeping w/our lifecycle doc in general.
> > > > > >>>>>
> > > > > >>>>>   Does that make sense? Should we consider revisiting and
> > > revising
> > > > > the
> > > > > >>>>>   lifecycle doc re: larger deprecation / changes and cycle
> > > stages?
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>   On Tue, May 26, 2020 at 12:53 PM Oleksandr Petrov <
> > > > > >>>>>   oleksandr.petrov@gmail.com> wrote:
> > > > > >>>>>
> > > > > >>>>>>> 1) Would you block the release over this ticket?
> > > > > >>>>>>
> > > > > >>>>>> I would definitely not block the release on this ticket.
> > > > > >>>>>>
> > > > > >>>>>>> 2) Would you prioritize this ticket over testing?
> > > > > >>>>>>
> > > > > >>>>>> Same here, I would prioritise testing.
> > > > > >>>>>>
> > > > > >>>>>>> 3) Does fixing this ticket make 4.0 a more stable release?
> > > > > >>>>>>
> > > > > >>>>>> I wanted to give some context: I wrote that in August 2018.
> > > While
> > > > I
> > > > > >>>>> still
> > > > > >>>>>> believe it is important to get rid of this code, I'm
> > disinclined
> > > > to
> > > > > >>>>> merge
> > > > > >>>>>> it into 4.0.
> > > > > >>>>>>
> > > > > >>>>>> Given that the patch is rather big (421 additions and 1,480
> > > > > >>>>> deletions) and
> > > > > >>>>>> touches many important places, including parser, I would be
> > > > > >>>> extremely
> > > > > >>>>>> cautious to merge it that late in release cycle. It would be
> > > great
> > > > > >>>>> to also
> > > > > >>>>>> hear arguments that would justify the risk.
> > > > > >>>>>>
> > > > > >>>>>> Thank you for starting this discussion,
> > > > > >>>>>> -- Alex
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On Tue, May 26, 2020 at 5:20 PM Ekaterina Dimitrova <
> > > > > >>>>>> ekaterina.dimitrova@datastax.com> wrote:
> > > > > >>>>>>
> > > > > >>>>>>> Dear all,
> > > > > >>>>>>>
> > > > > >>>>>>> Following the ticket review sent on 12th May I wanted to
> > bring
> > > up
> > > > > >>>>>>> https://issues.apache.org/jira/browse/CASSANDRA-13994:
> > Remove
> > > > > >>>>> COMPACT
> > > > > >>>>>>>
> > > > > >>>>>>> STORAGE internals before 4.0 release.
> > > > > >>>>>>>
> > > > > >>>>>>> It is already under review by Dinesh Joshi and Alex Petrov.
> > > Not a
> > > > > >>>>>>> blocker but already under review.
> > > > > >>>>>>>
> > > > > >>>>>>> Below are my responses to the questions brought up.
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> 1) Would you block the release over this
> > > > > >>>>>>>
> > > > > >>>>>>> ticket? - probably not
> > > > > >>>>>>>
> > > > > >>>>>>> 2) Would you prioritize this ticket over testing? - already
> > > > > >>>>>>> implemented but if there are some big changes needed after
> > the
> > > > > >>>>> review,
> > > > > >>>>>>> I doubt it we will want to prioritize over the testing
> > > > > >>>>>>>
> > > > > >>>>>>> 3) Does fixing
> > > > > >>>>>>> this ticket make 4.0 a more stable release? - I will just
> > cite
> > > > > >>>> Alex
> > > > > >>>>>>> Petrov who reported this Jira and I think the rest of us
> > would
> > > > > >>>>> agree
> > > > > >>>>>>> with him here.
> > > > > >>>>>>>
> > > > > >>>>>>> "I would say it's quite important to clean up compact
> storage
> > > > > >>>>>>> internals in 4.0 before the release. It should have no
> > visible
> > > > > >>>>>>> side-effects, but it'd be very good to have as it
> simplifies
> > > > > >>>>> multiple
> > > > > >>>>>>> code paths."
> > > > > >>>>>>>
> > > > > >>>>>>>
> > > > > >>>>>>> Ekaterina Dimitrova
> > > > > >>>>>>> e. ekaterina.dimitrova@datastax.com
> > > > > >>>>>>> w. www.datastax.com
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> --
> > > > > >>>>>> alex p
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>>
> > > > ---------------------------------------------------------------------
> > > > > >>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > > >>>>> For additional commands, e-mail:
> dev-help@cassandra.apache.org
> > > > > >>>>>
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > >
> ---------------------------------------------------------------------
> > > > > >>>   To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > > >>>   For additional commands, e-mail:
> dev-help@cassandra.apache.org
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > > > >>>
> > > ---------------------------------------------------------------------
> > > > > >>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > > >>> For additional commands, e-mail: dev-help@cassandra.apache.org
> > > > > >>>
> > > > > >>
> > > > > >>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > > >> For additional commands, e-mail: dev-help@cassandra.apache.org
> > > > > >>
> > > > > >>
> > > > > >>
> > > ---------------------------------------------------------------------
> > > > > >> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > > >> For additional commands, e-mail: dev-help@cassandra.apache.org
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > > > > For additional commands, e-mail: dev-help@cassandra.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>