You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Jason Brown <ja...@gmail.com> on 2018/07/17 04:27:58 UTC
reroll the builds?
Hey all,
The recent builds were -1'd, but it appears the issues have been resolved
(2.2.13 with CASSANDRA-14423, and 3.0.17 / 3.11.3 reverting
CASSANDRA-14252). Can we go ahead and reroll now?
Thanks,
-Jason
Re: reroll the builds?
Posted by "dinesh.joshi@yahoo.com.INVALID" <di...@yahoo.com.INVALID>.
If that's the case, I'm +1 on rerolling the builds.
Dinesh
On Tuesday, July 24, 2018, 9:18:14 AM PDT, Jason Brown <ja...@gmail.com> wrote:
I did run the dtests against the last release shas (3.0.16 and 3.11.2).
Notes are all the way at the bottom of the gist about those runs. Circleci
URLs: https://circleci.com/workflow-run/5a1df5a1-f0c1-4ab4-a7db-e5551e7a5d38
/ https://circleci.com/workflow-run/a4369ab0-ae11-497a-8e10-de3995d10f25.
Current HEAD of 3.0 & 3.11 branches have significantly lower failing
dtests, and the failing tests on HEAD are a subset of those from the last
release.
On Tue, Jul 24, 2018 at 9:03 AM, dinesh.joshi@yahoo.com.INVALID <
dinesh.joshi@yahoo.com.invalid> wrote:
> Hi Jason,
> I agree - we should release with the dataloss bug fix. I went over the
> gist - apart from the Python errors and test teardown failures, there seem
> to be a few failures that look legitimate. Any chance you can run the
> dtests on the previous release SHAs and compare the dtest failures? If
> they're the same / similar, we know at least we're at parity with the
> previous release :)
> Dinesh
>
> On Tuesday, July 24, 2018, 8:18:50 AM PDT, Jason Brown <
> jasedbrown@gmail.com> wrote:
>
> TL;DR We are in a better place than we were for the 3.0.16 and 3.11.2
> releases. The current fails are not fatal, although they warrant
> investigation. My opinion is that due the critical data loss bugs that are
> fixed by CASSANDRA-14513 and CASSANDRA-14515, we should cut the builds now.
>
> I've run the HEAD of the 3.0 and 3.11 branches vs the 3.0.16 and 3.11.2
> release shas, and there are far less failing dtests now. In comparison:
>
> - 3.11
> -- HEAD - 5-6 failing tests
> -- 3.11.2 - 18-20 failures
>
> - 3.0
> -- HEAD - 14-16 failures
> -- 3.0.16 - 22-25 failures
>
> The raw dump of my work can be found here:
> https://gist.github.com/jasobrown/e7ecf6d0bf875d1f4a08ee06ac7eaba0. I've
> applied no effort to clean it up, but it's available (includes links to the
> circleci runs). I haven't completed an exhautive analysis of the failures
> to see how far they go back as things become tricky (or, at least, very
> time intensive to research) with the pytest/python-3 update with
> CASSANDRA-14134. Thus some of the failures might be in the dtests
> themselves (I suspect a couple of the failures are), but most are proabaly
> legit failures.
>
> As this thread is about cutting the releases, I'll save any significiant
> analysis for a followup thread. I will say that the current failures are a
> subset of the previous release's failures, and those failures are not data
> loss bugs.
>
> Overall, I feel far more comfortable getting the data loss fixes out
> without any further delay than waiting for a few minor fixes. I will triage
> the dtest failures over the coming days. There are some open tickets, and
> I'll try to corral those with any new ones.
>
> Thanks,
>
> -Jason
>
>
> On Mon, Jul 23, 2018 at 10:26 AM, dinesh.joshi@yahoo.com.INVALID <
> dinesh.joshi@yahoo.com.invalid> wrote:
>
> > I can help out with the triage / rerunning dtests if needed.
> > Dinesh
> >
> > On Monday, July 23, 2018, 10:22:18 AM PDT, Jason Brown <
> > jasedbrown@gmail.com> wrote:
> >
> > I spoke with some people over here, and I'm going to spend a day doing a
> > quick triage of the failing dtests. There are some fixes for data loss
> bugs
> > that are critical to get out in these builds, so I'll ensure the current
> > failures are within an acceptable level of flakey-ness in order to
> unblock
> > those fixes.
> >
> > Will have an update shortly ...
> >
> > -Jason
> >
> > On Mon, Jul 23, 2018 at 9:18 AM, Jason Brown <ja...@gmail.com>
> wrote:
> >
> > > Hi all,
> > >
> > > First, thanks Joey for running the tests. Your pass/fail counts are
> > > basically what in line with what I've seen for the last several months.
> > (I
> > > don't have an aggregated list anywhere, just observations from recent
> > runs).
> > >
> > > Second, it's beyond me why there's such inertia to actually cutting a
> > > release. We're getting up to almost *six months* since the last
> release.
> > > Are there any grand objections at this point?
> > >
> > > Thanks,
> > >
> > > -Jason
> > >
> > >
> > > On Tue, Jul 17, 2018 at 4:01 PM, Joseph Lynch <jo...@gmail.com>
> > > wrote:
> > >
> > >> We ran the tests against 3.0, 2.2 and 3.11 using circleci and there
> are
> > >> various failing dtests but all three have green unit tests.
> > >>
> > >> 3.11.3 tentative (31d5d87, test branch
> > >> <https://circleci.com/gh/vinaykumarchella/cassandra/tree/
> > >> cassandra_3.11_temp_testing>,
> > >> unit tests <https://circleci.com/gh/vinaykumarchella/cassandra/258>
> > >> pass, 5
> > >> <https://circleci.com/gh/vinaykumarchella/cassandra/256> and 6
> > >> <https://circleci.com/gh/vinaykumarchella/cassandra/256#
> > >> tests/containers/8>
> > >> dtest failures)
> > >> 3.0.17 tentative (d52c7b8, test branch
> > >> <https://circleci.com/gh/jolynch/workflows/cassandra/tree/3.0-testing
> >,
> > >> unit
> > >> tests <https://circleci.com/gh/jolynch/cassandra/110> pass, 14
> > >> <https://circleci.com/gh/jolynch/cassandra/112> and 15
> > >> <https://circleci.com/gh/jolynch/cassandra/111> dtest failures)
> > >> 2.2.13 tentative (3482370, test branch
> > >> <https://circleci.com/gh/sumanth-pasupuleti/workflows/cassan
> > >> dra/tree/2.2-testing>,
> > >> unit tests <https://circleci.com/gh/sumanth-pasupuleti/cassandra/20>
> > >> pass, 9
> > >> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/21> and 10
> > >> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/22#
> > >> tests/containers/8>
> > >> dtest failures)
> > >>
> > >> It looks like many (~6) of the failures in 3.0.x are related to
> > >> snapshot_test.TestArchiveCommitlog. I'm not sure if this is abnormal.
> > >>
> > >> I don't see a good historical record to know if these are just flakes,
> > but
> > >> if we only want to go on green builds perhaps we can either disable
> the
> > >> flakey tests or fix them up? If someone feels strongly we should fix
> > >> particular tests up please link a jira and I can take a whack at some
> of
> > >> them.
> > >>
> > >> -Joey
> > >>
> > >> On Tue, Jul 17, 2018 at 9:35 AM Michael Shuler <
> michael@pbandjelly.org>
> > >> wrote:
> > >>
> > >> > On 07/16/2018 11:27 PM, Jason Brown wrote:
> > >> > > Hey all,
> > >> > >
> > >> > > The recent builds were -1'd, but it appears the issues have been
> > >> resolved
> > >> > > (2.2.13 with CASSANDRA-14423, and 3.0.17 / 3.11.3 reverting
> > >> > > CASSANDRA-14252). Can we go ahead and reroll now?
> > >> >
> > >> > Could someone run through the tests on 2.2, 3.0, 3.11 branches and
> > link
> > >> > them? Thanks!
> > >> >
> > >> > Michael
> > >> >
> > >> > ------------------------------------------------------------
> ---------
> > >> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > >> > For additional commands, e-mail: dev-help@cassandra.apache.org
> > >> >
> > >> >
> > >>
> > >
> > >
> >
> >
>
>
Re: reroll the builds?
Posted by Jason Brown <ja...@gmail.com>.
I did run the dtests against the last release shas (3.0.16 and 3.11.2).
Notes are all the way at the bottom of the gist about those runs. Circleci
URLs: https://circleci.com/workflow-run/5a1df5a1-f0c1-4ab4-a7db-e5551e7a5d38
/ https://circleci.com/workflow-run/a4369ab0-ae11-497a-8e10-de3995d10f25.
Current HEAD of 3.0 & 3.11 branches have significantly lower failing
dtests, and the failing tests on HEAD are a subset of those from the last
release.
On Tue, Jul 24, 2018 at 9:03 AM, dinesh.joshi@yahoo.com.INVALID <
dinesh.joshi@yahoo.com.invalid> wrote:
> Hi Jason,
> I agree - we should release with the dataloss bug fix. I went over the
> gist - apart from the Python errors and test teardown failures, there seem
> to be a few failures that look legitimate. Any chance you can run the
> dtests on the previous release SHAs and compare the dtest failures? If
> they're the same / similar, we know at least we're at parity with the
> previous release :)
> Dinesh
>
> On Tuesday, July 24, 2018, 8:18:50 AM PDT, Jason Brown <
> jasedbrown@gmail.com> wrote:
>
> TL;DR We are in a better place than we were for the 3.0.16 and 3.11.2
> releases. The current fails are not fatal, although they warrant
> investigation. My opinion is that due the critical data loss bugs that are
> fixed by CASSANDRA-14513 and CASSANDRA-14515, we should cut the builds now.
>
> I've run the HEAD of the 3.0 and 3.11 branches vs the 3.0.16 and 3.11.2
> release shas, and there are far less failing dtests now. In comparison:
>
> - 3.11
> -- HEAD - 5-6 failing tests
> -- 3.11.2 - 18-20 failures
>
> - 3.0
> -- HEAD - 14-16 failures
> -- 3.0.16 - 22-25 failures
>
> The raw dump of my work can be found here:
> https://gist.github.com/jasobrown/e7ecf6d0bf875d1f4a08ee06ac7eaba0. I've
> applied no effort to clean it up, but it's available (includes links to the
> circleci runs). I haven't completed an exhautive analysis of the failures
> to see how far they go back as things become tricky (or, at least, very
> time intensive to research) with the pytest/python-3 update with
> CASSANDRA-14134. Thus some of the failures might be in the dtests
> themselves (I suspect a couple of the failures are), but most are proabaly
> legit failures.
>
> As this thread is about cutting the releases, I'll save any significiant
> analysis for a followup thread. I will say that the current failures are a
> subset of the previous release's failures, and those failures are not data
> loss bugs.
>
> Overall, I feel far more comfortable getting the data loss fixes out
> without any further delay than waiting for a few minor fixes. I will triage
> the dtest failures over the coming days. There are some open tickets, and
> I'll try to corral those with any new ones.
>
> Thanks,
>
> -Jason
>
>
> On Mon, Jul 23, 2018 at 10:26 AM, dinesh.joshi@yahoo.com.INVALID <
> dinesh.joshi@yahoo.com.invalid> wrote:
>
> > I can help out with the triage / rerunning dtests if needed.
> > Dinesh
> >
> > On Monday, July 23, 2018, 10:22:18 AM PDT, Jason Brown <
> > jasedbrown@gmail.com> wrote:
> >
> > I spoke with some people over here, and I'm going to spend a day doing a
> > quick triage of the failing dtests. There are some fixes for data loss
> bugs
> > that are critical to get out in these builds, so I'll ensure the current
> > failures are within an acceptable level of flakey-ness in order to
> unblock
> > those fixes.
> >
> > Will have an update shortly ...
> >
> > -Jason
> >
> > On Mon, Jul 23, 2018 at 9:18 AM, Jason Brown <ja...@gmail.com>
> wrote:
> >
> > > Hi all,
> > >
> > > First, thanks Joey for running the tests. Your pass/fail counts are
> > > basically what in line with what I've seen for the last several months.
> > (I
> > > don't have an aggregated list anywhere, just observations from recent
> > runs).
> > >
> > > Second, it's beyond me why there's such inertia to actually cutting a
> > > release. We're getting up to almost *six months* since the last
> release.
> > > Are there any grand objections at this point?
> > >
> > > Thanks,
> > >
> > > -Jason
> > >
> > >
> > > On Tue, Jul 17, 2018 at 4:01 PM, Joseph Lynch <jo...@gmail.com>
> > > wrote:
> > >
> > >> We ran the tests against 3.0, 2.2 and 3.11 using circleci and there
> are
> > >> various failing dtests but all three have green unit tests.
> > >>
> > >> 3.11.3 tentative (31d5d87, test branch
> > >> <https://circleci.com/gh/vinaykumarchella/cassandra/tree/
> > >> cassandra_3.11_temp_testing>,
> > >> unit tests <https://circleci.com/gh/vinaykumarchella/cassandra/258>
> > >> pass, 5
> > >> <https://circleci.com/gh/vinaykumarchella/cassandra/256> and 6
> > >> <https://circleci.com/gh/vinaykumarchella/cassandra/256#
> > >> tests/containers/8>
> > >> dtest failures)
> > >> 3.0.17 tentative (d52c7b8, test branch
> > >> <https://circleci.com/gh/jolynch/workflows/cassandra/tree/3.0-testing
> >,
> > >> unit
> > >> tests <https://circleci.com/gh/jolynch/cassandra/110> pass, 14
> > >> <https://circleci.com/gh/jolynch/cassandra/112> and 15
> > >> <https://circleci.com/gh/jolynch/cassandra/111> dtest failures)
> > >> 2.2.13 tentative (3482370, test branch
> > >> <https://circleci.com/gh/sumanth-pasupuleti/workflows/cassan
> > >> dra/tree/2.2-testing>,
> > >> unit tests <https://circleci.com/gh/sumanth-pasupuleti/cassandra/20>
> > >> pass, 9
> > >> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/21> and 10
> > >> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/22#
> > >> tests/containers/8>
> > >> dtest failures)
> > >>
> > >> It looks like many (~6) of the failures in 3.0.x are related to
> > >> snapshot_test.TestArchiveCommitlog. I'm not sure if this is abnormal.
> > >>
> > >> I don't see a good historical record to know if these are just flakes,
> > but
> > >> if we only want to go on green builds perhaps we can either disable
> the
> > >> flakey tests or fix them up? If someone feels strongly we should fix
> > >> particular tests up please link a jira and I can take a whack at some
> of
> > >> them.
> > >>
> > >> -Joey
> > >>
> > >> On Tue, Jul 17, 2018 at 9:35 AM Michael Shuler <
> michael@pbandjelly.org>
> > >> wrote:
> > >>
> > >> > On 07/16/2018 11:27 PM, Jason Brown wrote:
> > >> > > Hey all,
> > >> > >
> > >> > > The recent builds were -1'd, but it appears the issues have been
> > >> resolved
> > >> > > (2.2.13 with CASSANDRA-14423, and 3.0.17 / 3.11.3 reverting
> > >> > > CASSANDRA-14252). Can we go ahead and reroll now?
> > >> >
> > >> > Could someone run through the tests on 2.2, 3.0, 3.11 branches and
> > link
> > >> > them? Thanks!
> > >> >
> > >> > Michael
> > >> >
> > >> > ------------------------------------------------------------
> ---------
> > >> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > >> > For additional commands, e-mail: dev-help@cassandra.apache.org
> > >> >
> > >> >
> > >>
> > >
> > >
> >
> >
>
>
Re: reroll the builds?
Posted by "dinesh.joshi@yahoo.com.INVALID" <di...@yahoo.com.INVALID>.
Hi Jason,
I agree - we should release with the dataloss bug fix. I went over the gist - apart from the Python errors and test teardown failures, there seem to be a few failures that look legitimate. Any chance you can run the dtests on the previous release SHAs and compare the dtest failures? If they're the same / similar, we know at least we're at parity with the previous release :)
Dinesh
On Tuesday, July 24, 2018, 8:18:50 AM PDT, Jason Brown <ja...@gmail.com> wrote:
TL;DR We are in a better place than we were for the 3.0.16 and 3.11.2
releases. The current fails are not fatal, although they warrant
investigation. My opinion is that due the critical data loss bugs that are
fixed by CASSANDRA-14513 and CASSANDRA-14515, we should cut the builds now.
I've run the HEAD of the 3.0 and 3.11 branches vs the 3.0.16 and 3.11.2
release shas, and there are far less failing dtests now. In comparison:
- 3.11
-- HEAD - 5-6 failing tests
-- 3.11.2 - 18-20 failures
- 3.0
-- HEAD - 14-16 failures
-- 3.0.16 - 22-25 failures
The raw dump of my work can be found here:
https://gist.github.com/jasobrown/e7ecf6d0bf875d1f4a08ee06ac7eaba0. I've
applied no effort to clean it up, but it's available (includes links to the
circleci runs). I haven't completed an exhautive analysis of the failures
to see how far they go back as things become tricky (or, at least, very
time intensive to research) with the pytest/python-3 update with
CASSANDRA-14134. Thus some of the failures might be in the dtests
themselves (I suspect a couple of the failures are), but most are proabaly
legit failures.
As this thread is about cutting the releases, I'll save any significiant
analysis for a followup thread. I will say that the current failures are a
subset of the previous release's failures, and those failures are not data
loss bugs.
Overall, I feel far more comfortable getting the data loss fixes out
without any further delay than waiting for a few minor fixes. I will triage
the dtest failures over the coming days. There are some open tickets, and
I'll try to corral those with any new ones.
Thanks,
-Jason
On Mon, Jul 23, 2018 at 10:26 AM, dinesh.joshi@yahoo.com.INVALID <
dinesh.joshi@yahoo.com.invalid> wrote:
> I can help out with the triage / rerunning dtests if needed.
> Dinesh
>
> On Monday, July 23, 2018, 10:22:18 AM PDT, Jason Brown <
> jasedbrown@gmail.com> wrote:
>
> I spoke with some people over here, and I'm going to spend a day doing a
> quick triage of the failing dtests. There are some fixes for data loss bugs
> that are critical to get out in these builds, so I'll ensure the current
> failures are within an acceptable level of flakey-ness in order to unblock
> those fixes.
>
> Will have an update shortly ...
>
> -Jason
>
> On Mon, Jul 23, 2018 at 9:18 AM, Jason Brown <ja...@gmail.com> wrote:
>
> > Hi all,
> >
> > First, thanks Joey for running the tests. Your pass/fail counts are
> > basically what in line with what I've seen for the last several months.
> (I
> > don't have an aggregated list anywhere, just observations from recent
> runs).
> >
> > Second, it's beyond me why there's such inertia to actually cutting a
> > release. We're getting up to almost *six months* since the last release.
> > Are there any grand objections at this point?
> >
> > Thanks,
> >
> > -Jason
> >
> >
> > On Tue, Jul 17, 2018 at 4:01 PM, Joseph Lynch <jo...@gmail.com>
> > wrote:
> >
> >> We ran the tests against 3.0, 2.2 and 3.11 using circleci and there are
> >> various failing dtests but all three have green unit tests.
> >>
> >> 3.11.3 tentative (31d5d87, test branch
> >> <https://circleci.com/gh/vinaykumarchella/cassandra/tree/
> >> cassandra_3.11_temp_testing>,
> >> unit tests <https://circleci.com/gh/vinaykumarchella/cassandra/258>
> >> pass, 5
> >> <https://circleci.com/gh/vinaykumarchella/cassandra/256> and 6
> >> <https://circleci.com/gh/vinaykumarchella/cassandra/256#
> >> tests/containers/8>
> >> dtest failures)
> >> 3.0.17 tentative (d52c7b8, test branch
> >> <https://circleci.com/gh/jolynch/workflows/cassandra/tree/3.0-testing>,
> >> unit
> >> tests <https://circleci.com/gh/jolynch/cassandra/110> pass, 14
> >> <https://circleci.com/gh/jolynch/cassandra/112> and 15
> >> <https://circleci.com/gh/jolynch/cassandra/111> dtest failures)
> >> 2.2.13 tentative (3482370, test branch
> >> <https://circleci.com/gh/sumanth-pasupuleti/workflows/cassan
> >> dra/tree/2.2-testing>,
> >> unit tests <https://circleci.com/gh/sumanth-pasupuleti/cassandra/20>
> >> pass, 9
> >> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/21> and 10
> >> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/22#
> >> tests/containers/8>
> >> dtest failures)
> >>
> >> It looks like many (~6) of the failures in 3.0.x are related to
> >> snapshot_test.TestArchiveCommitlog. I'm not sure if this is abnormal.
> >>
> >> I don't see a good historical record to know if these are just flakes,
> but
> >> if we only want to go on green builds perhaps we can either disable the
> >> flakey tests or fix them up? If someone feels strongly we should fix
> >> particular tests up please link a jira and I can take a whack at some of
> >> them.
> >>
> >> -Joey
> >>
> >> On Tue, Jul 17, 2018 at 9:35 AM Michael Shuler <mi...@pbandjelly.org>
> >> wrote:
> >>
> >> > On 07/16/2018 11:27 PM, Jason Brown wrote:
> >> > > Hey all,
> >> > >
> >> > > The recent builds were -1'd, but it appears the issues have been
> >> resolved
> >> > > (2.2.13 with CASSANDRA-14423, and 3.0.17 / 3.11.3 reverting
> >> > > CASSANDRA-14252). Can we go ahead and reroll now?
> >> >
> >> > Could someone run through the tests on 2.2, 3.0, 3.11 branches and
> link
> >> > them? Thanks!
> >> >
> >> > Michael
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >> >
> >> >
> >>
> >
> >
>
>
Re: reroll the builds?
Posted by Jason Brown <ja...@gmail.com>.
TL;DR We are in a better place than we were for the 3.0.16 and 3.11.2
releases. The current fails are not fatal, although they warrant
investigation. My opinion is that due the critical data loss bugs that are
fixed by CASSANDRA-14513 and CASSANDRA-14515, we should cut the builds now.
I've run the HEAD of the 3.0 and 3.11 branches vs the 3.0.16 and 3.11.2
release shas, and there are far less failing dtests now. In comparison:
- 3.11
-- HEAD - 5-6 failing tests
-- 3.11.2 - 18-20 failures
- 3.0
-- HEAD - 14-16 failures
-- 3.0.16 - 22-25 failures
The raw dump of my work can be found here:
https://gist.github.com/jasobrown/e7ecf6d0bf875d1f4a08ee06ac7eaba0. I've
applied no effort to clean it up, but it's available (includes links to the
circleci runs). I haven't completed an exhautive analysis of the failures
to see how far they go back as things become tricky (or, at least, very
time intensive to research) with the pytest/python-3 update with
CASSANDRA-14134. Thus some of the failures might be in the dtests
themselves (I suspect a couple of the failures are), but most are proabaly
legit failures.
As this thread is about cutting the releases, I'll save any significiant
analysis for a followup thread. I will say that the current failures are a
subset of the previous release's failures, and those failures are not data
loss bugs.
Overall, I feel far more comfortable getting the data loss fixes out
without any further delay than waiting for a few minor fixes. I will triage
the dtest failures over the coming days. There are some open tickets, and
I'll try to corral those with any new ones.
Thanks,
-Jason
On Mon, Jul 23, 2018 at 10:26 AM, dinesh.joshi@yahoo.com.INVALID <
dinesh.joshi@yahoo.com.invalid> wrote:
> I can help out with the triage / rerunning dtests if needed.
> Dinesh
>
> On Monday, July 23, 2018, 10:22:18 AM PDT, Jason Brown <
> jasedbrown@gmail.com> wrote:
>
> I spoke with some people over here, and I'm going to spend a day doing a
> quick triage of the failing dtests. There are some fixes for data loss bugs
> that are critical to get out in these builds, so I'll ensure the current
> failures are within an acceptable level of flakey-ness in order to unblock
> those fixes.
>
> Will have an update shortly ...
>
> -Jason
>
> On Mon, Jul 23, 2018 at 9:18 AM, Jason Brown <ja...@gmail.com> wrote:
>
> > Hi all,
> >
> > First, thanks Joey for running the tests. Your pass/fail counts are
> > basically what in line with what I've seen for the last several months.
> (I
> > don't have an aggregated list anywhere, just observations from recent
> runs).
> >
> > Second, it's beyond me why there's such inertia to actually cutting a
> > release. We're getting up to almost *six months* since the last release.
> > Are there any grand objections at this point?
> >
> > Thanks,
> >
> > -Jason
> >
> >
> > On Tue, Jul 17, 2018 at 4:01 PM, Joseph Lynch <jo...@gmail.com>
> > wrote:
> >
> >> We ran the tests against 3.0, 2.2 and 3.11 using circleci and there are
> >> various failing dtests but all three have green unit tests.
> >>
> >> 3.11.3 tentative (31d5d87, test branch
> >> <https://circleci.com/gh/vinaykumarchella/cassandra/tree/
> >> cassandra_3.11_temp_testing>,
> >> unit tests <https://circleci.com/gh/vinaykumarchella/cassandra/258>
> >> pass, 5
> >> <https://circleci.com/gh/vinaykumarchella/cassandra/256> and 6
> >> <https://circleci.com/gh/vinaykumarchella/cassandra/256#
> >> tests/containers/8>
> >> dtest failures)
> >> 3.0.17 tentative (d52c7b8, test branch
> >> <https://circleci.com/gh/jolynch/workflows/cassandra/tree/3.0-testing>,
> >> unit
> >> tests <https://circleci.com/gh/jolynch/cassandra/110> pass, 14
> >> <https://circleci.com/gh/jolynch/cassandra/112> and 15
> >> <https://circleci.com/gh/jolynch/cassandra/111> dtest failures)
> >> 2.2.13 tentative (3482370, test branch
> >> <https://circleci.com/gh/sumanth-pasupuleti/workflows/cassan
> >> dra/tree/2.2-testing>,
> >> unit tests <https://circleci.com/gh/sumanth-pasupuleti/cassandra/20>
> >> pass, 9
> >> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/21> and 10
> >> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/22#
> >> tests/containers/8>
> >> dtest failures)
> >>
> >> It looks like many (~6) of the failures in 3.0.x are related to
> >> snapshot_test.TestArchiveCommitlog. I'm not sure if this is abnormal.
> >>
> >> I don't see a good historical record to know if these are just flakes,
> but
> >> if we only want to go on green builds perhaps we can either disable the
> >> flakey tests or fix them up? If someone feels strongly we should fix
> >> particular tests up please link a jira and I can take a whack at some of
> >> them.
> >>
> >> -Joey
> >>
> >> On Tue, Jul 17, 2018 at 9:35 AM Michael Shuler <mi...@pbandjelly.org>
> >> wrote:
> >>
> >> > On 07/16/2018 11:27 PM, Jason Brown wrote:
> >> > > Hey all,
> >> > >
> >> > > The recent builds were -1'd, but it appears the issues have been
> >> resolved
> >> > > (2.2.13 with CASSANDRA-14423, and 3.0.17 / 3.11.3 reverting
> >> > > CASSANDRA-14252). Can we go ahead and reroll now?
> >> >
> >> > Could someone run through the tests on 2.2, 3.0, 3.11 branches and
> link
> >> > them? Thanks!
> >> >
> >> > Michael
> >> >
> >> > ---------------------------------------------------------------------
> >> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >> >
> >> >
> >>
> >
> >
>
>
Re: reroll the builds?
Posted by "dinesh.joshi@yahoo.com.INVALID" <di...@yahoo.com.INVALID>.
I can help out with the triage / rerunning dtests if needed.
Dinesh
On Monday, July 23, 2018, 10:22:18 AM PDT, Jason Brown <ja...@gmail.com> wrote:
I spoke with some people over here, and I'm going to spend a day doing a
quick triage of the failing dtests. There are some fixes for data loss bugs
that are critical to get out in these builds, so I'll ensure the current
failures are within an acceptable level of flakey-ness in order to unblock
those fixes.
Will have an update shortly ...
-Jason
On Mon, Jul 23, 2018 at 9:18 AM, Jason Brown <ja...@gmail.com> wrote:
> Hi all,
>
> First, thanks Joey for running the tests. Your pass/fail counts are
> basically what in line with what I've seen for the last several months. (I
> don't have an aggregated list anywhere, just observations from recent runs).
>
> Second, it's beyond me why there's such inertia to actually cutting a
> release. We're getting up to almost *six months* since the last release.
> Are there any grand objections at this point?
>
> Thanks,
>
> -Jason
>
>
> On Tue, Jul 17, 2018 at 4:01 PM, Joseph Lynch <jo...@gmail.com>
> wrote:
>
>> We ran the tests against 3.0, 2.2 and 3.11 using circleci and there are
>> various failing dtests but all three have green unit tests.
>>
>> 3.11.3 tentative (31d5d87, test branch
>> <https://circleci.com/gh/vinaykumarchella/cassandra/tree/
>> cassandra_3.11_temp_testing>,
>> unit tests <https://circleci.com/gh/vinaykumarchella/cassandra/258>
>> pass, 5
>> <https://circleci.com/gh/vinaykumarchella/cassandra/256> and 6
>> <https://circleci.com/gh/vinaykumarchella/cassandra/256#
>> tests/containers/8>
>> dtest failures)
>> 3.0.17 tentative (d52c7b8, test branch
>> <https://circleci.com/gh/jolynch/workflows/cassandra/tree/3.0-testing>,
>> unit
>> tests <https://circleci.com/gh/jolynch/cassandra/110> pass, 14
>> <https://circleci.com/gh/jolynch/cassandra/112> and 15
>> <https://circleci.com/gh/jolynch/cassandra/111> dtest failures)
>> 2.2.13 tentative (3482370, test branch
>> <https://circleci.com/gh/sumanth-pasupuleti/workflows/cassan
>> dra/tree/2.2-testing>,
>> unit tests <https://circleci.com/gh/sumanth-pasupuleti/cassandra/20>
>> pass, 9
>> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/21> and 10
>> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/22#
>> tests/containers/8>
>> dtest failures)
>>
>> It looks like many (~6) of the failures in 3.0.x are related to
>> snapshot_test.TestArchiveCommitlog. I'm not sure if this is abnormal.
>>
>> I don't see a good historical record to know if these are just flakes, but
>> if we only want to go on green builds perhaps we can either disable the
>> flakey tests or fix them up? If someone feels strongly we should fix
>> particular tests up please link a jira and I can take a whack at some of
>> them.
>>
>> -Joey
>>
>> On Tue, Jul 17, 2018 at 9:35 AM Michael Shuler <mi...@pbandjelly.org>
>> wrote:
>>
>> > On 07/16/2018 11:27 PM, Jason Brown wrote:
>> > > Hey all,
>> > >
>> > > The recent builds were -1'd, but it appears the issues have been
>> resolved
>> > > (2.2.13 with CASSANDRA-14423, and 3.0.17 / 3.11.3 reverting
>> > > CASSANDRA-14252). Can we go ahead and reroll now?
>> >
>> > Could someone run through the tests on 2.2, 3.0, 3.11 branches and link
>> > them? Thanks!
>> >
>> > Michael
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>> > For additional commands, e-mail: dev-help@cassandra.apache.org
>> >
>> >
>>
>
>
Re: reroll the builds?
Posted by Jason Brown <ja...@gmail.com>.
I spoke with some people over here, and I'm going to spend a day doing a
quick triage of the failing dtests. There are some fixes for data loss bugs
that are critical to get out in these builds, so I'll ensure the current
failures are within an acceptable level of flakey-ness in order to unblock
those fixes.
Will have an update shortly ...
-Jason
On Mon, Jul 23, 2018 at 9:18 AM, Jason Brown <ja...@gmail.com> wrote:
> Hi all,
>
> First, thanks Joey for running the tests. Your pass/fail counts are
> basically what in line with what I've seen for the last several months. (I
> don't have an aggregated list anywhere, just observations from recent runs).
>
> Second, it's beyond me why there's such inertia to actually cutting a
> release. We're getting up to almost *six months* since the last release.
> Are there any grand objections at this point?
>
> Thanks,
>
> -Jason
>
>
> On Tue, Jul 17, 2018 at 4:01 PM, Joseph Lynch <jo...@gmail.com>
> wrote:
>
>> We ran the tests against 3.0, 2.2 and 3.11 using circleci and there are
>> various failing dtests but all three have green unit tests.
>>
>> 3.11.3 tentative (31d5d87, test branch
>> <https://circleci.com/gh/vinaykumarchella/cassandra/tree/
>> cassandra_3.11_temp_testing>,
>> unit tests <https://circleci.com/gh/vinaykumarchella/cassandra/258>
>> pass, 5
>> <https://circleci.com/gh/vinaykumarchella/cassandra/256> and 6
>> <https://circleci.com/gh/vinaykumarchella/cassandra/256#
>> tests/containers/8>
>> dtest failures)
>> 3.0.17 tentative (d52c7b8, test branch
>> <https://circleci.com/gh/jolynch/workflows/cassandra/tree/3.0-testing>,
>> unit
>> tests <https://circleci.com/gh/jolynch/cassandra/110> pass, 14
>> <https://circleci.com/gh/jolynch/cassandra/112> and 15
>> <https://circleci.com/gh/jolynch/cassandra/111> dtest failures)
>> 2.2.13 tentative (3482370, test branch
>> <https://circleci.com/gh/sumanth-pasupuleti/workflows/cassan
>> dra/tree/2.2-testing>,
>> unit tests <https://circleci.com/gh/sumanth-pasupuleti/cassandra/20>
>> pass, 9
>> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/21> and 10
>> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/22#
>> tests/containers/8>
>> dtest failures)
>>
>> It looks like many (~6) of the failures in 3.0.x are related to
>> snapshot_test.TestArchiveCommitlog. I'm not sure if this is abnormal.
>>
>> I don't see a good historical record to know if these are just flakes, but
>> if we only want to go on green builds perhaps we can either disable the
>> flakey tests or fix them up? If someone feels strongly we should fix
>> particular tests up please link a jira and I can take a whack at some of
>> them.
>>
>> -Joey
>>
>> On Tue, Jul 17, 2018 at 9:35 AM Michael Shuler <mi...@pbandjelly.org>
>> wrote:
>>
>> > On 07/16/2018 11:27 PM, Jason Brown wrote:
>> > > Hey all,
>> > >
>> > > The recent builds were -1'd, but it appears the issues have been
>> resolved
>> > > (2.2.13 with CASSANDRA-14423, and 3.0.17 / 3.11.3 reverting
>> > > CASSANDRA-14252). Can we go ahead and reroll now?
>> >
>> > Could someone run through the tests on 2.2, 3.0, 3.11 branches and link
>> > them? Thanks!
>> >
>> > Michael
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>> > For additional commands, e-mail: dev-help@cassandra.apache.org
>> >
>> >
>>
>
>
Re: reroll the builds?
Posted by Jason Brown <ja...@gmail.com>.
Hi all,
First, thanks Joey for running the tests. Your pass/fail counts are
basically what in line with what I've seen for the last several months. (I
don't have an aggregated list anywhere, just observations from recent runs).
Second, it's beyond me why there's such inertia to actually cutting a
release. We're getting up to almost *six months* since the last release.
Are there any grand objections at this point?
Thanks,
-Jason
On Tue, Jul 17, 2018 at 4:01 PM, Joseph Lynch <jo...@gmail.com> wrote:
> We ran the tests against 3.0, 2.2 and 3.11 using circleci and there are
> various failing dtests but all three have green unit tests.
>
> 3.11.3 tentative (31d5d87, test branch
> <https://circleci.com/gh/vinaykumarchella/cassandra/
> tree/cassandra_3.11_temp_testing>,
> unit tests <https://circleci.com/gh/vinaykumarchella/cassandra/258> pass,
> 5
> <https://circleci.com/gh/vinaykumarchella/cassandra/256> and 6
> <https://circleci.com/gh/vinaykumarchella/cassandra/256#tests/containers/8
> >
> dtest failures)
> 3.0.17 tentative (d52c7b8, test branch
> <https://circleci.com/gh/jolynch/workflows/cassandra/tree/3.0-testing>,
> unit
> tests <https://circleci.com/gh/jolynch/cassandra/110> pass, 14
> <https://circleci.com/gh/jolynch/cassandra/112> and 15
> <https://circleci.com/gh/jolynch/cassandra/111> dtest failures)
> 2.2.13 tentative (3482370, test branch
> <https://circleci.com/gh/sumanth-pasupuleti/workflows/
> cassandra/tree/2.2-testing>,
> unit tests <https://circleci.com/gh/sumanth-pasupuleti/cassandra/20>
> pass, 9
> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/21> and 10
> <https://circleci.com/gh/sumanth-pasupuleti/cassandra/
> 22#tests/containers/8>
> dtest failures)
>
> It looks like many (~6) of the failures in 3.0.x are related to
> snapshot_test.TestArchiveCommitlog. I'm not sure if this is abnormal.
>
> I don't see a good historical record to know if these are just flakes, but
> if we only want to go on green builds perhaps we can either disable the
> flakey tests or fix them up? If someone feels strongly we should fix
> particular tests up please link a jira and I can take a whack at some of
> them.
>
> -Joey
>
> On Tue, Jul 17, 2018 at 9:35 AM Michael Shuler <mi...@pbandjelly.org>
> wrote:
>
> > On 07/16/2018 11:27 PM, Jason Brown wrote:
> > > Hey all,
> > >
> > > The recent builds were -1'd, but it appears the issues have been
> resolved
> > > (2.2.13 with CASSANDRA-14423, and 3.0.17 / 3.11.3 reverting
> > > CASSANDRA-14252). Can we go ahead and reroll now?
> >
> > Could someone run through the tests on 2.2, 3.0, 3.11 branches and link
> > them? Thanks!
> >
> > Michael
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >
> >
>
Re: reroll the builds?
Posted by Joseph Lynch <jo...@gmail.com>.
We ran the tests against 3.0, 2.2 and 3.11 using circleci and there are
various failing dtests but all three have green unit tests.
3.11.3 tentative (31d5d87, test branch
<https://circleci.com/gh/vinaykumarchella/cassandra/tree/cassandra_3.11_temp_testing>,
unit tests <https://circleci.com/gh/vinaykumarchella/cassandra/258> pass, 5
<https://circleci.com/gh/vinaykumarchella/cassandra/256> and 6
<https://circleci.com/gh/vinaykumarchella/cassandra/256#tests/containers/8>
dtest failures)
3.0.17 tentative (d52c7b8, test branch
<https://circleci.com/gh/jolynch/workflows/cassandra/tree/3.0-testing>, unit
tests <https://circleci.com/gh/jolynch/cassandra/110> pass, 14
<https://circleci.com/gh/jolynch/cassandra/112> and 15
<https://circleci.com/gh/jolynch/cassandra/111> dtest failures)
2.2.13 tentative (3482370, test branch
<https://circleci.com/gh/sumanth-pasupuleti/workflows/cassandra/tree/2.2-testing>,
unit tests <https://circleci.com/gh/sumanth-pasupuleti/cassandra/20> pass, 9
<https://circleci.com/gh/sumanth-pasupuleti/cassandra/21> and 10
<https://circleci.com/gh/sumanth-pasupuleti/cassandra/22#tests/containers/8>
dtest failures)
It looks like many (~6) of the failures in 3.0.x are related to
snapshot_test.TestArchiveCommitlog. I'm not sure if this is abnormal.
I don't see a good historical record to know if these are just flakes, but
if we only want to go on green builds perhaps we can either disable the
flakey tests or fix them up? If someone feels strongly we should fix
particular tests up please link a jira and I can take a whack at some of
them.
-Joey
On Tue, Jul 17, 2018 at 9:35 AM Michael Shuler <mi...@pbandjelly.org>
wrote:
> On 07/16/2018 11:27 PM, Jason Brown wrote:
> > Hey all,
> >
> > The recent builds were -1'd, but it appears the issues have been resolved
> > (2.2.13 with CASSANDRA-14423, and 3.0.17 / 3.11.3 reverting
> > CASSANDRA-14252). Can we go ahead and reroll now?
>
> Could someone run through the tests on 2.2, 3.0, 3.11 branches and link
> them? Thanks!
>
> Michael
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>
Re: reroll the builds?
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 07/16/2018 11:27 PM, Jason Brown wrote:
> Hey all,
>
> The recent builds were -1'd, but it appears the issues have been resolved
> (2.2.13 with CASSANDRA-14423, and 3.0.17 / 3.11.3 reverting
> CASSANDRA-14252). Can we go ahead and reroll now?
Could someone run through the tests on 2.2, 3.0, 3.11 branches and link
them? Thanks!
Michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org