[DISCUSS] is it time to make Windows tests gating?

[Owen Nichols <on...@pivotal.io>]

For a very long time we’ve had Windows tests in the main pipeline (hidden away, not in the default view), but the pipeline proceeds to publish regardless of whether Windows tests fail or even run at all.

Now seems like a good time to review whether to:
a) treat Windows tests as first-class tests and prevent the pipeline from proceeding if any test fails on Windows
b) keep as-is
c) change Windows tests to trigger only once a week rather than on every commit, if they are going to remain "informational only"

One disadvantage to making Windows tests gating is that they currently take much longer to run (around 5 hours, vs 2 hours for Linux tests).

Re: [DISCUSS] is it time to make Windows tests gating?

[Owen Nichols <on...@pivotal.io>]

Windows tests are now in the main pipeline.  There are still a handful of intermittent failures, but generally I expect to see them mostly green.  The increased visibility should help drive the process of shaking out any remaining issues.

The shape of the main pipeline is now very tall, which most people don’t seem to mind, but if you feel strongly that the font is just too small, a demo of an alternative layout is at https://github.com/apache/geode/pull/3606

> On May 17, 2019, at 9:53 AM, Dan Smith <ds...@pivotal.io> wrote:
> 
> You're awesome Jens!
> 
> -Dan
> 
> On Fri, May 17, 2019 at 9:48 AM Jens Deppe <jd...@pivotal.io> wrote:
> 
>> I fixed the first two and started ignoring
>> AvailablePortHelperIntegrationTest for Windows (Windows has a default
>> active port range wildly different than Linux which makes this test very
>> flaky, and since it's just testing test code it seemed reasonable to ignore
>> it).
>> 
>> LauncherMemberMXBeanIntegration test still needs some love and I can look
>> at that since I've worked on it in the past.
>> 
>> --Jens
>> 
>> On Fri, May 17, 2019 at 9:33 AM Dan Smith <ds...@pivotal.io> wrote:
>> 
>>> Looking at the metrics for the windows jobs, it looks like the windows
>>> tests are mostly red due to a few specific tests. The acceptance and gfsh
>>> distributed tests jobs seem to be ok, it's just the unit tests and
>>> integration tests that have problems. It also looks like
>>> ExportConfigCommandTest stopped failing recently. Anyone want to take the
>>> credit?
>>> 
>>> I'd be in favor of putting them in the main pipeline but not the PR
>>> pipeline, since they take so long. Based on the metrics it looks like we
>>> are pretty close to green with these jobs in terms of effort. Anyone want
>>> to volunteer to fix the remaining failures (or see if they have been
>> fixed
>>> recently)?
>>> 
>>> -Dan
>>> 
>>> 
>>> 
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsUnitTestOpenJDK8Metrics/builds/209
>>> 
>>> CreateDiskStoreCommandTest:  2 failures
>>> ExportConfigCommandTest:  34 failures
>>> 
>>> 
>>> 
>> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsIntegrationTestOpenJDK8Metrics/builds/209
>>> 
>>> LauncherMemberMXBeanIntegrationTest:  4 failures
>>> AvailablePortHelperIntegrationTest:  17 failures
>>> 
>>> 
>>> 
>>> 
>>> On Thu, May 16, 2019 at 5:32 PM Owen Nichols <on...@pivotal.io>
>> wrote:
>>> 
>>>> I’ve created a PR for this: https://github.com/apache/geode/pull/3597
>>>> 
>>>>> On May 16, 2019, at 3:06 PM, Blake Bender <bb...@pivotal.io>
>> wrote:
>>>>> 
>>>>> +1 this needs to happen.  I hope that doesn't cause too much pain for
>>> the
>>>>> dev team, but the native client team has a hard requirement that all
>>> our
>>>>> stuff works properly on Windows at all times, and it can cause
>> trouble
>>> if
>>>>> random builds of the server can break us on Windows.
>>>>> 
>>>>> I would hesitate to run these per-commit if they're taking that long,
>>> but
>>>>> daily is a thing that can easily happen.
>>>>> 
>>>>> On Thu, May 16, 2019 at 2:23 PM Bruce Schuchardt <
>>> bschuchardt@pivotal.io
>>>>> 
>>>>> wrote:
>>>>> 
>>>>>> big +1, as long as artifacts of failed runs can be downloaded
>>>>>> 
>>>>>> On 5/15/19 6:28 PM, Owen Nichols wrote:
>>>>>>> For a very long time we’ve had Windows tests in the main pipeline
>>>>>> (hidden away, not in the default view), but the pipeline proceeds to
>>>>>> publish regardless of whether Windows tests fail or even run at all.
>>>>>>> 
>>>>>>> Now seems like a good time to review whether to:
>>>>>>> a) treat Windows tests as first-class tests and prevent the
>> pipeline
>>>>>> from proceeding if any test fails on Windows
>>>>>>> b) keep as-is
>>>>>>> c) change Windows tests to trigger only once a week rather than on
>>>> every
>>>>>> commit, if they are going to remain "informational only"
>>>>>>> 
>>>>>>> One disadvantage to making Windows tests gating is that they
>>> currently
>>>>>> take much longer to run (around 5 hours, vs 2 hours for Linux
>> tests).
>>>>>> 
>>>> 
>>>> 
>>> 
>> 

Re: [DISCUSS] is it time to make Windows tests gating?

[Dan Smith <ds...@pivotal.io>]

You're awesome Jens!

-Dan

On Fri, May 17, 2019 at 9:48 AM Jens Deppe <jd...@pivotal.io> wrote:

> I fixed the first two and started ignoring
> AvailablePortHelperIntegrationTest for Windows (Windows has a default
> active port range wildly different than Linux which makes this test very
> flaky, and since it's just testing test code it seemed reasonable to ignore
> it).
>
> LauncherMemberMXBeanIntegration test still needs some love and I can look
> at that since I've worked on it in the past.
>
> --Jens
>
> On Fri, May 17, 2019 at 9:33 AM Dan Smith <ds...@pivotal.io> wrote:
>
> > Looking at the metrics for the windows jobs, it looks like the windows
> > tests are mostly red due to a few specific tests. The acceptance and gfsh
> > distributed tests jobs seem to be ok, it's just the unit tests and
> > integration tests that have problems. It also looks like
> > ExportConfigCommandTest stopped failing recently. Anyone want to take the
> > credit?
> >
> > I'd be in favor of putting them in the main pipeline but not the PR
> > pipeline, since they take so long. Based on the metrics it looks like we
> > are pretty close to green with these jobs in terms of effort. Anyone want
> > to volunteer to fix the remaining failures (or see if they have been
> fixed
> > recently)?
> >
> > -Dan
> >
> >
> >
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsUnitTestOpenJDK8Metrics/builds/209
> >
> > CreateDiskStoreCommandTest:  2 failures
> > ExportConfigCommandTest:  34 failures
> >
> >
> >
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsIntegrationTestOpenJDK8Metrics/builds/209
> >
> > LauncherMemberMXBeanIntegrationTest:  4 failures
> > AvailablePortHelperIntegrationTest:  17 failures
> >
> >
> >
> >
> > On Thu, May 16, 2019 at 5:32 PM Owen Nichols <on...@pivotal.io>
> wrote:
> >
> > > I’ve created a PR for this: https://github.com/apache/geode/pull/3597
> > >
> > > > On May 16, 2019, at 3:06 PM, Blake Bender <bb...@pivotal.io>
> wrote:
> > > >
> > > > +1 this needs to happen.  I hope that doesn't cause too much pain for
> > the
> > > > dev team, but the native client team has a hard requirement that all
> > our
> > > > stuff works properly on Windows at all times, and it can cause
> trouble
> > if
> > > > random builds of the server can break us on Windows.
> > > >
> > > > I would hesitate to run these per-commit if they're taking that long,
> > but
> > > > daily is a thing that can easily happen.
> > > >
> > > > On Thu, May 16, 2019 at 2:23 PM Bruce Schuchardt <
> > bschuchardt@pivotal.io
> > > >
> > > > wrote:
> > > >
> > > >> big +1, as long as artifacts of failed runs can be downloaded
> > > >>
> > > >> On 5/15/19 6:28 PM, Owen Nichols wrote:
> > > >>> For a very long time we’ve had Windows tests in the main pipeline
> > > >> (hidden away, not in the default view), but the pipeline proceeds to
> > > >> publish regardless of whether Windows tests fail or even run at all.
> > > >>>
> > > >>> Now seems like a good time to review whether to:
> > > >>> a) treat Windows tests as first-class tests and prevent the
> pipeline
> > > >> from proceeding if any test fails on Windows
> > > >>> b) keep as-is
> > > >>> c) change Windows tests to trigger only once a week rather than on
> > > every
> > > >> commit, if they are going to remain "informational only"
> > > >>>
> > > >>> One disadvantage to making Windows tests gating is that they
> > currently
> > > >> take much longer to run (around 5 hours, vs 2 hours for Linux
> tests).
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] is it time to make Windows tests gating?

[Jens Deppe <jd...@pivotal.io>]

I fixed the first two and started ignoring
AvailablePortHelperIntegrationTest for Windows (Windows has a default
active port range wildly different than Linux which makes this test very
flaky, and since it's just testing test code it seemed reasonable to ignore
it).

LauncherMemberMXBeanIntegration test still needs some love and I can look
at that since I've worked on it in the past.

--Jens

On Fri, May 17, 2019 at 9:33 AM Dan Smith <ds...@pivotal.io> wrote:

> Looking at the metrics for the windows jobs, it looks like the windows
> tests are mostly red due to a few specific tests. The acceptance and gfsh
> distributed tests jobs seem to be ok, it's just the unit tests and
> integration tests that have problems. It also looks like
> ExportConfigCommandTest stopped failing recently. Anyone want to take the
> credit?
>
> I'd be in favor of putting them in the main pipeline but not the PR
> pipeline, since they take so long. Based on the metrics it looks like we
> are pretty close to green with these jobs in terms of effort. Anyone want
> to volunteer to fix the remaining failures (or see if they have been fixed
> recently)?
>
> -Dan
>
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsUnitTestOpenJDK8Metrics/builds/209
>
> CreateDiskStoreCommandTest:  2 failures
> ExportConfigCommandTest:  34 failures
>
>
> https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsIntegrationTestOpenJDK8Metrics/builds/209
>
> LauncherMemberMXBeanIntegrationTest:  4 failures
> AvailablePortHelperIntegrationTest:  17 failures
>
>
>
>
> On Thu, May 16, 2019 at 5:32 PM Owen Nichols <on...@pivotal.io> wrote:
>
> > I’ve created a PR for this: https://github.com/apache/geode/pull/3597
> >
> > > On May 16, 2019, at 3:06 PM, Blake Bender <bb...@pivotal.io> wrote:
> > >
> > > +1 this needs to happen.  I hope that doesn't cause too much pain for
> the
> > > dev team, but the native client team has a hard requirement that all
> our
> > > stuff works properly on Windows at all times, and it can cause trouble
> if
> > > random builds of the server can break us on Windows.
> > >
> > > I would hesitate to run these per-commit if they're taking that long,
> but
> > > daily is a thing that can easily happen.
> > >
> > > On Thu, May 16, 2019 at 2:23 PM Bruce Schuchardt <
> bschuchardt@pivotal.io
> > >
> > > wrote:
> > >
> > >> big +1, as long as artifacts of failed runs can be downloaded
> > >>
> > >> On 5/15/19 6:28 PM, Owen Nichols wrote:
> > >>> For a very long time we’ve had Windows tests in the main pipeline
> > >> (hidden away, not in the default view), but the pipeline proceeds to
> > >> publish regardless of whether Windows tests fail or even run at all.
> > >>>
> > >>> Now seems like a good time to review whether to:
> > >>> a) treat Windows tests as first-class tests and prevent the pipeline
> > >> from proceeding if any test fails on Windows
> > >>> b) keep as-is
> > >>> c) change Windows tests to trigger only once a week rather than on
> > every
> > >> commit, if they are going to remain "informational only"
> > >>>
> > >>> One disadvantage to making Windows tests gating is that they
> currently
> > >> take much longer to run (around 5 hours, vs 2 hours for Linux tests).
> > >>
> >
> >
>

Re: [DISCUSS] is it time to make Windows tests gating?

[Dan Smith <ds...@pivotal.io>]

Looking at the metrics for the windows jobs, it looks like the windows
tests are mostly red due to a few specific tests. The acceptance and gfsh
distributed tests jobs seem to be ok, it's just the unit tests and
integration tests that have problems. It also looks like
ExportConfigCommandTest stopped failing recently. Anyone want to take the
credit?

I'd be in favor of putting them in the main pipeline but not the PR
pipeline, since they take so long. Based on the metrics it looks like we
are pretty close to green with these jobs in terms of effort. Anyone want
to volunteer to fix the remaining failures (or see if they have been fixed
recently)?

-Dan

https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsUnitTestOpenJDK8Metrics/builds/209

CreateDiskStoreCommandTest:  2 failures
ExportConfigCommandTest:  34 failures

https://concourse.apachegeode-ci.info/teams/main/pipelines/apache-develop-metrics/jobs/GeodeWindowsIntegrationTestOpenJDK8Metrics/builds/209

LauncherMemberMXBeanIntegrationTest:  4 failures
AvailablePortHelperIntegrationTest:  17 failures




On Thu, May 16, 2019 at 5:32 PM Owen Nichols <on...@pivotal.io> wrote:

> I’ve created a PR for this: https://github.com/apache/geode/pull/3597
>
> > On May 16, 2019, at 3:06 PM, Blake Bender <bb...@pivotal.io> wrote:
> >
> > +1 this needs to happen.  I hope that doesn't cause too much pain for the
> > dev team, but the native client team has a hard requirement that all our
> > stuff works properly on Windows at all times, and it can cause trouble if
> > random builds of the server can break us on Windows.
> >
> > I would hesitate to run these per-commit if they're taking that long, but
> > daily is a thing that can easily happen.
> >
> > On Thu, May 16, 2019 at 2:23 PM Bruce Schuchardt <bschuchardt@pivotal.io
> >
> > wrote:
> >
> >> big +1, as long as artifacts of failed runs can be downloaded
> >>
> >> On 5/15/19 6:28 PM, Owen Nichols wrote:
> >>> For a very long time we’ve had Windows tests in the main pipeline
> >> (hidden away, not in the default view), but the pipeline proceeds to
> >> publish regardless of whether Windows tests fail or even run at all.
> >>>
> >>> Now seems like a good time to review whether to:
> >>> a) treat Windows tests as first-class tests and prevent the pipeline
> >> from proceeding if any test fails on Windows
> >>> b) keep as-is
> >>> c) change Windows tests to trigger only once a week rather than on
> every
> >> commit, if they are going to remain "informational only"
> >>>
> >>> One disadvantage to making Windows tests gating is that they currently
> >> take much longer to run (around 5 hours, vs 2 hours for Linux tests).
> >>
>
>

Re: [DISCUSS] is it time to make Windows tests gating?

[Owen Nichols <on...@pivotal.io>]

I’ve created a PR for this: https://github.com/apache/geode/pull/3597

> On May 16, 2019, at 3:06 PM, Blake Bender <bb...@pivotal.io> wrote:
> 
> +1 this needs to happen.  I hope that doesn't cause too much pain for the
> dev team, but the native client team has a hard requirement that all our
> stuff works properly on Windows at all times, and it can cause trouble if
> random builds of the server can break us on Windows.
> 
> I would hesitate to run these per-commit if they're taking that long, but
> daily is a thing that can easily happen.
> 
> On Thu, May 16, 2019 at 2:23 PM Bruce Schuchardt <bs...@pivotal.io>
> wrote:
> 
>> big +1, as long as artifacts of failed runs can be downloaded
>> 
>> On 5/15/19 6:28 PM, Owen Nichols wrote:
>>> For a very long time we’ve had Windows tests in the main pipeline
>> (hidden away, not in the default view), but the pipeline proceeds to
>> publish regardless of whether Windows tests fail or even run at all.
>>> 
>>> Now seems like a good time to review whether to:
>>> a) treat Windows tests as first-class tests and prevent the pipeline
>> from proceeding if any test fails on Windows
>>> b) keep as-is
>>> c) change Windows tests to trigger only once a week rather than on every
>> commit, if they are going to remain "informational only"
>>> 
>>> One disadvantage to making Windows tests gating is that they currently
>> take much longer to run (around 5 hours, vs 2 hours for Linux tests).
>> 

Re: [DISCUSS] is it time to make Windows tests gating?

[Blake Bender <bb...@pivotal.io>]

+1 this needs to happen.  I hope that doesn't cause too much pain for the
dev team, but the native client team has a hard requirement that all our
stuff works properly on Windows at all times, and it can cause trouble if
random builds of the server can break us on Windows.

I would hesitate to run these per-commit if they're taking that long, but
daily is a thing that can easily happen.

On Thu, May 16, 2019 at 2:23 PM Bruce Schuchardt <bs...@pivotal.io>
wrote:

> big +1, as long as artifacts of failed runs can be downloaded
>
> On 5/15/19 6:28 PM, Owen Nichols wrote:
> > For a very long time we’ve had Windows tests in the main pipeline
> (hidden away, not in the default view), but the pipeline proceeds to
> publish regardless of whether Windows tests fail or even run at all.
> >
> > Now seems like a good time to review whether to:
> > a) treat Windows tests as first-class tests and prevent the pipeline
> from proceeding if any test fails on Windows
> > b) keep as-is
> > c) change Windows tests to trigger only once a week rather than on every
> commit, if they are going to remain "informational only"
> >
> > One disadvantage to making Windows tests gating is that they currently
> take much longer to run (around 5 hours, vs 2 hours for Linux tests).
>

Re: [DISCUSS] is it time to make Windows tests gating?

[Bruce Schuchardt <bs...@pivotal.io>]

big +1, as long as artifacts of failed runs can be downloaded

On 5/15/19 6:28 PM, Owen Nichols wrote:
> For a very long time we’ve had Windows tests in the main pipeline (hidden away, not in the default view), but the pipeline proceeds to publish regardless of whether Windows tests fail or even run at all.
>
> Now seems like a good time to review whether to:
> a) treat Windows tests as first-class tests and prevent the pipeline from proceeding if any test fails on Windows
> b) keep as-is
> c) change Windows tests to trigger only once a week rather than on every commit, if they are going to remain "informational only"
>
> One disadvantage to making Windows tests gating is that they currently take much longer to run (around 5 hours, vs 2 hours for Linux tests).

Re: [DISCUSS] is it time to make Windows tests gating?

[Michael Oleske <mo...@pivotal.io>]

Agreed on making them gating if we want to say we run on Windows. If we
don't want to say we run on Windows, delete the jobs.

-michael

On Thursday, May 16, 2019, Anilkumar Gingade <ag...@pivotal.io> wrote:

> >> around 5 hours, vs 2 hours for Linux tests).
> May be a good time to look at reducing/optimizing this.
>
>
> On Thu, May 16, 2019 at 9:57 AM Ernest Burghardt <eb...@pivotal.io>
> wrote:
>
> > Yes make them gating.
> > Run them every commit, Windows is a supported platform.
> > Red boxes get attention and Red boxes get fixed.
> >
> > EB
> >
> > On Thu, May 16, 2019 at 1:09 AM Udo Kohlmeyer <ud...@apache.org> wrote:
> >
> > > I think we need to make sure our windows tests get to green... If we
> > > make them gating then we will never release, but at the time be
> > > motivated to fix them, in order to release.
> > >
> > > Maybe they run once every day... to at least start getting an idea of
> > > health
> > >
> > > On 5/15/19 18:28, Owen Nichols wrote:
> > > > For a very long time we’ve had Windows tests in the main pipeline
> > > (hidden away, not in the default view), but the pipeline proceeds to
> > > publish regardless of whether Windows tests fail or even run at all.
> > > >
> > > > Now seems like a good time to review whether to:
> > > > a) treat Windows tests as first-class tests and prevent the pipeline
> > > from proceeding if any test fails on Windows
> > > > b) keep as-is
> > > > c) change Windows tests to trigger only once a week rather than on
> > every
> > > commit, if they are going to remain "informational only"
> > > >
> > > > One disadvantage to making Windows tests gating is that they
> currently
> > > take much longer to run (around 5 hours, vs 2 hours for Linux tests).
> > >
> >
>

Re: [DISCUSS] is it time to make Windows tests gating?

[Anilkumar Gingade <ag...@pivotal.io>]

>> around 5 hours, vs 2 hours for Linux tests).
May be a good time to look at reducing/optimizing this.


On Thu, May 16, 2019 at 9:57 AM Ernest Burghardt <eb...@pivotal.io>
wrote:

> Yes make them gating.
> Run them every commit, Windows is a supported platform.
> Red boxes get attention and Red boxes get fixed.
>
> EB
>
> On Thu, May 16, 2019 at 1:09 AM Udo Kohlmeyer <ud...@apache.org> wrote:
>
> > I think we need to make sure our windows tests get to green... If we
> > make them gating then we will never release, but at the time be
> > motivated to fix them, in order to release.
> >
> > Maybe they run once every day... to at least start getting an idea of
> > health
> >
> > On 5/15/19 18:28, Owen Nichols wrote:
> > > For a very long time we’ve had Windows tests in the main pipeline
> > (hidden away, not in the default view), but the pipeline proceeds to
> > publish regardless of whether Windows tests fail or even run at all.
> > >
> > > Now seems like a good time to review whether to:
> > > a) treat Windows tests as first-class tests and prevent the pipeline
> > from proceeding if any test fails on Windows
> > > b) keep as-is
> > > c) change Windows tests to trigger only once a week rather than on
> every
> > commit, if they are going to remain "informational only"
> > >
> > > One disadvantage to making Windows tests gating is that they currently
> > take much longer to run (around 5 hours, vs 2 hours for Linux tests).
> >
>

Re: [DISCUSS] is it time to make Windows tests gating?

[Ernest Burghardt <eb...@pivotal.io>]

Yes make them gating.
Run them every commit, Windows is a supported platform.
Red boxes get attention and Red boxes get fixed.

EB

On Thu, May 16, 2019 at 1:09 AM Udo Kohlmeyer <ud...@apache.org> wrote:

> I think we need to make sure our windows tests get to green... If we
> make them gating then we will never release, but at the time be
> motivated to fix them, in order to release.
>
> Maybe they run once every day... to at least start getting an idea of
> health
>
> On 5/15/19 18:28, Owen Nichols wrote:
> > For a very long time we’ve had Windows tests in the main pipeline
> (hidden away, not in the default view), but the pipeline proceeds to
> publish regardless of whether Windows tests fail or even run at all.
> >
> > Now seems like a good time to review whether to:
> > a) treat Windows tests as first-class tests and prevent the pipeline
> from proceeding if any test fails on Windows
> > b) keep as-is
> > c) change Windows tests to trigger only once a week rather than on every
> commit, if they are going to remain "informational only"
> >
> > One disadvantage to making Windows tests gating is that they currently
> take much longer to run (around 5 hours, vs 2 hours for Linux tests).
>

Re: [DISCUSS] is it time to make Windows tests gating?

[Udo Kohlmeyer <ud...@apache.org>]

I think we need to make sure our windows tests get to green... If we 
make them gating then we will never release, but at the time be 
motivated to fix them, in order to release.

Maybe they run once every day... to at least start getting an idea of health

On 5/15/19 18:28, Owen Nichols wrote:
> For a very long time we’ve had Windows tests in the main pipeline (hidden away, not in the default view), but the pipeline proceeds to publish regardless of whether Windows tests fail or even run at all.
>
> Now seems like a good time to review whether to:
> a) treat Windows tests as first-class tests and prevent the pipeline from proceeding if any test fails on Windows
> b) keep as-is
> c) change Windows tests to trigger only once a week rather than on every commit, if they are going to remain "informational only"
>
> One disadvantage to making Windows tests gating is that they currently take much longer to run (around 5 hours, vs 2 hours for Linux tests).

Re: [DISCUSS] is it time to make Windows tests gating?

[Robert Houghton <rh...@pivotal.io>]

They are also as red as the day is long...

On Wed, May 15, 2019, 18:28 Owen Nichols <on...@pivotal.io> wrote:

> For a very long time we’ve had Windows tests in the main pipeline (hidden
> away, not in the default view), but the pipeline proceeds to publish
> regardless of whether Windows tests fail or even run at all.
>
> Now seems like a good time to review whether to:
> a) treat Windows tests as first-class tests and prevent the pipeline from
> proceeding if any test fails on Windows
> b) keep as-is
> c) change Windows tests to trigger only once a week rather than on every
> commit, if they are going to remain "informational only"
>
> One disadvantage to making Windows tests gating is that they currently
> take much longer to run (around 5 hours, vs 2 hours for Linux tests).