qpid-python-test script: should skipped tests be considered failures?

[Alan Conway <ac...@redhat.com>]

The qpid-python-test script has a facility for skipping tests (by
raising a Skipped exception) which works fine BUT if tests are skipped
the script exits with non-0 status - i.e. failure.

I propose we change this behavior. It is clear in the test output that
tests were skipped rather than failed, but that's not very useful when
the script is incorporated in larger test suites, CI frameworks etc.
where returning non-0 will be considered a failure and set people off in
failure investigation mode only to cause much annoyance when they dig
down and find skipped, not failed tests are causing the alarms. 

IMO skipping a test is different from failing - you skip because you
can't run the test for some environmental reason that has no bearing on
whether the functionality works, e.g. there's some library not installed
or what have you. We should certainly try to get skip stats reflected in
higher level tools that measure test health but I don't think we should
be ringing alarm bells as if something had failed.

Opinions? This behavior has been around for a long time so I'm wary of
changing it unilaterally. It is however the reason I've never used the
skip functionality and resorted to hacks like making tests pass but
print SKIP which is not as nice as doing it properly. 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: qpid-python-test script: should skipped tests be considered failures?

[Alan Conway <ac...@redhat.com>]

On Tue, 2014-01-28 at 08:09 -0500, Rafael Schloming wrote:
> FWIW, I didn't intent skpped tests to be counted as failures when I wrote
> the original script.. I'm guessing the fact that they are was either an
> oversight in the original code, or something that was introduced (possibly
> accidentally) at a later point. Either way I agree with you, it doesn't
> make much sense for them to count as failures. I'd classify it as a bug and
> go ahead and fix it.
> 

Done.
https://issues.apache.org/jira/browse/QPID-5520

> --Rafael
> 
> 
> On Mon, Jan 27, 2014 at 4:20 PM, Alan Conway <ac...@redhat.com> wrote:
> 
> > The qpid-python-test script has a facility for skipping tests (by
> > raising a Skipped exception) which works fine BUT if tests are skipped
> > the script exits with non-0 status - i.e. failure.
> >
> > I propose we change this behavior. It is clear in the test output that
> > tests were skipped rather than failed, but that's not very useful when
> > the script is incorporated in larger test suites, CI frameworks etc.
> > where returning non-0 will be considered a failure and set people off in
> > failure investigation mode only to cause much annoyance when they dig
> > down and find skipped, not failed tests are causing the alarms.
> >
> > IMO skipping a test is different from failing - you skip because you
> > can't run the test for some environmental reason that has no bearing on
> > whether the functionality works, e.g. there's some library not installed
> > or what have you. We should certainly try to get skip stats reflected in
> > higher level tools that measure test health but I don't think we should
> > be ringing alarm bells as if something had failed.
> >
> > Opinions? This behavior has been around for a long time so I'm wary of
> > changing it unilaterally. It is however the reason I've never used the
> > skip functionality and resorted to hacks like making tests pass but
> > print SKIP which is not as nice as doing it properly.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
> > For additional commands, e-mail: dev-help@qpid.apache.org
> >
> >



---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org

Re: qpid-python-test script: should skipped tests be considered failures?

[Rafael Schloming <rh...@alum.mit.edu>]

FWIW, I didn't intent skpped tests to be counted as failures when I wrote
the original script.. I'm guessing the fact that they are was either an
oversight in the original code, or something that was introduced (possibly
accidentally) at a later point. Either way I agree with you, it doesn't
make much sense for them to count as failures. I'd classify it as a bug and
go ahead and fix it.

--Rafael


On Mon, Jan 27, 2014 at 4:20 PM, Alan Conway <ac...@redhat.com> wrote:

> The qpid-python-test script has a facility for skipping tests (by
> raising a Skipped exception) which works fine BUT if tests are skipped
> the script exits with non-0 status - i.e. failure.
>
> I propose we change this behavior. It is clear in the test output that
> tests were skipped rather than failed, but that's not very useful when
> the script is incorporated in larger test suites, CI frameworks etc.
> where returning non-0 will be considered a failure and set people off in
> failure investigation mode only to cause much annoyance when they dig
> down and find skipped, not failed tests are causing the alarms.
>
> IMO skipping a test is different from failing - you skip because you
> can't run the test for some environmental reason that has no bearing on
> whether the functionality works, e.g. there's some library not installed
> or what have you. We should certainly try to get skip stats reflected in
> higher level tools that measure test health but I don't think we should
> be ringing alarm bells as if something had failed.
>
> Opinions? This behavior has been around for a long time so I'm wary of
> changing it unilaterally. It is however the reason I've never used the
> skip functionality and resorted to hacks like making tests pass but
> print SKIP which is not as nice as doing it properly.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
> For additional commands, e-mail: dev-help@qpid.apache.org
>
>