You are viewing a plain text version of this content. The canonical link for it is here.
Posted to qa@openoffice.apache.org by Rob Weir <ro...@apache.org> on 2013/08/14 18:59:47 UTC
Some thoughts on quality
We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
that we're doing this, and their are no arguments against it, shows
that we value quality. I'd like to take this a step further, and see
what we can learn from the defects in AOO 4.0.0 and what we can do
going forward to improve.
Quality, in the end, is a process, not a state of grace. We improve
by working smarter, not working harder. The goal should be to learn
and improve, as individuals and as a community.
Every regression that made it into 4.0.0 was added there by a
programmer. And the defect went undetected by testers. This is not
to blame. It just means that we're all human. We know that. We all
make mistakes. I make mistakes. A quality process is not about
becoming perfect, but about acknowledging that we make mistakes and
that certain formal and informal practices are needed to prevent and
detect these mistakes.
But enough about generalities. I'm hoping you'll join with me in
examining the 32 confirmed 4.0.0 regression defects and answering a
few questions:
1) What caused the bug? What was the "root cause"? Note:
"programmer error" is not really a cause. We should ask what caused
the error.
2) What can we do to prevent bugs like this from being checked in?
3) Why wasn't the bug found during testing? Was it not covered by any
existing test case? Was a test case run but the defect was not
recognized? Was the defect introduced into the software after the
tests had already been executed?
4) What can we do to ensure that bugs like this are caught during testing?
So 2 basic questions -- what went wrong and how can we prevent it in
the future, looked at from perspective of programmers and testers. If
we can keep these questions in mind, and try to answer them, we may be
able to find some patterns that can lead to some process changes for
AOO 4.1.
You can find the 4.0.0 regressions in Bugzilla here:
https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
Regards,
-Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
I apologize in advance if my note was note clear. I'm not at all
interested in off-the-cuff opinions. We all have our opinions. But
I'm only interested in fact-based analysis of the actual regressions
reported in BZ. Specifically: what caused the actually defects that
ended up in 4.0.0 and what could have been done to prevent it.
General recommendations, like "more time", not backed by specific
analysis, are not very useful. And remember, there will never be
enough time to improve quality with a suboptimal process. The goal
should be (IMHO) to improve the process, i.e., work smarter, not
harder.
Regards,
-Rob
On Wed, Aug 14, 2013 at 12:59 PM, Rob Weir <ro...@apache.org> wrote:
> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
> that we're doing this, and their are no arguments against it, shows
> that we value quality. I'd like to take this a step further, and see
> what we can learn from the defects in AOO 4.0.0 and what we can do
> going forward to improve.
>
> Quality, in the end, is a process, not a state of grace. We improve
> by working smarter, not working harder. The goal should be to learn
> and improve, as individuals and as a community.
>
> Every regression that made it into 4.0.0 was added there by a
> programmer. And the defect went undetected by testers. This is not
> to blame. It just means that we're all human. We know that. We all
> make mistakes. I make mistakes. A quality process is not about
> becoming perfect, but about acknowledging that we make mistakes and
> that certain formal and informal practices are needed to prevent and
> detect these mistakes.
>
> But enough about generalities. I'm hoping you'll join with me in
> examining the 32 confirmed 4.0.0 regression defects and answering a
> few questions:
>
> 1) What caused the bug? What was the "root cause"? Note:
> "programmer error" is not really a cause. We should ask what caused
> the error.
>
> 2) What can we do to prevent bugs like this from being checked in?
>
> 3) Why wasn't the bug found during testing? Was it not covered by any
> existing test case? Was a test case run but the defect was not
> recognized? Was the defect introduced into the software after the
> tests had already been executed?
>
> 4) What can we do to ensure that bugs like this are caught during testing?
>
> So 2 basic questions -- what went wrong and how can we prevent it in
> the future, looked at from perspective of programmers and testers. If
> we can keep these questions in mind, and try to answer them, we may be
> able to find some patterns that can lead to some process changes for
> AOO 4.1.
>
> You can find the 4.0.0 regressions in Bugzilla here:
>
> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>
>
> Regards,
>
> -Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
I apologize in advance if my note was note clear. I'm not at all
interested in off-the-cuff opinions. We all have our opinions. But
I'm only interested in fact-based analysis of the actual regressions
reported in BZ. Specifically: what caused the actually defects that
ended up in 4.0.0 and what could have been done to prevent it.
General recommendations, like "more time", not backed by specific
analysis, are not very useful. And remember, there will never be
enough time to improve quality with a suboptimal process. The goal
should be (IMHO) to improve the process, i.e., work smarter, not
harder.
Regards,
-Rob
On Wed, Aug 14, 2013 at 12:59 PM, Rob Weir <ro...@apache.org> wrote:
> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
> that we're doing this, and their are no arguments against it, shows
> that we value quality. I'd like to take this a step further, and see
> what we can learn from the defects in AOO 4.0.0 and what we can do
> going forward to improve.
>
> Quality, in the end, is a process, not a state of grace. We improve
> by working smarter, not working harder. The goal should be to learn
> and improve, as individuals and as a community.
>
> Every regression that made it into 4.0.0 was added there by a
> programmer. And the defect went undetected by testers. This is not
> to blame. It just means that we're all human. We know that. We all
> make mistakes. I make mistakes. A quality process is not about
> becoming perfect, but about acknowledging that we make mistakes and
> that certain formal and informal practices are needed to prevent and
> detect these mistakes.
>
> But enough about generalities. I'm hoping you'll join with me in
> examining the 32 confirmed 4.0.0 regression defects and answering a
> few questions:
>
> 1) What caused the bug? What was the "root cause"? Note:
> "programmer error" is not really a cause. We should ask what caused
> the error.
>
> 2) What can we do to prevent bugs like this from being checked in?
>
> 3) Why wasn't the bug found during testing? Was it not covered by any
> existing test case? Was a test case run but the defect was not
> recognized? Was the defect introduced into the software after the
> tests had already been executed?
>
> 4) What can we do to ensure that bugs like this are caught during testing?
>
> So 2 basic questions -- what went wrong and how can we prevent it in
> the future, looked at from perspective of programmers and testers. If
> we can keep these questions in mind, and try to answer them, we may be
> able to find some patterns that can lead to some process changes for
> AOO 4.1.
>
> You can find the 4.0.0 regressions in Bugzilla here:
>
> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>
>
> Regards,
>
> -Rob
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by David Gerard <dg...@gmail.com>.
On 14 August 2013 20:39, Rob Weir <ro...@apache.org> wrote:
> Maybe we need to call an earlier build the "RC" so it will get more
> attention? We had a complete test build that we were testing for
> over a month. But maybe it is ignored unless we call it an "RC"? In
> other words, there were many opportunities for users to help try 4.0
> before it was released, but maybe there opportunities were not well
> known.
>
It basically wasn't publicised. I knew about it because I actually went
looking for it (goal: to put an image on the Wikipedia article of the 4.0
sidebar). It took a bit of ferreting about to find the prereleases. I've
followed the list all this year, so I knew (a) it existed (b) what to ask
about; the percentage of AOO users on the dev list is all but invisible.
Suggestion 1: note prereleases on the blog and in the social media channels.
Suggestion 2: do an RC for 4.0.1 as well. Even if you don't think there's a
need to, and fully expect the RC to be byte-for-byte identical to the final
release, people will appreciate being asked.
The Cathedral and the Bazaar still applies. "Release early, release often."
You could have had six months' intense testing from users who were
seriously interested.
- d.
Re: Some thoughts on quality
Posted by Kay Schenk <ka...@gmail.com>.
On Wed, Aug 14, 2013 at 12:51 PM, Marcus (OOo) <ma...@wtnet.de> wrote:
> Am 08/14/2013 09:39 PM, schrieb Rob Weir:
>
> On Wed, Aug 14, 2013 at 3:33 PM, Hagar Delest<ha...@laposte.net>>
>> wrote:
>>
>>> Le 14/08/2013 21:29, Hagar Delest a écrit :
>>>
>>> Le 14/08/2013 21:01, Raphael Bircher a écrit :
>>>>
>>>> I like the idea of a public beta. But consider the numbers. The 40
>>>>>> or so regressions that were reported came from an install base (based
>>>>>> on download figures since 4.0.0 was released) of around 3 million
>>>>>> users. Realistically, can we expect anywhere near that number in a
>>>>>> public beta? Or is it more likely that a beta program has 10,000
>>>>>> users or fewer? I don't know the answer here. But certainly a
>>>>>> well-publicized and used beta will find more than a beta used by just
>>>>>> a few hundred users.
>>>>>>
>>>>>
>>>>> The public beta is from my point of view realy important. Even you have
>>>>> only 10'000 Downloads of a beta, you have normaly verry experianced
>>>>> Users
>>>>> there, like power users from Companies. They provide realy valua
>>>>> feedback.
>>>>> So from my point of view, this is one of the moast important changes
>>>>> we have
>>>>> to do. For all Feature release a beta version. And don't forget,
>>>>> people are
>>>>> realy happy to do beta tests. but many of them are maybe not willing to
>>>>> follow a mailing list.
>>>>>
>>>>
>>>>
>>>> +1.
>>>> OOo used to have RC versions strongly advertised, it could go up to 6 RC
>>>> before going final and it was very useful to spot the main bugs.
>>>>
>>>> And I forgot: the 2 main bugs (slow saving in MS Office formats and
>>> the Calc
>>> display issue under Windows) were reported in the forum only 2 days and 5
>>> days after the release! and we have had many topics for both afterward.
>>> A RC
>>> would have clearly spared the hassle.
>>>
>>> See:
>>> http://forum.openoffice.org/**en/forum/viewtopic.php?f=9&t=**63082<http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63082>
>>> http://forum.openoffice.org/**en/forum/viewtopic.php?f=9&t=**63161<http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63161>
>>>
>>>
>>
>> Maybe we need to call an earlier build the "RC" so it will get more
>> attention? We had a complete test build that we were testing for
>> over a month. But maybe it is ignored unless we call it an "RC"? In
>> other words, there were many opportunities for users to help try 4.0
>> before it was released, but maybe there opportunities were not well
>> known.
>>
>
> Déjà vu ;-)
>
> I remember that we had the same problem in the old project.
>
> Call it "Developer Snapshot" and you will have a few hundreads downloads
> with respective number of feedback. But call it (and announce it!) with a
> name that sounds more familar (like Early Access, Preview, Beta, RC) then
> we had much more. So, going public more early should bring us a higher
> number of feedback.
>
> Marcus
If this would help, then we should do it. Hopefully this will get users'
attention more.
>
>
>
> ------------------------------**------------------------------**---------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.**apache.org<de...@openoffice.apache.org>
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
>
--
-------------------------------------------------------------------------------------------------
MzK
"When in doubt, cop an attitude."
-- Cat laws
Re: Some thoughts on quality
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 08/14/2013 09:39 PM, schrieb Rob Weir:
> On Wed, Aug 14, 2013 at 3:33 PM, Hagar Delest<ha...@laposte.net> wrote:
>> Le 14/08/2013 21:29, Hagar Delest a écrit :
>>
>>> Le 14/08/2013 21:01, Raphael Bircher a écrit :
>>>
>>>>> I like the idea of a public beta. But consider the numbers. The 40
>>>>> or so regressions that were reported came from an install base (based
>>>>> on download figures since 4.0.0 was released) of around 3 million
>>>>> users. Realistically, can we expect anywhere near that number in a
>>>>> public beta? Or is it more likely that a beta program has 10,000
>>>>> users or fewer? I don't know the answer here. But certainly a
>>>>> well-publicized and used beta will find more than a beta used by just
>>>>> a few hundred users.
>>>>
>>>> The public beta is from my point of view realy important. Even you have
>>>> only 10'000 Downloads of a beta, you have normaly verry experianced Users
>>>> there, like power users from Companies. They provide realy valua feedback.
>>>> So from my point of view, this is one of the moast important changes we have
>>>> to do. For all Feature release a beta version. And don't forget, people are
>>>> realy happy to do beta tests. but many of them are maybe not willing to
>>>> follow a mailing list.
>>>
>>>
>>> +1.
>>> OOo used to have RC versions strongly advertised, it could go up to 6 RC
>>> before going final and it was very useful to spot the main bugs.
>>>
>> And I forgot: the 2 main bugs (slow saving in MS Office formats and the Calc
>> display issue under Windows) were reported in the forum only 2 days and 5
>> days after the release! and we have had many topics for both afterward. A RC
>> would have clearly spared the hassle.
>>
>> See:
>> http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63082
>> http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63161
>>
>
>
> Maybe we need to call an earlier build the "RC" so it will get more
> attention? We had a complete test build that we were testing for
> over a month. But maybe it is ignored unless we call it an "RC"? In
> other words, there were many opportunities for users to help try 4.0
> before it was released, but maybe there opportunities were not well
> known.
Déjà vu ;-)
I remember that we had the same problem in the old project.
Call it "Developer Snapshot" and you will have a few hundreads downloads
with respective number of feedback. But call it (and announce it!) with
a name that sounds more familar (like Early Access, Preview, Beta, RC)
then we had much more. So, going public more early should bring us a
higher number of feedback.
Marcus
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Fri, Aug 16, 2013 at 1:06 PM, Kay Schenk <ka...@gmail.com> wrote:
> On Fri, Aug 16, 2013 at 9:47 AM, Rob Weir <ro...@apache.org> wrote:
>
>> On Fri, Aug 16, 2013 at 12:30 PM, Kay Schenk <ka...@gmail.com> wrote:
>> > On Wed, Aug 14, 2013 at 3:35 PM, sebb <se...@gmail.com> wrote:
>> >
>> >> On 14 August 2013 23:10, Rob Weir <ro...@apache.org> wrote:
>> >> > On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) <ma...@wtnet.de>
>> >> wrote:
>> >> >> Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
>> >> >>
>> >> >>> Le 14/08/2013 21:39, Rob Weir a écrit :
>> >> >>>>
>> >> >>>> Maybe we need to call an earlier build the "RC" so it will get more
>> >> >>>> attention? We had a complete test build that we were testing for
>> >> >>>> over a month. But maybe it is ignored unless we call it an "RC"? In
>> >> >>>> other words, there were many opportunities for users to help try
>> 4.0
>> >> >>>> before it was released, but maybe there opportunities were not well
>> >> >>>> known.
>> >> >>>
>> >> >>>
>> >> >>> I think that it was too difficult to get to the dev versions and it
>> was
>> >> >>> too much complicated. There was no clear link to download a dev
>> version
>> >> >>> (I had to rely on the url in the messages from the dev list).
>> >> >>
>> >> >>
>> >> >> This was intended.
>> >> >>
>> >> >> We hadn't published the dev builds via Apache or SF mirrors but only
>> on
>> >> 2
>> >> >> people accounts. Apache policy says it's not allowed to publish them
>> >> for a
>> >> >> wider audience to save the servers from a high traffic load. It's the
>> >> job of
>> >> >> the mirrors to handle this.
>> >> >>
>> >> >>
>> >> >>> What I see (from a standard user point of view) for a RC:
>> >> >>> - When a dev version is almost done, rename it RC and make it known
>> >> >>> (blog and we would relay the announcement in the forums of course)
>> >> >>> - Have a link visible under the main download button of the download
>> >> >>
>> >> >>
>> >> >> Both can be done, depending where the install files are located.
>> >> >>
>> >> >>
>> >> >>> page (perhaps a similar button as a dedicated entry)
>> >> >>> - Make that RC installable in parallel with a stable version
>> >> >>> - No file association allowed for that RC by design
>> >> >>
>> >> >>
>> >> >> IMHO the last both points doesn't apply to a RC [1] as it wouldn't
>> be a
>> >> RC
>> >> >> anymore. One of the RC attributes is to change it into the final
>> release
>> >> >> with, e.g., just a file name change. But this has to be done without
>> any
>> >> >> code changes. Otherwise you have to change code parts, build again,
>> test
>> >> >> again, ... ;-)
>> >> >>
>> >> >> But a Beta release could go this separated way.
>> >> >>
>> >> >
>> >> > Right. A release is a release is a release. The basic requirements
>> >> > for every release still apply:
>> >> >
>> >> > 1) 3 PMC +1 votes
>> >> > 2) Must include source files
>> >> > 3) Digital signatures, hash files, etc.
>> >> >
>> >> > But we can have a "beta release", that follows these rules, and it
>> >> > would be acceptable. We can then host on the mirrors, publicize, etc.
>> >>
>> >> There would likely be some restrictions on how many extra downloads
>> >> are permitted.
>> >> For example, the ASF mirrors probably could not cope with a set of
>> >> betas of all the languages for all the OSes in addition to the current
>> >> GA release.
>> >>
>> >
>> > Looking again at this thread, and knowing the amount of time and
>> resources
>> > it takes to actually mount an approved release, I wonder if some of the
>> > issues we had with the 4.0 release could have been addressed by a longer
>> RC
>> > vote timeframe -- like 2 weeks or so. The vote this time was relatively
>> > short as I recall on both the original RC and RC2. We have over 400
>> people
>> > on this list. I would hope given a longer test period, we might receive
>> > better testing and feedback if we took a bit longer for approval.
>> >
>>
>> You can answer that question yourself. Look at the bugs that we're
>> fixing in 4.0.1. Which of those would have been found if the same
>> project members spent two more weeks reviewing the same code?
>>
>
> Well this was my point. Not the same members, but different members on this
> list who, due to the timeframe we had for voting on the RC, were not
> available for more extensive testing. It is the psychological difference
> between a "developer snapshot" and an RC candidate.
>
But then you have the fact that one of the more serious bugs was in
the code since November. So time alone is not the solution. It is
how we use the time that counts.
-Rob
>
>> I doubt it would have made much of a difference. The slow Excel
>> saving bug, for example, was in the code since November 2012. So this
>> is not question of time. I'm hoping we can all look closely at the
>> facts and come to a similar conclusion.
>>
>> The basic concept here is "defect yield", which is a measure of what %
>> of the defects in the code are found by a given testing technique.
>> We can consider the formal QA test cases as one technique. We can
>> consider the PMC voting period as another "testing technique",
>> although an informal one. We could consider beta testing, automated
>> testing, etc., all as different techniques.
>>
>> Hopefully it is clear that the defect yield of a technique does not
>> increase greatly by spending more time doing it. For example, suppose
>> our test cases were capable of finding 25% of the bugs in the code.
>> So we run the test cases once and find 23% of the bugs. We don't get
>> 25% because of human error in testing. We could double the amount of
>> time spent testing and run all the tests again. That might get us to
>> 24%. Not a very efficient use of incremental time.
>>
>> Ditto for PMC voting. The informal testing is going to just poke at
>> the surface. It will have some bug yield, less than the formal
>> testing did. But we all have our usage patterns, the subset of
>> features that we use. It is very likely that in two weeks we
>> exercise only slightly more of the features than we did in a one week
>> voting period. Diminishing returns. In any case, we need to escape
>> the incorrect notion that a RC vote is an invitation to start testing.
>> It is not. It is a sign-off period to confirm that testing has
>> already been satisfactorily completed and that there are no known
>> remaining serious issues. If anyone wants to help testing this should
>> be done far before the RC vote.
>>
>> In any case, I hope you see the pattern here. Repeating a testing
>> technique or doesn't really increase defect yields. It is the curse
>> of diminishing returns. If you want to find more bugs then you need
>> to do something like:
>>
>> 1) Write more test cases
>>
>
>> 2) Introduce a new testing technique that is orthogonal to the ones
>> already in use. A performance test would certainly be orthogonal to
>> the functional tests.
>>
>> 3) An early beta test cycle could help here, since that introduces a
>> new class of users.
>>
>
>
>>
>> Regards,
>>
>> -Rob
>>
>>
>> > I realize this is not the same as a beta -- we would still be using
>> > "development snapshots", but the RC could be provided in all languages we
>> > intend to use, etc.
>> >
>> >
>> >
>> >
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> >> For additional commands, e-mail: dev-help@openoffice.apache.org
>> >>
>> >>
>> >
>> >
>> > --
>> >
>> -------------------------------------------------------------------------------------------------
>> > MzK
>> >
>> > "When in doubt, cop an attitude."
>> > -- Cat laws
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>>
>
>
> --
> -------------------------------------------------------------------------------------------------
> MzK
>
> "When in doubt, cop an attitude."
> -- Cat laws
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Kay Schenk <ka...@gmail.com>.
On Fri, Aug 16, 2013 at 9:47 AM, Rob Weir <ro...@apache.org> wrote:
> On Fri, Aug 16, 2013 at 12:30 PM, Kay Schenk <ka...@gmail.com> wrote:
> > On Wed, Aug 14, 2013 at 3:35 PM, sebb <se...@gmail.com> wrote:
> >
> >> On 14 August 2013 23:10, Rob Weir <ro...@apache.org> wrote:
> >> > On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) <ma...@wtnet.de>
> >> wrote:
> >> >> Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
> >> >>
> >> >>> Le 14/08/2013 21:39, Rob Weir a écrit :
> >> >>>>
> >> >>>> Maybe we need to call an earlier build the "RC" so it will get more
> >> >>>> attention? We had a complete test build that we were testing for
> >> >>>> over a month. But maybe it is ignored unless we call it an "RC"? In
> >> >>>> other words, there were many opportunities for users to help try
> 4.0
> >> >>>> before it was released, but maybe there opportunities were not well
> >> >>>> known.
> >> >>>
> >> >>>
> >> >>> I think that it was too difficult to get to the dev versions and it
> was
> >> >>> too much complicated. There was no clear link to download a dev
> version
> >> >>> (I had to rely on the url in the messages from the dev list).
> >> >>
> >> >>
> >> >> This was intended.
> >> >>
> >> >> We hadn't published the dev builds via Apache or SF mirrors but only
> on
> >> 2
> >> >> people accounts. Apache policy says it's not allowed to publish them
> >> for a
> >> >> wider audience to save the servers from a high traffic load. It's the
> >> job of
> >> >> the mirrors to handle this.
> >> >>
> >> >>
> >> >>> What I see (from a standard user point of view) for a RC:
> >> >>> - When a dev version is almost done, rename it RC and make it known
> >> >>> (blog and we would relay the announcement in the forums of course)
> >> >>> - Have a link visible under the main download button of the download
> >> >>
> >> >>
> >> >> Both can be done, depending where the install files are located.
> >> >>
> >> >>
> >> >>> page (perhaps a similar button as a dedicated entry)
> >> >>> - Make that RC installable in parallel with a stable version
> >> >>> - No file association allowed for that RC by design
> >> >>
> >> >>
> >> >> IMHO the last both points doesn't apply to a RC [1] as it wouldn't
> be a
> >> RC
> >> >> anymore. One of the RC attributes is to change it into the final
> release
> >> >> with, e.g., just a file name change. But this has to be done without
> any
> >> >> code changes. Otherwise you have to change code parts, build again,
> test
> >> >> again, ... ;-)
> >> >>
> >> >> But a Beta release could go this separated way.
> >> >>
> >> >
> >> > Right. A release is a release is a release. The basic requirements
> >> > for every release still apply:
> >> >
> >> > 1) 3 PMC +1 votes
> >> > 2) Must include source files
> >> > 3) Digital signatures, hash files, etc.
> >> >
> >> > But we can have a "beta release", that follows these rules, and it
> >> > would be acceptable. We can then host on the mirrors, publicize, etc.
> >>
> >> There would likely be some restrictions on how many extra downloads
> >> are permitted.
> >> For example, the ASF mirrors probably could not cope with a set of
> >> betas of all the languages for all the OSes in addition to the current
> >> GA release.
> >>
> >
> > Looking again at this thread, and knowing the amount of time and
> resources
> > it takes to actually mount an approved release, I wonder if some of the
> > issues we had with the 4.0 release could have been addressed by a longer
> RC
> > vote timeframe -- like 2 weeks or so. The vote this time was relatively
> > short as I recall on both the original RC and RC2. We have over 400
> people
> > on this list. I would hope given a longer test period, we might receive
> > better testing and feedback if we took a bit longer for approval.
> >
>
> You can answer that question yourself. Look at the bugs that we're
> fixing in 4.0.1. Which of those would have been found if the same
> project members spent two more weeks reviewing the same code?
>
Well this was my point. Not the same members, but different members on this
list who, due to the timeframe we had for voting on the RC, were not
available for more extensive testing. It is the psychological difference
between a "developer snapshot" and an RC candidate.
> I doubt it would have made much of a difference. The slow Excel
> saving bug, for example, was in the code since November 2012. So this
> is not question of time. I'm hoping we can all look closely at the
> facts and come to a similar conclusion.
>
> The basic concept here is "defect yield", which is a measure of what %
> of the defects in the code are found by a given testing technique.
> We can consider the formal QA test cases as one technique. We can
> consider the PMC voting period as another "testing technique",
> although an informal one. We could consider beta testing, automated
> testing, etc., all as different techniques.
>
> Hopefully it is clear that the defect yield of a technique does not
> increase greatly by spending more time doing it. For example, suppose
> our test cases were capable of finding 25% of the bugs in the code.
> So we run the test cases once and find 23% of the bugs. We don't get
> 25% because of human error in testing. We could double the amount of
> time spent testing and run all the tests again. That might get us to
> 24%. Not a very efficient use of incremental time.
>
> Ditto for PMC voting. The informal testing is going to just poke at
> the surface. It will have some bug yield, less than the formal
> testing did. But we all have our usage patterns, the subset of
> features that we use. It is very likely that in two weeks we
> exercise only slightly more of the features than we did in a one week
> voting period. Diminishing returns. In any case, we need to escape
> the incorrect notion that a RC vote is an invitation to start testing.
> It is not. It is a sign-off period to confirm that testing has
> already been satisfactorily completed and that there are no known
> remaining serious issues. If anyone wants to help testing this should
> be done far before the RC vote.
>
> In any case, I hope you see the pattern here. Repeating a testing
> technique or doesn't really increase defect yields. It is the curse
> of diminishing returns. If you want to find more bugs then you need
> to do something like:
>
> 1) Write more test cases
>
> 2) Introduce a new testing technique that is orthogonal to the ones
> already in use. A performance test would certainly be orthogonal to
> the functional tests.
>
> 3) An early beta test cycle could help here, since that introduces a
> new class of users.
>
>
> Regards,
>
> -Rob
>
>
> > I realize this is not the same as a beta -- we would still be using
> > "development snapshots", but the RC could be provided in all languages we
> > intend to use, etc.
> >
> >
> >
> >
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> >> For additional commands, e-mail: dev-help@openoffice.apache.org
> >>
> >>
> >
> >
> > --
> >
> -------------------------------------------------------------------------------------------------
> > MzK
> >
> > "When in doubt, cop an attitude."
> > -- Cat laws
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
>
--
-------------------------------------------------------------------------------------------------
MzK
"When in doubt, cop an attitude."
-- Cat laws
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Fri, Aug 16, 2013 at 12:30 PM, Kay Schenk <ka...@gmail.com> wrote:
> On Wed, Aug 14, 2013 at 3:35 PM, sebb <se...@gmail.com> wrote:
>
>> On 14 August 2013 23:10, Rob Weir <ro...@apache.org> wrote:
>> > On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) <ma...@wtnet.de>
>> wrote:
>> >> Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
>> >>
>> >>> Le 14/08/2013 21:39, Rob Weir a écrit :
>> >>>>
>> >>>> Maybe we need to call an earlier build the "RC" so it will get more
>> >>>> attention? We had a complete test build that we were testing for
>> >>>> over a month. But maybe it is ignored unless we call it an "RC"? In
>> >>>> other words, there were many opportunities for users to help try 4.0
>> >>>> before it was released, but maybe there opportunities were not well
>> >>>> known.
>> >>>
>> >>>
>> >>> I think that it was too difficult to get to the dev versions and it was
>> >>> too much complicated. There was no clear link to download a dev version
>> >>> (I had to rely on the url in the messages from the dev list).
>> >>
>> >>
>> >> This was intended.
>> >>
>> >> We hadn't published the dev builds via Apache or SF mirrors but only on
>> 2
>> >> people accounts. Apache policy says it's not allowed to publish them
>> for a
>> >> wider audience to save the servers from a high traffic load. It's the
>> job of
>> >> the mirrors to handle this.
>> >>
>> >>
>> >>> What I see (from a standard user point of view) for a RC:
>> >>> - When a dev version is almost done, rename it RC and make it known
>> >>> (blog and we would relay the announcement in the forums of course)
>> >>> - Have a link visible under the main download button of the download
>> >>
>> >>
>> >> Both can be done, depending where the install files are located.
>> >>
>> >>
>> >>> page (perhaps a similar button as a dedicated entry)
>> >>> - Make that RC installable in parallel with a stable version
>> >>> - No file association allowed for that RC by design
>> >>
>> >>
>> >> IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a
>> RC
>> >> anymore. One of the RC attributes is to change it into the final release
>> >> with, e.g., just a file name change. But this has to be done without any
>> >> code changes. Otherwise you have to change code parts, build again, test
>> >> again, ... ;-)
>> >>
>> >> But a Beta release could go this separated way.
>> >>
>> >
>> > Right. A release is a release is a release. The basic requirements
>> > for every release still apply:
>> >
>> > 1) 3 PMC +1 votes
>> > 2) Must include source files
>> > 3) Digital signatures, hash files, etc.
>> >
>> > But we can have a "beta release", that follows these rules, and it
>> > would be acceptable. We can then host on the mirrors, publicize, etc.
>>
>> There would likely be some restrictions on how many extra downloads
>> are permitted.
>> For example, the ASF mirrors probably could not cope with a set of
>> betas of all the languages for all the OSes in addition to the current
>> GA release.
>>
>
> Looking again at this thread, and knowing the amount of time and resources
> it takes to actually mount an approved release, I wonder if some of the
> issues we had with the 4.0 release could have been addressed by a longer RC
> vote timeframe -- like 2 weeks or so. The vote this time was relatively
> short as I recall on both the original RC and RC2. We have over 400 people
> on this list. I would hope given a longer test period, we might receive
> better testing and feedback if we took a bit longer for approval.
>
You can answer that question yourself. Look at the bugs that we're
fixing in 4.0.1. Which of those would have been found if the same
project members spent two more weeks reviewing the same code?
I doubt it would have made much of a difference. The slow Excel
saving bug, for example, was in the code since November 2012. So this
is not question of time. I'm hoping we can all look closely at the
facts and come to a similar conclusion.
The basic concept here is "defect yield", which is a measure of what %
of the defects in the code are found by a given testing technique.
We can consider the formal QA test cases as one technique. We can
consider the PMC voting period as another "testing technique",
although an informal one. We could consider beta testing, automated
testing, etc., all as different techniques.
Hopefully it is clear that the defect yield of a technique does not
increase greatly by spending more time doing it. For example, suppose
our test cases were capable of finding 25% of the bugs in the code.
So we run the test cases once and find 23% of the bugs. We don't get
25% because of human error in testing. We could double the amount of
time spent testing and run all the tests again. That might get us to
24%. Not a very efficient use of incremental time.
Ditto for PMC voting. The informal testing is going to just poke at
the surface. It will have some bug yield, less than the formal
testing did. But we all have our usage patterns, the subset of
features that we use. It is very likely that in two weeks we
exercise only slightly more of the features than we did in a one week
voting period. Diminishing returns. In any case, we need to escape
the incorrect notion that a RC vote is an invitation to start testing.
It is not. It is a sign-off period to confirm that testing has
already been satisfactorily completed and that there are no known
remaining serious issues. If anyone wants to help testing this should
be done far before the RC vote.
In any case, I hope you see the pattern here. Repeating a testing
technique or doesn't really increase defect yields. It is the curse
of diminishing returns. If you want to find more bugs then you need
to do something like:
1) Write more test cases
2) Introduce a new testing technique that is orthogonal to the ones
already in use. A performance test would certainly be orthogonal to
the functional tests.
3) An early beta test cycle could help here, since that introduces a
new class of users.
Regards,
-Rob
> I realize this is not the same as a beta -- we would still be using
> "development snapshots", but the RC could be provided in all languages we
> intend to use, etc.
>
>
>
>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>>
>
>
> --
> -------------------------------------------------------------------------------------------------
> MzK
>
> "When in doubt, cop an attitude."
> -- Cat laws
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Kay Schenk <ka...@gmail.com>.
On Wed, Aug 14, 2013 at 3:35 PM, sebb <se...@gmail.com> wrote:
> On 14 August 2013 23:10, Rob Weir <ro...@apache.org> wrote:
> > On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) <ma...@wtnet.de>
> wrote:
> >> Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
> >>
> >>> Le 14/08/2013 21:39, Rob Weir a écrit :
> >>>>
> >>>> Maybe we need to call an earlier build the "RC" so it will get more
> >>>> attention? We had a complete test build that we were testing for
> >>>> over a month. But maybe it is ignored unless we call it an "RC"? In
> >>>> other words, there were many opportunities for users to help try 4.0
> >>>> before it was released, but maybe there opportunities were not well
> >>>> known.
> >>>
> >>>
> >>> I think that it was too difficult to get to the dev versions and it was
> >>> too much complicated. There was no clear link to download a dev version
> >>> (I had to rely on the url in the messages from the dev list).
> >>
> >>
> >> This was intended.
> >>
> >> We hadn't published the dev builds via Apache or SF mirrors but only on
> 2
> >> people accounts. Apache policy says it's not allowed to publish them
> for a
> >> wider audience to save the servers from a high traffic load. It's the
> job of
> >> the mirrors to handle this.
> >>
> >>
> >>> What I see (from a standard user point of view) for a RC:
> >>> - When a dev version is almost done, rename it RC and make it known
> >>> (blog and we would relay the announcement in the forums of course)
> >>> - Have a link visible under the main download button of the download
> >>
> >>
> >> Both can be done, depending where the install files are located.
> >>
> >>
> >>> page (perhaps a similar button as a dedicated entry)
> >>> - Make that RC installable in parallel with a stable version
> >>> - No file association allowed for that RC by design
> >>
> >>
> >> IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a
> RC
> >> anymore. One of the RC attributes is to change it into the final release
> >> with, e.g., just a file name change. But this has to be done without any
> >> code changes. Otherwise you have to change code parts, build again, test
> >> again, ... ;-)
> >>
> >> But a Beta release could go this separated way.
> >>
> >
> > Right. A release is a release is a release. The basic requirements
> > for every release still apply:
> >
> > 1) 3 PMC +1 votes
> > 2) Must include source files
> > 3) Digital signatures, hash files, etc.
> >
> > But we can have a "beta release", that follows these rules, and it
> > would be acceptable. We can then host on the mirrors, publicize, etc.
>
> There would likely be some restrictions on how many extra downloads
> are permitted.
> For example, the ASF mirrors probably could not cope with a set of
> betas of all the languages for all the OSes in addition to the current
> GA release.
>
Looking again at this thread, and knowing the amount of time and resources
it takes to actually mount an approved release, I wonder if some of the
issues we had with the 4.0 release could have been addressed by a longer RC
vote timeframe -- like 2 weeks or so. The vote this time was relatively
short as I recall on both the original RC and RC2. We have over 400 people
on this list. I would hope given a longer test period, we might receive
better testing and feedback if we took a bit longer for approval.
I realize this is not the same as a beta -- we would still be using
"development snapshots", but the RC could be provided in all languages we
intend to use, etc.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
>
--
-------------------------------------------------------------------------------------------------
MzK
"When in doubt, cop an attitude."
-- Cat laws
Re: Some thoughts on quality
Posted by sebb <se...@gmail.com>.
On 14 August 2013 23:10, Rob Weir <ro...@apache.org> wrote:
> On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) <ma...@wtnet.de> wrote:
>> Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
>>
>>> Le 14/08/2013 21:39, Rob Weir a écrit :
>>>>
>>>> Maybe we need to call an earlier build the "RC" so it will get more
>>>> attention? We had a complete test build that we were testing for
>>>> over a month. But maybe it is ignored unless we call it an "RC"? In
>>>> other words, there were many opportunities for users to help try 4.0
>>>> before it was released, but maybe there opportunities were not well
>>>> known.
>>>
>>>
>>> I think that it was too difficult to get to the dev versions and it was
>>> too much complicated. There was no clear link to download a dev version
>>> (I had to rely on the url in the messages from the dev list).
>>
>>
>> This was intended.
>>
>> We hadn't published the dev builds via Apache or SF mirrors but only on 2
>> people accounts. Apache policy says it's not allowed to publish them for a
>> wider audience to save the servers from a high traffic load. It's the job of
>> the mirrors to handle this.
>>
>>
>>> What I see (from a standard user point of view) for a RC:
>>> - When a dev version is almost done, rename it RC and make it known
>>> (blog and we would relay the announcement in the forums of course)
>>> - Have a link visible under the main download button of the download
>>
>>
>> Both can be done, depending where the install files are located.
>>
>>
>>> page (perhaps a similar button as a dedicated entry)
>>> - Make that RC installable in parallel with a stable version
>>> - No file association allowed for that RC by design
>>
>>
>> IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a RC
>> anymore. One of the RC attributes is to change it into the final release
>> with, e.g., just a file name change. But this has to be done without any
>> code changes. Otherwise you have to change code parts, build again, test
>> again, ... ;-)
>>
>> But a Beta release could go this separated way.
>>
>
> Right. A release is a release is a release. The basic requirements
> for every release still apply:
>
> 1) 3 PMC +1 votes
> 2) Must include source files
> 3) Digital signatures, hash files, etc.
>
> But we can have a "beta release", that follows these rules, and it
> would be acceptable. We can then host on the mirrors, publicize, etc.
There would likely be some restrictions on how many extra downloads
are permitted.
For example, the ASF mirrors probably could not cope with a set of
betas of all the languages for all the OSes in addition to the current
GA release.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 14, 2013 at 4:24 PM, Marcus (OOo) <ma...@wtnet.de> wrote:
> Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
>
>> Le 14/08/2013 21:39, Rob Weir a écrit :
>>>
>>> Maybe we need to call an earlier build the "RC" so it will get more
>>> attention? We had a complete test build that we were testing for
>>> over a month. But maybe it is ignored unless we call it an "RC"? In
>>> other words, there were many opportunities for users to help try 4.0
>>> before it was released, but maybe there opportunities were not well
>>> known.
>>
>>
>> I think that it was too difficult to get to the dev versions and it was
>> too much complicated. There was no clear link to download a dev version
>> (I had to rely on the url in the messages from the dev list).
>
>
> This was intended.
>
> We hadn't published the dev builds via Apache or SF mirrors but only on 2
> people accounts. Apache policy says it's not allowed to publish them for a
> wider audience to save the servers from a high traffic load. It's the job of
> the mirrors to handle this.
>
>
>> What I see (from a standard user point of view) for a RC:
>> - When a dev version is almost done, rename it RC and make it known
>> (blog and we would relay the announcement in the forums of course)
>> - Have a link visible under the main download button of the download
>
>
> Both can be done, depending where the install files are located.
>
>
>> page (perhaps a similar button as a dedicated entry)
>> - Make that RC installable in parallel with a stable version
>> - No file association allowed for that RC by design
>
>
> IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a RC
> anymore. One of the RC attributes is to change it into the final release
> with, e.g., just a file name change. But this has to be done without any
> code changes. Otherwise you have to change code parts, build again, test
> again, ... ;-)
>
> But a Beta release could go this separated way.
>
Right. A release is a release is a release. The basic requirements
for every release still apply:
1) 3 PMC +1 votes
2) Must include source files
3) Digital signatures, hash files, etc.
But we can have a "beta release", that follows these rules, and it
would be acceptable. We can then host on the mirrors, publicize, etc.
However, back to the original topic of this thread: We should look
to see when the bugs we're fixing in 4.0.1 were added to the code.
Not to blame or make anyone feel bad. But to understand. If these
were late bugs then an earlier beta would not have found them.
-Rob
>
>> The wiki page for the dev builds were too complicated and sounded to
>> much beta to make the users confident in using them (when they managed
>> to get on that page!).
>
>
> Marcus
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 08/14/2013 10:24 PM, schrieb Marcus (OOo):
> Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
>> Le 14/08/2013 21:39, Rob Weir a écrit :
>>> Maybe we need to call an earlier build the "RC" so it will get more
>>> attention? We had a complete test build that we were testing for
>>> over a month. But maybe it is ignored unless we call it an "RC"? In
>>> other words, there were many opportunities for users to help try 4.0
>>> before it was released, but maybe there opportunities were not well
>>> known.
>>
>> I think that it was too difficult to get to the dev versions and it was
>> too much complicated. There was no clear link to download a dev version
>> (I had to rely on the url in the messages from the dev list).
>
> This was intended.
>
> We hadn't published the dev builds via Apache or SF mirrors but only on
> 2 people accounts. Apache policy says it's not allowed to publish them
> for a wider audience to save the servers from a high traffic load. It's
> the job of the mirrors to handle this.
>
>> What I see (from a standard user point of view) for a RC:
>> - When a dev version is almost done, rename it RC and make it known
>> (blog and we would relay the announcement in the forums of course)
>> - Have a link visible under the main download button of the download
>
> Both can be done, depending where the install files are located.
>
>> page (perhaps a similar button as a dedicated entry)
>> - Make that RC installable in parallel with a stable version
>> - No file association allowed for that RC by design
>
> IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a
> RC anymore. One of the RC attributes is to change it into the final
> release with, e.g., just a file name change. But this has to be done
> without any code changes. Otherwise you have to change code parts, build
> again, test again, ... ;-)
>
> But a Beta release could go this separated way.
>
>> The wiki page for the dev builds were too complicated and sounded to
>> much beta to make the users confident in using them (when they managed
>> to get on that page!).
>
> Marcus
Sorry, forgot the link:
[1]
http://en.wikipedia.org/wiki/Software_release_life_cycle#Release_candidate
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 08/14/2013 10:05 PM, schrieb Hagar Delest:
> Le 14/08/2013 21:39, Rob Weir a écrit :
>> Maybe we need to call an earlier build the "RC" so it will get more
>> attention? We had a complete test build that we were testing for
>> over a month. But maybe it is ignored unless we call it an "RC"? In
>> other words, there were many opportunities for users to help try 4.0
>> before it was released, but maybe there opportunities were not well
>> known.
>
> I think that it was too difficult to get to the dev versions and it was
> too much complicated. There was no clear link to download a dev version
> (I had to rely on the url in the messages from the dev list).
This was intended.
We hadn't published the dev builds via Apache or SF mirrors but only on
2 people accounts. Apache policy says it's not allowed to publish them
for a wider audience to save the servers from a high traffic load. It's
the job of the mirrors to handle this.
> What I see (from a standard user point of view) for a RC:
> - When a dev version is almost done, rename it RC and make it known
> (blog and we would relay the announcement in the forums of course)
> - Have a link visible under the main download button of the download
Both can be done, depending where the install files are located.
> page (perhaps a similar button as a dedicated entry)
> - Make that RC installable in parallel with a stable version
> - No file association allowed for that RC by design
IMHO the last both points doesn't apply to a RC [1] as it wouldn't be a
RC anymore. One of the RC attributes is to change it into the final
release with, e.g., just a file name change. But this has to be done
without any code changes. Otherwise you have to change code parts, build
again, test again, ... ;-)
But a Beta release could go this separated way.
> The wiki page for the dev builds were too complicated and sounded to
> much beta to make the users confident in using them (when they managed
> to get on that page!).
Marcus
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Hagar Delest <ha...@laposte.net>.
Le 14/08/2013 21:39, Rob Weir a écrit :
> Maybe we need to call an earlier build the "RC" so it will get more
> attention? We had a complete test build that we were testing for
> over a month. But maybe it is ignored unless we call it an "RC"? In
> other words, there were many opportunities for users to help try 4.0
> before it was released, but maybe there opportunities were not well
> known.
I think that it was too difficult to get to the dev versions and it was too much complicated. There was no clear link to download a dev version (I had to rely on the url in the messages from the dev list).
What I see (from a standard user point of view) for a RC:
- When a dev version is almost done, rename it RC and make it known (blog and we would relay the announcement in the forums of course)
- Have a link visible under the main download button of the download page (perhaps a similar button as a dedicated entry)
- Make that RC installable in parallel with a stable version
- No file association allowed for that RC by design
The wiki page for the dev builds were too complicated and sounded to much beta to make the users confident in using them (when they managed to get on that page!).
Hagar
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 14, 2013 at 3:33 PM, Hagar Delest <ha...@laposte.net> wrote:
> Le 14/08/2013 21:29, Hagar Delest a écrit :
>
>> Le 14/08/2013 21:01, Raphael Bircher a écrit :
>>
>>>> I like the idea of a public beta. But consider the numbers. The 40
>>>> or so regressions that were reported came from an install base (based
>>>> on download figures since 4.0.0 was released) of around 3 million
>>>> users. Realistically, can we expect anywhere near that number in a
>>>> public beta? Or is it more likely that a beta program has 10,000
>>>> users or fewer? I don't know the answer here. But certainly a
>>>> well-publicized and used beta will find more than a beta used by just
>>>> a few hundred users.
>>>
>>> The public beta is from my point of view realy important. Even you have
>>> only 10'000 Downloads of a beta, you have normaly verry experianced Users
>>> there, like power users from Companies. They provide realy valua feedback.
>>> So from my point of view, this is one of the moast important changes we have
>>> to do. For all Feature release a beta version. And don't forget, people are
>>> realy happy to do beta tests. but many of them are maybe not willing to
>>> follow a mailing list.
>>
>>
>> +1.
>> OOo used to have RC versions strongly advertised, it could go up to 6 RC
>> before going final and it was very useful to spot the main bugs.
>>
> And I forgot: the 2 main bugs (slow saving in MS Office formats and the Calc
> display issue under Windows) were reported in the forum only 2 days and 5
> days after the release! and we have had many topics for both afterward. A RC
> would have clearly spared the hassle.
>
> See:
> http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63082
> http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63161
>
Maybe we need to call an earlier build the "RC" so it will get more
attention? We had a complete test build that we were testing for
over a month. But maybe it is ignored unless we call it an "RC"? In
other words, there were many opportunities for users to help try 4.0
before it was released, but maybe there opportunities were not well
known.
-Rob
>
> Hagar
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Hagar Delest <ha...@laposte.net>.
Le 14/08/2013 21:29, Hagar Delest a écrit :
> Le 14/08/2013 21:01, Raphael Bircher a écrit :
>
>>> I like the idea of a public beta. But consider the numbers. The 40
>>> or so regressions that were reported came from an install base (based
>>> on download figures since 4.0.0 was released) of around 3 million
>>> users. Realistically, can we expect anywhere near that number in a
>>> public beta? Or is it more likely that a beta program has 10,000
>>> users or fewer? I don't know the answer here. But certainly a
>>> well-publicized and used beta will find more than a beta used by just
>>> a few hundred users.
>> The public beta is from my point of view realy important. Even you have only 10'000 Downloads of a beta, you have normaly verry experianced Users there, like power users from Companies. They provide realy valua feedback. So from my point of view, this is one of the moast important changes we have to do. For all Feature release a beta version. And don't forget, people are realy happy to do beta tests. but many of them are maybe not willing to follow a mailing list.
>
> +1.
> OOo used to have RC versions strongly advertised, it could go up to 6 RC before going final and it was very useful to spot the main bugs.
>
And I forgot: the 2 main bugs (slow saving in MS Office formats and the Calc display issue under Windows) were reported in the forum only 2 days and 5 days after the release! and we have had many topics for both afterward. A RC would have clearly spared the hassle.
See:
http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63082
http://forum.openoffice.org/en/forum/viewtopic.php?f=9&t=63161
Hagar
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Hagar Delest <ha...@laposte.net>.
Le 14/08/2013 21:01, Raphael Bircher a écrit :
>> I like the idea of a public beta. But consider the numbers. The 40
>> or so regressions that were reported came from an install base (based
>> on download figures since 4.0.0 was released) of around 3 million
>> users. Realistically, can we expect anywhere near that number in a
>> public beta? Or is it more likely that a beta program has 10,000
>> users or fewer? I don't know the answer here. But certainly a
>> well-publicized and used beta will find more than a beta used by just
>> a few hundred users.
> The public beta is from my point of view realy important. Even you have only 10'000 Downloads of a beta, you have normaly verry experianced Users there, like power users from Companies. They provide realy valua feedback. So from my point of view, this is one of the moast important changes we have to do. For all Feature release a beta version. And don't forget, people are realy happy to do beta tests. but many of them are maybe not willing to follow a mailing list.
+1.
OOo used to have RC versions strongly advertised, it could go up to 6 RC before going final and it was very useful to spot the main bugs.
Hagar
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 08/14/2013 09:50 PM, schrieb Hagar Delest:
> Le 14/08/2013 21:37, Marcus (OOo) a écrit :
>> In general, my feeling is that it's too early to do a retrospective
>> and ask "what was good, what needs to be improved?" (to say it with
>> SCRUM words ;-) ) Just after the first major release.
> Well, I visit forums and contribute since OOo 2.0 and it's the first
> time we (volunteers in forums) advise to downgrade so often, even for a
> major upgrade (because of the slow saving time and the Calc display
> issue mainly).
> So there has been clearly a quality issue.
Sure, I don't want to deny this.
However, IMHO it's too early to look at the big picture and try to find
the black spots.
> About test case, it's strange that the 2 bugs above were not seen.
That seems indeen one hotspot I mentioned.
Marcus
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Hagar Delest <ha...@laposte.net>.
Le 14/08/2013 21:37, Marcus (OOo) a écrit :
> In general, my feeling is that it's too early to do a retrospective and ask "what was good, what needs to be improved?" (to say it with SCRUM words ;-) ) Just after the first major release.
Well, I visit forums and contribute since OOo 2.0 and it's the first time we (volunteers in forums) advise to downgrade so often, even for a major upgrade (because of the slow saving time and the Calc display issue mainly).
So there has been clearly a quality issue.
About test case, it's strange that the 2 bugs above were not seen.
Hagar
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Thu, Aug 15, 2013 at 2:59 AM, Edwin Sharp <el...@mail-page.com> wrote:
>
>
> On Wed, Aug 14, 2013, at 22:58, Rob Weir wrote:
>>
>> Finally, I think we can all point to a similar open source project
>> that has numerous betas, but still suffers from poor quality. So a
>> public beta, by itself, is not sufficient. We need some upstream
>> improvements as well, I think. But we should do a beta as well. But
>> aim to have the highest quality beta we can, right?
>>
>> Regards,
>>
>> -Rob
>
> This comparison is not fair because the release frequency of the two projects is totally different.
>
To the original question you had suggested more frequent releases.
Raphael and Hagar suggested beta releases. So it is entirely relevant
to point out that a sister project that had adopted both of those
practices has not achieved any great quality improvements.
When you think of it, why should release pacing have any impact on
quality? Quality is increased if you prevent bugs, detect them before
you release, or fix them before you release. Anything that claims to
improve quality should have a direct impact on one of those three
actions. Does a release prevent bugs? Detect them? Fix them? No,
not really.
What a release can do is prompt increase QA attention as the release
approaches. But this is an indirect and not obligatory practice. We
could have frequent releases that are tested less, because of the less
time available for testing. This would lead to lower quality. Or we
could have multiple full test passes with a longer release, taking
better advantage of the time, and leading to higher quality.
So the thing that matters here is the testing time, testing coverage,
testing efficiency, etc. The release pace has no direct effect on
quality.
> Quality can also be improved by better community culture. No offense, but I found this inappropriate:
>
Culture is important, yes, to the extent it leads to the adoption of
quality practices. One good cultural point would be a community that
is not offended by facts and fact-based reasoning.
Regards,
-Rob
> On Fri, Jun 7, 2013, at 3:33, Rob Weir wrote:
>>
>> I'll try to clean up a few tonight while I watch TV.
>>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: qa-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Thu, Aug 15, 2013 at 2:59 AM, Edwin Sharp <el...@mail-page.com> wrote:
>
>
> On Wed, Aug 14, 2013, at 22:58, Rob Weir wrote:
>>
>> Finally, I think we can all point to a similar open source project
>> that has numerous betas, but still suffers from poor quality. So a
>> public beta, by itself, is not sufficient. We need some upstream
>> improvements as well, I think. But we should do a beta as well. But
>> aim to have the highest quality beta we can, right?
>>
>> Regards,
>>
>> -Rob
>
> This comparison is not fair because the release frequency of the two projects is totally different.
>
To the original question you had suggested more frequent releases.
Raphael and Hagar suggested beta releases. So it is entirely relevant
to point out that a sister project that had adopted both of those
practices has not achieved any great quality improvements.
When you think of it, why should release pacing have any impact on
quality? Quality is increased if you prevent bugs, detect them before
you release, or fix them before you release. Anything that claims to
improve quality should have a direct impact on one of those three
actions. Does a release prevent bugs? Detect them? Fix them? No,
not really.
What a release can do is prompt increase QA attention as the release
approaches. But this is an indirect and not obligatory practice. We
could have frequent releases that are tested less, because of the less
time available for testing. This would lead to lower quality. Or we
could have multiple full test passes with a longer release, taking
better advantage of the time, and leading to higher quality.
So the thing that matters here is the testing time, testing coverage,
testing efficiency, etc. The release pace has no direct effect on
quality.
> Quality can also be improved by better community culture. No offense, but I found this inappropriate:
>
Culture is important, yes, to the extent it leads to the adoption of
quality practices. One good cultural point would be a community that
is not offended by facts and fact-based reasoning.
Regards,
-Rob
> On Fri, Jun 7, 2013, at 3:33, Rob Weir wrote:
>>
>> I'll try to clean up a few tonight while I watch TV.
>>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: qa-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Edwin Sharp <el...@mail-page.com>.
On Wed, Aug 14, 2013, at 22:58, Rob Weir wrote:
>
> Finally, I think we can all point to a similar open source project
> that has numerous betas, but still suffers from poor quality. So a
> public beta, by itself, is not sufficient. We need some upstream
> improvements as well, I think. But we should do a beta as well. But
> aim to have the highest quality beta we can, right?
>
> Regards,
>
> -Rob
This comparison is not fair because the release frequency of the two projects is totally different.
Quality can also be improved by better community culture. No offense, but I found this inappropriate:
On Fri, Jun 7, 2013, at 3:33, Rob Weir wrote:
>
> I'll try to clean up a few tonight while I watch TV.
>
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 08/14/2013 09:58 PM, schrieb Rob Weir:
> On Wed, Aug 14, 2013 at 3:37 PM, Marcus (OOo)<ma...@wtnet.de> wrote:
>> Am 08/14/2013 09:01 PM, schrieb Raphael Bircher:
>>
>>> Am 14.08.13 20:21, schrieb Rob Weir:
>>>>
>>>> On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp<el...@mail-page.com> wrote:
>>>>>
>>>>> Dear Rob
>>>>> The 4.0 release was too ambitious - we should advance in smaller steps.
>>>>> Nothing compares to general public testing - betas and release
>>>>> candidates should not be avoided.
>>>>> TestLink cases should be less comprehesive (in terms of feature
>>>>> coverage) and more stress testing oriented.
>>>>
>>>> The number to consider here is how many defects were found and fixed
>>>> during the 4.0.0 testing, before the general public users had access?
>>>> I assume it was quite substantial. If so, the TestLink usage was
>>>> effective. In other words, we might have found fewer bugs without
>>>> using it.
>>
>>
>> In general, my feeling is that it's too early to do a retrospective and ask
>> "what was good, what needs to be improved?" (to say it with SCRUM words ;-)
>> ) Just after the first major release.
>>
>
> Memories are still fresh and programmers are looking at the relevant
> code. This is the best time to answer these questions.
>
>> We should look on the BZ query you have mentioned and see if there is one or
>> more hotspots that should be improved fast. That's it.
>>
>
> That would be fine. I'm not suggesting anything radical. Gradual,
> but constant improvements are the way to high quality.
Then this should be sufficiant IMHO.
>>>> This is important to keep in mind: we want to prevent or find more
>>>> bugs, but we're not starting from zero. We're starting from a process
>>>> that does a lot of things right.
>>>>
>>>> I like the idea of a public beta. But consider the numbers. The 40
>>>> or so regressions that were reported came from an install base (based
>>>> on download figures since 4.0.0 was released) of around 3 million
>>>> users. Realistically, can we expect anywhere near that number in a
>>>> public beta? Or is it more likely that a beta program has 10,000
>>>> users or fewer? I don't know the answer here. But certainly a
>>>> well-publicized and used beta will find more than a beta used by just
>>>> a few hundred users.
>>>
>>> The public beta is from my point of view realy important. Even you have
>>> only 10'000 Downloads of a beta, you have normaly verry experianced
>>> Users there, like power users from Companies. They provide realy valua
>>> feedback. So from my point of view, this is one of the moast important
>>> changes we have to do. For all Feature release a beta version. And don't
>>> forget, people are realy happy to do beta tests. but many of them are
>>> maybe not willing to follow a mailing list.
>>
>>
>> A public beta release is of course not the golden solution but could
>> activate some power users that give us the feedback we want and need.
>>
>> So, +1 for going this way.
>>
>> After the 4.1 release is done we can see if this was much better - and ask
>> ourself why? :-)
>>
>
> But several of the bugs in 4.0.0 should never have made it to even a
> beta. For example, the very slow saving in Excel format. What
> happened there?
>
> Sure, this could be fixed later in the process, in a beta, or in a
> 4.0.1 as we're doing now. But this really should have been detected
> and fixed much earlier in the process.
>
> What are betas good for? Betas are good for expanding the set of
> configurations. Platforms, languages, extensions, etc. But the
> informal "tests" done by real users tend to be shallow. Also, we have
> no way of determining what the test coverage is. In particular we
> have no basis for telling the difference between the coverage of a 2
> week beta versus a 4 week beta. But with out formal test cases we can
> easily look at how many test cases were executed. We could even look
> at code coverage if we wanted to.
>
> Maybe if there was a way to record what features beta testers were
> using... Or even have a little survey where they report what
> platform they ran on, etc.
>
> But very slow saving to XLS files? A defect like that should have
> been caught by us before a beta and before a RC. We shouldn't expect
> to find bugs like that in a beta.
Of course. I've said nothing different. ;-)
> Finally, I think we can all point to a similar open source project
> that has numerous betas, but still suffers from poor quality. So a
> public beta, by itself, is not sufficient. We need some upstream
> improvements as well, I think. But we should do a beta as well. But
> aim to have the highest quality beta we can, right?
Sure, the XLS bug has nothing to with "this wouldn't happen with a
Beta". As I answered to Hagar this is one hotspot that should be looked
at closer.
Marcus
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 08/14/2013 09:58 PM, schrieb Rob Weir:
> On Wed, Aug 14, 2013 at 3:37 PM, Marcus (OOo)<ma...@wtnet.de> wrote:
>> Am 08/14/2013 09:01 PM, schrieb Raphael Bircher:
>>
>>> Am 14.08.13 20:21, schrieb Rob Weir:
>>>>
>>>> On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp<el...@mail-page.com> wrote:
>>>>>
>>>>> Dear Rob
>>>>> The 4.0 release was too ambitious - we should advance in smaller steps.
>>>>> Nothing compares to general public testing - betas and release
>>>>> candidates should not be avoided.
>>>>> TestLink cases should be less comprehesive (in terms of feature
>>>>> coverage) and more stress testing oriented.
>>>>
>>>> The number to consider here is how many defects were found and fixed
>>>> during the 4.0.0 testing, before the general public users had access?
>>>> I assume it was quite substantial. If so, the TestLink usage was
>>>> effective. In other words, we might have found fewer bugs without
>>>> using it.
>>
>>
>> In general, my feeling is that it's too early to do a retrospective and ask
>> "what was good, what needs to be improved?" (to say it with SCRUM words ;-)
>> ) Just after the first major release.
>>
>
> Memories are still fresh and programmers are looking at the relevant
> code. This is the best time to answer these questions.
>
>> We should look on the BZ query you have mentioned and see if there is one or
>> more hotspots that should be improved fast. That's it.
>>
>
> That would be fine. I'm not suggesting anything radical. Gradual,
> but constant improvements are the way to high quality.
Then this should be sufficiant IMHO.
>>>> This is important to keep in mind: we want to prevent or find more
>>>> bugs, but we're not starting from zero. We're starting from a process
>>>> that does a lot of things right.
>>>>
>>>> I like the idea of a public beta. But consider the numbers. The 40
>>>> or so regressions that were reported came from an install base (based
>>>> on download figures since 4.0.0 was released) of around 3 million
>>>> users. Realistically, can we expect anywhere near that number in a
>>>> public beta? Or is it more likely that a beta program has 10,000
>>>> users or fewer? I don't know the answer here. But certainly a
>>>> well-publicized and used beta will find more than a beta used by just
>>>> a few hundred users.
>>>
>>> The public beta is from my point of view realy important. Even you have
>>> only 10'000 Downloads of a beta, you have normaly verry experianced
>>> Users there, like power users from Companies. They provide realy valua
>>> feedback. So from my point of view, this is one of the moast important
>>> changes we have to do. For all Feature release a beta version. And don't
>>> forget, people are realy happy to do beta tests. but many of them are
>>> maybe not willing to follow a mailing list.
>>
>>
>> A public beta release is of course not the golden solution but could
>> activate some power users that give us the feedback we want and need.
>>
>> So, +1 for going this way.
>>
>> After the 4.1 release is done we can see if this was much better - and ask
>> ourself why? :-)
>>
>
> But several of the bugs in 4.0.0 should never have made it to even a
> beta. For example, the very slow saving in Excel format. What
> happened there?
>
> Sure, this could be fixed later in the process, in a beta, or in a
> 4.0.1 as we're doing now. But this really should have been detected
> and fixed much earlier in the process.
>
> What are betas good for? Betas are good for expanding the set of
> configurations. Platforms, languages, extensions, etc. But the
> informal "tests" done by real users tend to be shallow. Also, we have
> no way of determining what the test coverage is. In particular we
> have no basis for telling the difference between the coverage of a 2
> week beta versus a 4 week beta. But with out formal test cases we can
> easily look at how many test cases were executed. We could even look
> at code coverage if we wanted to.
>
> Maybe if there was a way to record what features beta testers were
> using... Or even have a little survey where they report what
> platform they ran on, etc.
>
> But very slow saving to XLS files? A defect like that should have
> been caught by us before a beta and before a RC. We shouldn't expect
> to find bugs like that in a beta.
Of course. I've said nothing different. ;-)
> Finally, I think we can all point to a similar open source project
> that has numerous betas, but still suffers from poor quality. So a
> public beta, by itself, is not sufficient. We need some upstream
> improvements as well, I think. But we should do a beta as well. But
> aim to have the highest quality beta we can, right?
Sure, the XLS bug has nothing to with "this wouldn't happen with a
Beta". As I answered to Hagar this is one hotspot that should be looked
at closer.
Marcus
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 14, 2013 at 3:37 PM, Marcus (OOo) <ma...@wtnet.de> wrote:
> Am 08/14/2013 09:01 PM, schrieb Raphael Bircher:
>
>> Am 14.08.13 20:21, schrieb Rob Weir:
>>>
>>> On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp <el...@mail-page.com> wrote:
>>>>
>>>> Dear Rob
>>>> The 4.0 release was too ambitious - we should advance in smaller steps.
>>>> Nothing compares to general public testing - betas and release
>>>> candidates should not be avoided.
>>>> TestLink cases should be less comprehesive (in terms of feature
>>>> coverage) and more stress testing oriented.
>>>
>>> The number to consider here is how many defects were found and fixed
>>> during the 4.0.0 testing, before the general public users had access?
>>> I assume it was quite substantial. If so, the TestLink usage was
>>> effective. In other words, we might have found fewer bugs without
>>> using it.
>
>
> In general, my feeling is that it's too early to do a retrospective and ask
> "what was good, what needs to be improved?" (to say it with SCRUM words ;-)
> ) Just after the first major release.
>
Memories are still fresh and programmers are looking at the relevant
code. This is the best time to answer these questions.
> We should look on the BZ query you have mentioned and see if there is one or
> more hotspots that should be improved fast. That's it.
>
That would be fine. I'm not suggesting anything radical. Gradual,
but constant improvements are the way to high quality.
>
>>> This is important to keep in mind: we want to prevent or find more
>>> bugs, but we're not starting from zero. We're starting from a process
>>> that does a lot of things right.
>>>
>>> I like the idea of a public beta. But consider the numbers. The 40
>>> or so regressions that were reported came from an install base (based
>>> on download figures since 4.0.0 was released) of around 3 million
>>> users. Realistically, can we expect anywhere near that number in a
>>> public beta? Or is it more likely that a beta program has 10,000
>>> users or fewer? I don't know the answer here. But certainly a
>>> well-publicized and used beta will find more than a beta used by just
>>> a few hundred users.
>>
>> The public beta is from my point of view realy important. Even you have
>> only 10'000 Downloads of a beta, you have normaly verry experianced
>> Users there, like power users from Companies. They provide realy valua
>> feedback. So from my point of view, this is one of the moast important
>> changes we have to do. For all Feature release a beta version. And don't
>> forget, people are realy happy to do beta tests. but many of them are
>> maybe not willing to follow a mailing list.
>
>
> A public beta release is of course not the golden solution but could
> activate some power users that give us the feedback we want and need.
>
> So, +1 for going this way.
>
> After the 4.1 release is done we can see if this was much better - and ask
> ourself why? :-)
>
But several of the bugs in 4.0.0 should never have made it to even a
beta. For example, the very slow saving in Excel format. What
happened there?
Sure, this could be fixed later in the process, in a beta, or in a
4.0.1 as we're doing now. But this really should have been detected
and fixed much earlier in the process.
What are betas good for? Betas are good for expanding the set of
configurations. Platforms, languages, extensions, etc. But the
informal "tests" done by real users tend to be shallow. Also, we have
no way of determining what the test coverage is. In particular we
have no basis for telling the difference between the coverage of a 2
week beta versus a 4 week beta. But with out formal test cases we can
easily look at how many test cases were executed. We could even look
at code coverage if we wanted to.
Maybe if there was a way to record what features beta testers were
using... Or even have a little survey where they report what
platform they ran on, etc.
But very slow saving to XLS files? A defect like that should have
been caught by us before a beta and before a RC. We shouldn't expect
to find bugs like that in a beta.
Finally, I think we can all point to a similar open source project
that has numerous betas, but still suffers from poor quality. So a
public beta, by itself, is not sufficient. We need some upstream
improvements as well, I think. But we should do a beta as well. But
aim to have the highest quality beta we can, right?
Regards,
-Rob
> Marcus
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 14, 2013 at 3:37 PM, Marcus (OOo) <ma...@wtnet.de> wrote:
> Am 08/14/2013 09:01 PM, schrieb Raphael Bircher:
>
>> Am 14.08.13 20:21, schrieb Rob Weir:
>>>
>>> On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp <el...@mail-page.com> wrote:
>>>>
>>>> Dear Rob
>>>> The 4.0 release was too ambitious - we should advance in smaller steps.
>>>> Nothing compares to general public testing - betas and release
>>>> candidates should not be avoided.
>>>> TestLink cases should be less comprehesive (in terms of feature
>>>> coverage) and more stress testing oriented.
>>>
>>> The number to consider here is how many defects were found and fixed
>>> during the 4.0.0 testing, before the general public users had access?
>>> I assume it was quite substantial. If so, the TestLink usage was
>>> effective. In other words, we might have found fewer bugs without
>>> using it.
>
>
> In general, my feeling is that it's too early to do a retrospective and ask
> "what was good, what needs to be improved?" (to say it with SCRUM words ;-)
> ) Just after the first major release.
>
Memories are still fresh and programmers are looking at the relevant
code. This is the best time to answer these questions.
> We should look on the BZ query you have mentioned and see if there is one or
> more hotspots that should be improved fast. That's it.
>
That would be fine. I'm not suggesting anything radical. Gradual,
but constant improvements are the way to high quality.
>
>>> This is important to keep in mind: we want to prevent or find more
>>> bugs, but we're not starting from zero. We're starting from a process
>>> that does a lot of things right.
>>>
>>> I like the idea of a public beta. But consider the numbers. The 40
>>> or so regressions that were reported came from an install base (based
>>> on download figures since 4.0.0 was released) of around 3 million
>>> users. Realistically, can we expect anywhere near that number in a
>>> public beta? Or is it more likely that a beta program has 10,000
>>> users or fewer? I don't know the answer here. But certainly a
>>> well-publicized and used beta will find more than a beta used by just
>>> a few hundred users.
>>
>> The public beta is from my point of view realy important. Even you have
>> only 10'000 Downloads of a beta, you have normaly verry experianced
>> Users there, like power users from Companies. They provide realy valua
>> feedback. So from my point of view, this is one of the moast important
>> changes we have to do. For all Feature release a beta version. And don't
>> forget, people are realy happy to do beta tests. but many of them are
>> maybe not willing to follow a mailing list.
>
>
> A public beta release is of course not the golden solution but could
> activate some power users that give us the feedback we want and need.
>
> So, +1 for going this way.
>
> After the 4.1 release is done we can see if this was much better - and ask
> ourself why? :-)
>
But several of the bugs in 4.0.0 should never have made it to even a
beta. For example, the very slow saving in Excel format. What
happened there?
Sure, this could be fixed later in the process, in a beta, or in a
4.0.1 as we're doing now. But this really should have been detected
and fixed much earlier in the process.
What are betas good for? Betas are good for expanding the set of
configurations. Platforms, languages, extensions, etc. But the
informal "tests" done by real users tend to be shallow. Also, we have
no way of determining what the test coverage is. In particular we
have no basis for telling the difference between the coverage of a 2
week beta versus a 4 week beta. But with out formal test cases we can
easily look at how many test cases were executed. We could even look
at code coverage if we wanted to.
Maybe if there was a way to record what features beta testers were
using... Or even have a little survey where they report what
platform they ran on, etc.
But very slow saving to XLS files? A defect like that should have
been caught by us before a beta and before a RC. We shouldn't expect
to find bugs like that in a beta.
Finally, I think we can all point to a similar open source project
that has numerous betas, but still suffers from poor quality. So a
public beta, by itself, is not sufficient. We need some upstream
improvements as well, I think. But we should do a beta as well. But
aim to have the highest quality beta we can, right?
Regards,
-Rob
> Marcus
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 08/14/2013 09:01 PM, schrieb Raphael Bircher:
> Am 14.08.13 20:21, schrieb Rob Weir:
>> On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp <el...@mail-page.com> wrote:
>>> Dear Rob
>>> The 4.0 release was too ambitious - we should advance in smaller steps.
>>> Nothing compares to general public testing - betas and release
>>> candidates should not be avoided.
>>> TestLink cases should be less comprehesive (in terms of feature
>>> coverage) and more stress testing oriented.
>> The number to consider here is how many defects were found and fixed
>> during the 4.0.0 testing, before the general public users had access?
>> I assume it was quite substantial. If so, the TestLink usage was
>> effective. In other words, we might have found fewer bugs without
>> using it.
In general, my feeling is that it's too early to do a retrospective and
ask "what was good, what needs to be improved?" (to say it with SCRUM
words ;-) ) Just after the first major release.
We should look on the BZ query you have mentioned and see if there is
one or more hotspots that should be improved fast. That's it.
>> This is important to keep in mind: we want to prevent or find more
>> bugs, but we're not starting from zero. We're starting from a process
>> that does a lot of things right.
>>
>> I like the idea of a public beta. But consider the numbers. The 40
>> or so regressions that were reported came from an install base (based
>> on download figures since 4.0.0 was released) of around 3 million
>> users. Realistically, can we expect anywhere near that number in a
>> public beta? Or is it more likely that a beta program has 10,000
>> users or fewer? I don't know the answer here. But certainly a
>> well-publicized and used beta will find more than a beta used by just
>> a few hundred users.
> The public beta is from my point of view realy important. Even you have
> only 10'000 Downloads of a beta, you have normaly verry experianced
> Users there, like power users from Companies. They provide realy valua
> feedback. So from my point of view, this is one of the moast important
> changes we have to do. For all Feature release a beta version. And don't
> forget, people are realy happy to do beta tests. but many of them are
> maybe not willing to follow a mailing list.
A public beta release is of course not the golden solution but could
activate some power users that give us the feedback we want and need.
So, +1 for going this way.
After the 4.1 release is done we can see if this was much better - and
ask ourself why? :-)
Marcus
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by "Marcus (OOo)" <ma...@wtnet.de>.
Am 08/14/2013 09:01 PM, schrieb Raphael Bircher:
> Am 14.08.13 20:21, schrieb Rob Weir:
>> On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp <el...@mail-page.com> wrote:
>>> Dear Rob
>>> The 4.0 release was too ambitious - we should advance in smaller steps.
>>> Nothing compares to general public testing - betas and release
>>> candidates should not be avoided.
>>> TestLink cases should be less comprehesive (in terms of feature
>>> coverage) and more stress testing oriented.
>> The number to consider here is how many defects were found and fixed
>> during the 4.0.0 testing, before the general public users had access?
>> I assume it was quite substantial. If so, the TestLink usage was
>> effective. In other words, we might have found fewer bugs without
>> using it.
In general, my feeling is that it's too early to do a retrospective and
ask "what was good, what needs to be improved?" (to say it with SCRUM
words ;-) ) Just after the first major release.
We should look on the BZ query you have mentioned and see if there is
one or more hotspots that should be improved fast. That's it.
>> This is important to keep in mind: we want to prevent or find more
>> bugs, but we're not starting from zero. We're starting from a process
>> that does a lot of things right.
>>
>> I like the idea of a public beta. But consider the numbers. The 40
>> or so regressions that were reported came from an install base (based
>> on download figures since 4.0.0 was released) of around 3 million
>> users. Realistically, can we expect anywhere near that number in a
>> public beta? Or is it more likely that a beta program has 10,000
>> users or fewer? I don't know the answer here. But certainly a
>> well-publicized and used beta will find more than a beta used by just
>> a few hundred users.
> The public beta is from my point of view realy important. Even you have
> only 10'000 Downloads of a beta, you have normaly verry experianced
> Users there, like power users from Companies. They provide realy valua
> feedback. So from my point of view, this is one of the moast important
> changes we have to do. For all Feature release a beta version. And don't
> forget, people are realy happy to do beta tests. but many of them are
> maybe not willing to follow a mailing list.
A public beta release is of course not the golden solution but could
activate some power users that give us the feedback we want and need.
So, +1 for going this way.
After the 4.1 release is done we can see if this was much better - and
ask ourself why? :-)
Marcus
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Raphael Bircher <r....@gmx.ch>.
Am 14.08.13 20:21, schrieb Rob Weir:
> On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp <el...@mail-page.com> wrote:
>> Dear Rob
>> The 4.0 release was too ambitious - we should advance in smaller steps.
>> Nothing compares to general public testing - betas and release candidates should not be avoided.
>> TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented.
> The number to consider here is how many defects were found and fixed
> during the 4.0.0 testing, before the general public users had access?
> I assume it was quite substantial. If so, the TestLink usage was
> effective. In other words, we might have found fewer bugs without
> using it.
>
> This is important to keep in mind: we want to prevent or find more
> bugs, but we're not starting from zero. We're starting from a process
> that does a lot of things right.
>
> I like the idea of a public beta. But consider the numbers. The 40
> or so regressions that were reported came from an install base (based
> on download figures since 4.0.0 was released) of around 3 million
> users. Realistically, can we expect anywhere near that number in a
> public beta? Or is it more likely that a beta program has 10,000
> users or fewer? I don't know the answer here. But certainly a
> well-publicized and used beta will find more than a beta used by just
> a few hundred users.
The public beta is from my point of view realy important. Even you have
only 10'000 Downloads of a beta, you have normaly verry experianced
Users there, like power users from Companies. They provide realy valua
feedback. So from my point of view, this is one of the moast important
changes we have to do. For all Feature release a beta version. And don't
forget, people are realy happy to do beta tests. but many of them are
maybe not willing to follow a mailing list.
Greetings Raphael
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Raphael Bircher <r....@gmx.ch>.
Am 14.08.13 20:21, schrieb Rob Weir:
> On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp <el...@mail-page.com> wrote:
>> Dear Rob
>> The 4.0 release was too ambitious - we should advance in smaller steps.
>> Nothing compares to general public testing - betas and release candidates should not be avoided.
>> TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented.
> The number to consider here is how many defects were found and fixed
> during the 4.0.0 testing, before the general public users had access?
> I assume it was quite substantial. If so, the TestLink usage was
> effective. In other words, we might have found fewer bugs without
> using it.
>
> This is important to keep in mind: we want to prevent or find more
> bugs, but we're not starting from zero. We're starting from a process
> that does a lot of things right.
>
> I like the idea of a public beta. But consider the numbers. The 40
> or so regressions that were reported came from an install base (based
> on download figures since 4.0.0 was released) of around 3 million
> users. Realistically, can we expect anywhere near that number in a
> public beta? Or is it more likely that a beta program has 10,000
> users or fewer? I don't know the answer here. But certainly a
> well-publicized and used beta will find more than a beta used by just
> a few hundred users.
The public beta is from my point of view realy important. Even you have
only 10'000 Downloads of a beta, you have normaly verry experianced
Users there, like power users from Companies. They provide realy valua
feedback. So from my point of view, this is one of the moast important
changes we have to do. For all Feature release a beta version. And don't
forget, people are realy happy to do beta tests. but many of them are
maybe not willing to follow a mailing list.
Greetings Raphael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp <el...@mail-page.com> wrote:
> Dear Rob
> The 4.0 release was too ambitious - we should advance in smaller steps.
> Nothing compares to general public testing - betas and release candidates should not be avoided.
> TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented.
The number to consider here is how many defects were found and fixed
during the 4.0.0 testing, before the general public users had access?
I assume it was quite substantial. If so, the TestLink usage was
effective. In other words, we might have found fewer bugs without
using it.
This is important to keep in mind: we want to prevent or find more
bugs, but we're not starting from zero. We're starting from a process
that does a lot of things right.
I like the idea of a public beta. But consider the numbers. The 40
or so regressions that were reported came from an install base (based
on download figures since 4.0.0 was released) of around 3 million
users. Realistically, can we expect anywhere near that number in a
public beta? Or is it more likely that a beta program has 10,000
users or fewer? I don't know the answer here. But certainly a
well-publicized and used beta will find more than a beta used by just
a few hundred users.
Regards,
-Rob
> Regards,
> Edwin
>
> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
>> that we're doing this, and their are no arguments against it, shows
>> that we value quality. I'd like to take this a step further, and see
>> what we can learn from the defects in AOO 4.0.0 and what we can do
>> going forward to improve.
>>
>> Quality, in the end, is a process, not a state of grace. We improve
>> by working smarter, not working harder. The goal should be to learn
>> and improve, as individuals and as a community.
>>
>> Every regression that made it into 4.0.0 was added there by a
>> programmer. And the defect went undetected by testers. This is not
>> to blame. It just means that we're all human. We know that. We all
>> make mistakes. I make mistakes. A quality process is not about
>> becoming perfect, but about acknowledging that we make mistakes and
>> that certain formal and informal practices are needed to prevent and
>> detect these mistakes.
>>
>> But enough about generalities. I'm hoping you'll join with me in
>> examining the 32 confirmed 4.0.0 regression defects and answering a
>> few questions:
>>
>> 1) What caused the bug? What was the "root cause"? Note:
>> "programmer error" is not really a cause. We should ask what caused
>> the error.
>>
>> 2) What can we do to prevent bugs like this from being checked in?
>>
>> 3) Why wasn't the bug found during testing? Was it not covered by any
>> existing test case? Was a test case run but the defect was not
>> recognized? Was the defect introduced into the software after the
>> tests had already been executed?
>>
>> 4) What can we do to ensure that bugs like this are caught during testing?
>>
>> So 2 basic questions -- what went wrong and how can we prevent it in
>> the future, looked at from perspective of programmers and testers. If
>> we can keep these questions in mind, and try to answer them, we may be
>> able to find some patterns that can lead to some process changes for
>> AOO 4.1.
>>
>> You can find the 4.0.0 regressions in Bugzilla here:
>>
>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>
>>
>> Regards,
>>
>> -Rob
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: qa-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Fri, Aug 16, 2013 at 6:55 AM, Andrea Pescetti <pe...@apache.org> wrote:
> On 15/08/2013 Rob Weir wrote:
>>
>> A thought experiment: If we ran the existing test automation on
>> 4.0.0, how many of the bugs that we're fixing in 4.0.1 do you think
>> would be detected?
>
>
> I would expect this 4.0.1 blocker issue (wrong results in certain Calc
> functions) to be detectable by automated testing:
> https://issues.apache.org/ooo/show_bug.cgi?id=122997
>
I don't believe this is actually covered in our test automation. So
maybe the actual root causes is that there were no test cases for this
function. If there were test cases then the defect could have been
detected by manual or automated testing. I agree with you that
spreadsheets functions lend themselves to automated testing. But the
problem here was not lack of automation. It was lack of a test case,
period.
-Rob
> Regards,
> Andrea.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Andrea Pescetti <pe...@apache.org>.
On 15/08/2013 Rob Weir wrote:
> A thought experiment: If we ran the existing test automation on
> 4.0.0, how many of the bugs that we're fixing in 4.0.1 do you think
> would be detected?
I would expect this 4.0.1 blocker issue (wrong results in certain Calc
functions) to be detectable by automated testing:
https://issues.apache.org/ooo/show_bug.cgi?id=122997
Regards,
Andrea.
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Thu, Aug 15, 2013 at 8:29 AM, Jürgen Schmidt <jo...@gmail.com> wrote:
> On 8/15/13 1:33 PM, Rob Weir wrote:
>> On Thu, Aug 15, 2013 at 6:58 AM, Jürgen Schmidt <jo...@gmail.com> wrote:
>>> On 8/15/13 12:19 PM, janI wrote:
>>>> On Aug 15, 2013 11:14 AM, "Jürgen Schmidt" <jo...@gmail.com> wrote:
>>>>>
>>>>> On 8/14/13 8:30 PM, Rob Weir wrote:
>>>>>> On Wed, Aug 14, 2013 at 1:55 PM, janI <ja...@apache.org> wrote:
>>>>>>> On 14 August 2013 19:36, Edwin Sharp <el...@mail-page.com> wrote:
>>>>>>>
>>>>>>>> Dear Rob
>>>>>>>> The 4.0 release was too ambitious - we should advance in smaller
>>>> steps.
>>>>>>>> Nothing compares to general public testing - betas and release
>>>> candidates
>>>>>>>> should not be avoided.
>>>>>>>> TestLink cases should be less comprehesive (in terms of feature
>>>> coverage)
>>>>>>>> and more stress testing oriented.
>>>>>>>> Regards,
>>>>>>>> Edwin
>>>>>>>>
>>>>>>>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>>>>>>>>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The
>>>> fact
>>>>>>>>> that we're doing this, and their are no arguments against it, shows
>>>>>>>>> that we value quality. I'd like to take this a step further, and
>>>> see
>>>>>>>>> what we can learn from the defects in AOO 4.0.0 and what we can do
>>>>>>>>> going forward to improve.
>>>>>>>>>
>>>>>>>>> Quality, in the end, is a process, not a state of grace. We improve
>>>>>>>>> by working smarter, not working harder. The goal should be to learn
>>>>>>>>> and improve, as individuals and as a community.
>>>>>>>>>
>>>>>>>>> Every regression that made it into 4.0.0 was added there by a
>>>>>>>>> programmer. And the defect went undetected by testers. This is not
>>>>>>>>> to blame. It just means that we're all human. We know that. We all
>>>>>>>>> make mistakes. I make mistakes. A quality process is not about
>>>>>>>>> becoming perfect, but about acknowledging that we make mistakes and
>>>>>>>>> that certain formal and informal practices are needed to prevent and
>>>>>>>>> detect these mistakes.
>>>>>>>>>
>>>>>>>>> But enough about generalities. I'm hoping you'll join with me in
>>>>>>>>> examining the 32 confirmed 4.0.0 regression defects and answering a
>>>>>>>>> few questions:
>>>>>>>>>
>>>>>>>>> 1) What caused the bug? What was the "root cause"? Note:
>>>>>>>>> "programmer error" is not really a cause. We should ask what caused
>>>>>>>>> the error.
>>>>>>>>>
>>>>>>>>> 2) What can we do to prevent bugs like this from being checked in?
>>>>>>>>>
>>>>>>>>> 3) Why wasn't the bug found during testing? Was it not covered by
>>>> any
>>>>>>>>> existing test case? Was a test case run but the defect was not
>>>>>>>>> recognized? Was the defect introduced into the software after the
>>>>>>>>> tests had already been executed?
>>>>>>>>>
>>>>>>>>> 4) What can we do to ensure that bugs like this are caught during
>>>>>>>> testing?
>>>>>>>>>
>>>>>>>>> So 2 basic questions -- what went wrong and how can we prevent it in
>>>>>>>>> the future, looked at from perspective of programmers and testers.
>>>> If
>>>>>>>>> we can keep these questions in mind, and try to answer them, we may
>>>> be
>>>>>>>>> able to find some patterns that can lead to some process changes for
>>>>>>>>> AOO 4.1.
>>>>>>>>>
>>>>>>>>> You can find the 4.0.0 regressions in Bugzilla here:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> -Rob
>>>>>>>>
>>>>>>>
>>>>>>> I strongly believe that one of the things that went wrong is our
>>>> limited
>>>>>>> possibility to retest (due to resources), when I look at our current
>>>> manual
>>>>>>
>>>>>> I wonder about that as well. That's one reason it would be good to
>>>>>> know how many of the confirmed regressions were introduced late in the
>>>>>> release process, and thus missed coverage in our full test pass.
>>>>>>
>>>>>>> testcases, a lot of those could be automated, e.g. with a simple UI
>>>> macro,
>>>>>>> that would enable us to run these test cases with every build. It may
>>>> sound
>>>>>>> like a dream but where I come from, we did that every night, and it
>>>> caught
>>>>>>> a lot of regression bugs and sideeffects.
>>>>>>>
>>>>>>
>>>>>> This begs the question: Is the functionality of the regressions
>>>>>> covered by our test cases? Or are they covered but we didn't execute
>>>>>> them? Or we executed them but didn't recognize the defect? I don't
>>>>>> know (yet).
>>>>>>
>>>>>>> A simple start, if to request that every bug fix, is issued with at
>>>> least
>>>>>>> one test case (automated or manual).
>>>>>>>
>>>>>>
>>>>>> Often there is, though this information lives in Bugzilla. One thing
>>>>>> we did on another (non open source) project is to mark defects in our
>>>>>> bugtracking system that should become test cases. Not every bug did
>>>>>> that. For example, a defect report to update a mispelling in the UI
>>>>>> would not lead to a new test case. But many would.
>>>>>
>>>>> we have the automated test framework that needs some more attention and
>>>>> polishing. And of course the tests have to improved to get satisfying
>>>>> result.
>>>>>
>>>>> We have
>>>>>
>>>>> BVT - build verification test
>>>>> FVT - functional verification test
>>>>> PVT - performance verification test
>>>>> SVT - system verification test
>>>>>
>>>>> But I have to confess that I have limited knowledge about it yet
>>>>
>>>> I aware that we ha a limited automated framework, at least thats what I
>>>> found and played with.
>>>>
>>>> but, it is not integrated into our build, or our buildbot. Especially
>>>> testing in buildbot gives better qa. An manually controlled automated test
>>>> is not really an ideal solution.
>>>
>>> +1 and I think it was the intended idea behind this, have it run on a
>>> regular basis ideally on the build bots. The work is not finished and
>>> have to be done as so many open work items.
>>> If thet run more stable and the results are in good shape we can
>>> rpobably quite easy includ ethem in our build bots.
>>> I know that hdu has some experience with this and can share some infos
>>>
>>
>> A thought experiment: If we ran the existing test automation on
>> 4.0.0, how many of the bugs that we're fixing in 4.0.1 do you think
>> would be detected?
>
> I don't think that we would have detected these problems.
>
> The performance issue could have been detected if the correct reference
> data would have been used. But the related changes are in the code for
> ~1 year. I don't know if we have a related test. But for sure a
> candidate that could have been found in time if we would have a test and
> good reference data.
>
I think it is reasonable to expect that a performance verification
test would include a test saving a large XLS file. Of course, this
only works if we run the test automation.
> All related problems with external extension are not detected.
>
> Some of these problems are only visible when you see the screen.,
> difficult to find.
>
> The copy/paste problem with Punjabi? Not easy to detect.
>
Right.
GUI-based automation tends to be broad but shallow. It requires
special skills (Java programming in our case) to develop and maintain.
If run regularly, like with every build, it can catch catastrophic
errors almost immediately. Automation has the benefit of costing
nothing to run (once automated) and not making mistakes. The downside
is it does not find bugs that it is not programmed to detect.
The old saying applies here: "Every class of users finds a new class
of bugs". There is no one perfect way of testing that finds all bugs.
The best approach is a mix of techniques.
One thing I wonder about: Do we use test assertions in our code,
like the old C/C++ assert() macro or similar? The blocking issue we
found in RC1 would likely have been found that way, for example. The
nice thing about test assertions is they can even help developers find
the bug before code is checked in. And unlike test automation a test
assertion does not fall out of synch with the code or the UI, since it
is in the code. It is more likely to be maintained. Also, adding a
test assertion after a bug fix is easier than writing a GUI test case.
Regards,
-Rob
> Juergen
>
>>
>> -Rob
>>
>>>
>>>>
>>>> Take a look at the manual test cases, a lot could be automated, and free qa
>>>> resources for more complex testing.
>>>
>>> sure but again the work have to be done and volunteers are welcome as
>>> always. You have probably enough to do, the same for me or others ...
>>>
>>> Juergen
>>>
>>>>
>>>> rgds
>>>> jan I
>>>>>
>>>>> Juergen
>>>>>
>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> -Rob
>>>>>>
>>>>>>> rgds
>>>>>>> jan I.
>>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>>>>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>>>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: qa-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Thu, Aug 15, 2013 at 8:29 AM, Jürgen Schmidt <jo...@gmail.com> wrote:
> On 8/15/13 1:33 PM, Rob Weir wrote:
>> On Thu, Aug 15, 2013 at 6:58 AM, Jürgen Schmidt <jo...@gmail.com> wrote:
>>> On 8/15/13 12:19 PM, janI wrote:
>>>> On Aug 15, 2013 11:14 AM, "Jürgen Schmidt" <jo...@gmail.com> wrote:
>>>>>
>>>>> On 8/14/13 8:30 PM, Rob Weir wrote:
>>>>>> On Wed, Aug 14, 2013 at 1:55 PM, janI <ja...@apache.org> wrote:
>>>>>>> On 14 August 2013 19:36, Edwin Sharp <el...@mail-page.com> wrote:
>>>>>>>
>>>>>>>> Dear Rob
>>>>>>>> The 4.0 release was too ambitious - we should advance in smaller
>>>> steps.
>>>>>>>> Nothing compares to general public testing - betas and release
>>>> candidates
>>>>>>>> should not be avoided.
>>>>>>>> TestLink cases should be less comprehesive (in terms of feature
>>>> coverage)
>>>>>>>> and more stress testing oriented.
>>>>>>>> Regards,
>>>>>>>> Edwin
>>>>>>>>
>>>>>>>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>>>>>>>>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The
>>>> fact
>>>>>>>>> that we're doing this, and their are no arguments against it, shows
>>>>>>>>> that we value quality. I'd like to take this a step further, and
>>>> see
>>>>>>>>> what we can learn from the defects in AOO 4.0.0 and what we can do
>>>>>>>>> going forward to improve.
>>>>>>>>>
>>>>>>>>> Quality, in the end, is a process, not a state of grace. We improve
>>>>>>>>> by working smarter, not working harder. The goal should be to learn
>>>>>>>>> and improve, as individuals and as a community.
>>>>>>>>>
>>>>>>>>> Every regression that made it into 4.0.0 was added there by a
>>>>>>>>> programmer. And the defect went undetected by testers. This is not
>>>>>>>>> to blame. It just means that we're all human. We know that. We all
>>>>>>>>> make mistakes. I make mistakes. A quality process is not about
>>>>>>>>> becoming perfect, but about acknowledging that we make mistakes and
>>>>>>>>> that certain formal and informal practices are needed to prevent and
>>>>>>>>> detect these mistakes.
>>>>>>>>>
>>>>>>>>> But enough about generalities. I'm hoping you'll join with me in
>>>>>>>>> examining the 32 confirmed 4.0.0 regression defects and answering a
>>>>>>>>> few questions:
>>>>>>>>>
>>>>>>>>> 1) What caused the bug? What was the "root cause"? Note:
>>>>>>>>> "programmer error" is not really a cause. We should ask what caused
>>>>>>>>> the error.
>>>>>>>>>
>>>>>>>>> 2) What can we do to prevent bugs like this from being checked in?
>>>>>>>>>
>>>>>>>>> 3) Why wasn't the bug found during testing? Was it not covered by
>>>> any
>>>>>>>>> existing test case? Was a test case run but the defect was not
>>>>>>>>> recognized? Was the defect introduced into the software after the
>>>>>>>>> tests had already been executed?
>>>>>>>>>
>>>>>>>>> 4) What can we do to ensure that bugs like this are caught during
>>>>>>>> testing?
>>>>>>>>>
>>>>>>>>> So 2 basic questions -- what went wrong and how can we prevent it in
>>>>>>>>> the future, looked at from perspective of programmers and testers.
>>>> If
>>>>>>>>> we can keep these questions in mind, and try to answer them, we may
>>>> be
>>>>>>>>> able to find some patterns that can lead to some process changes for
>>>>>>>>> AOO 4.1.
>>>>>>>>>
>>>>>>>>> You can find the 4.0.0 regressions in Bugzilla here:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> -Rob
>>>>>>>>
>>>>>>>
>>>>>>> I strongly believe that one of the things that went wrong is our
>>>> limited
>>>>>>> possibility to retest (due to resources), when I look at our current
>>>> manual
>>>>>>
>>>>>> I wonder about that as well. That's one reason it would be good to
>>>>>> know how many of the confirmed regressions were introduced late in the
>>>>>> release process, and thus missed coverage in our full test pass.
>>>>>>
>>>>>>> testcases, a lot of those could be automated, e.g. with a simple UI
>>>> macro,
>>>>>>> that would enable us to run these test cases with every build. It may
>>>> sound
>>>>>>> like a dream but where I come from, we did that every night, and it
>>>> caught
>>>>>>> a lot of regression bugs and sideeffects.
>>>>>>>
>>>>>>
>>>>>> This begs the question: Is the functionality of the regressions
>>>>>> covered by our test cases? Or are they covered but we didn't execute
>>>>>> them? Or we executed them but didn't recognize the defect? I don't
>>>>>> know (yet).
>>>>>>
>>>>>>> A simple start, if to request that every bug fix, is issued with at
>>>> least
>>>>>>> one test case (automated or manual).
>>>>>>>
>>>>>>
>>>>>> Often there is, though this information lives in Bugzilla. One thing
>>>>>> we did on another (non open source) project is to mark defects in our
>>>>>> bugtracking system that should become test cases. Not every bug did
>>>>>> that. For example, a defect report to update a mispelling in the UI
>>>>>> would not lead to a new test case. But many would.
>>>>>
>>>>> we have the automated test framework that needs some more attention and
>>>>> polishing. And of course the tests have to improved to get satisfying
>>>>> result.
>>>>>
>>>>> We have
>>>>>
>>>>> BVT - build verification test
>>>>> FVT - functional verification test
>>>>> PVT - performance verification test
>>>>> SVT - system verification test
>>>>>
>>>>> But I have to confess that I have limited knowledge about it yet
>>>>
>>>> I aware that we ha a limited automated framework, at least thats what I
>>>> found and played with.
>>>>
>>>> but, it is not integrated into our build, or our buildbot. Especially
>>>> testing in buildbot gives better qa. An manually controlled automated test
>>>> is not really an ideal solution.
>>>
>>> +1 and I think it was the intended idea behind this, have it run on a
>>> regular basis ideally on the build bots. The work is not finished and
>>> have to be done as so many open work items.
>>> If thet run more stable and the results are in good shape we can
>>> rpobably quite easy includ ethem in our build bots.
>>> I know that hdu has some experience with this and can share some infos
>>>
>>
>> A thought experiment: If we ran the existing test automation on
>> 4.0.0, how many of the bugs that we're fixing in 4.0.1 do you think
>> would be detected?
>
> I don't think that we would have detected these problems.
>
> The performance issue could have been detected if the correct reference
> data would have been used. But the related changes are in the code for
> ~1 year. I don't know if we have a related test. But for sure a
> candidate that could have been found in time if we would have a test and
> good reference data.
>
I think it is reasonable to expect that a performance verification
test would include a test saving a large XLS file. Of course, this
only works if we run the test automation.
> All related problems with external extension are not detected.
>
> Some of these problems are only visible when you see the screen.,
> difficult to find.
>
> The copy/paste problem with Punjabi? Not easy to detect.
>
Right.
GUI-based automation tends to be broad but shallow. It requires
special skills (Java programming in our case) to develop and maintain.
If run regularly, like with every build, it can catch catastrophic
errors almost immediately. Automation has the benefit of costing
nothing to run (once automated) and not making mistakes. The downside
is it does not find bugs that it is not programmed to detect.
The old saying applies here: "Every class of users finds a new class
of bugs". There is no one perfect way of testing that finds all bugs.
The best approach is a mix of techniques.
One thing I wonder about: Do we use test assertions in our code,
like the old C/C++ assert() macro or similar? The blocking issue we
found in RC1 would likely have been found that way, for example. The
nice thing about test assertions is they can even help developers find
the bug before code is checked in. And unlike test automation a test
assertion does not fall out of synch with the code or the UI, since it
is in the code. It is more likely to be maintained. Also, adding a
test assertion after a bug fix is easier than writing a GUI test case.
Regards,
-Rob
> Juergen
>
>>
>> -Rob
>>
>>>
>>>>
>>>> Take a look at the manual test cases, a lot could be automated, and free qa
>>>> resources for more complex testing.
>>>
>>> sure but again the work have to be done and volunteers are welcome as
>>> always. You have probably enough to do, the same for me or others ...
>>>
>>> Juergen
>>>
>>>>
>>>> rgds
>>>> jan I
>>>>>
>>>>> Juergen
>>>>>
>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> -Rob
>>>>>>
>>>>>>> rgds
>>>>>>> jan I.
>>>>>>>
>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>>>>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>>>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>>>
>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>>
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: qa-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Jürgen Schmidt <jo...@gmail.com>.
On 8/15/13 1:33 PM, Rob Weir wrote:
> On Thu, Aug 15, 2013 at 6:58 AM, Jürgen Schmidt <jo...@gmail.com> wrote:
>> On 8/15/13 12:19 PM, janI wrote:
>>> On Aug 15, 2013 11:14 AM, "Jürgen Schmidt" <jo...@gmail.com> wrote:
>>>>
>>>> On 8/14/13 8:30 PM, Rob Weir wrote:
>>>>> On Wed, Aug 14, 2013 at 1:55 PM, janI <ja...@apache.org> wrote:
>>>>>> On 14 August 2013 19:36, Edwin Sharp <el...@mail-page.com> wrote:
>>>>>>
>>>>>>> Dear Rob
>>>>>>> The 4.0 release was too ambitious - we should advance in smaller
>>> steps.
>>>>>>> Nothing compares to general public testing - betas and release
>>> candidates
>>>>>>> should not be avoided.
>>>>>>> TestLink cases should be less comprehesive (in terms of feature
>>> coverage)
>>>>>>> and more stress testing oriented.
>>>>>>> Regards,
>>>>>>> Edwin
>>>>>>>
>>>>>>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>>>>>>>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The
>>> fact
>>>>>>>> that we're doing this, and their are no arguments against it, shows
>>>>>>>> that we value quality. I'd like to take this a step further, and
>>> see
>>>>>>>> what we can learn from the defects in AOO 4.0.0 and what we can do
>>>>>>>> going forward to improve.
>>>>>>>>
>>>>>>>> Quality, in the end, is a process, not a state of grace. We improve
>>>>>>>> by working smarter, not working harder. The goal should be to learn
>>>>>>>> and improve, as individuals and as a community.
>>>>>>>>
>>>>>>>> Every regression that made it into 4.0.0 was added there by a
>>>>>>>> programmer. And the defect went undetected by testers. This is not
>>>>>>>> to blame. It just means that we're all human. We know that. We all
>>>>>>>> make mistakes. I make mistakes. A quality process is not about
>>>>>>>> becoming perfect, but about acknowledging that we make mistakes and
>>>>>>>> that certain formal and informal practices are needed to prevent and
>>>>>>>> detect these mistakes.
>>>>>>>>
>>>>>>>> But enough about generalities. I'm hoping you'll join with me in
>>>>>>>> examining the 32 confirmed 4.0.0 regression defects and answering a
>>>>>>>> few questions:
>>>>>>>>
>>>>>>>> 1) What caused the bug? What was the "root cause"? Note:
>>>>>>>> "programmer error" is not really a cause. We should ask what caused
>>>>>>>> the error.
>>>>>>>>
>>>>>>>> 2) What can we do to prevent bugs like this from being checked in?
>>>>>>>>
>>>>>>>> 3) Why wasn't the bug found during testing? Was it not covered by
>>> any
>>>>>>>> existing test case? Was a test case run but the defect was not
>>>>>>>> recognized? Was the defect introduced into the software after the
>>>>>>>> tests had already been executed?
>>>>>>>>
>>>>>>>> 4) What can we do to ensure that bugs like this are caught during
>>>>>>> testing?
>>>>>>>>
>>>>>>>> So 2 basic questions -- what went wrong and how can we prevent it in
>>>>>>>> the future, looked at from perspective of programmers and testers.
>>> If
>>>>>>>> we can keep these questions in mind, and try to answer them, we may
>>> be
>>>>>>>> able to find some patterns that can lead to some process changes for
>>>>>>>> AOO 4.1.
>>>>>>>>
>>>>>>>> You can find the 4.0.0 regressions in Bugzilla here:
>>>>>>>>
>>>>>>>>
>>>>>>>
>>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> -Rob
>>>>>>>
>>>>>>
>>>>>> I strongly believe that one of the things that went wrong is our
>>> limited
>>>>>> possibility to retest (due to resources), when I look at our current
>>> manual
>>>>>
>>>>> I wonder about that as well. That's one reason it would be good to
>>>>> know how many of the confirmed regressions were introduced late in the
>>>>> release process, and thus missed coverage in our full test pass.
>>>>>
>>>>>> testcases, a lot of those could be automated, e.g. with a simple UI
>>> macro,
>>>>>> that would enable us to run these test cases with every build. It may
>>> sound
>>>>>> like a dream but where I come from, we did that every night, and it
>>> caught
>>>>>> a lot of regression bugs and sideeffects.
>>>>>>
>>>>>
>>>>> This begs the question: Is the functionality of the regressions
>>>>> covered by our test cases? Or are they covered but we didn't execute
>>>>> them? Or we executed them but didn't recognize the defect? I don't
>>>>> know (yet).
>>>>>
>>>>>> A simple start, if to request that every bug fix, is issued with at
>>> least
>>>>>> one test case (automated or manual).
>>>>>>
>>>>>
>>>>> Often there is, though this information lives in Bugzilla. One thing
>>>>> we did on another (non open source) project is to mark defects in our
>>>>> bugtracking system that should become test cases. Not every bug did
>>>>> that. For example, a defect report to update a mispelling in the UI
>>>>> would not lead to a new test case. But many would.
>>>>
>>>> we have the automated test framework that needs some more attention and
>>>> polishing. And of course the tests have to improved to get satisfying
>>>> result.
>>>>
>>>> We have
>>>>
>>>> BVT - build verification test
>>>> FVT - functional verification test
>>>> PVT - performance verification test
>>>> SVT - system verification test
>>>>
>>>> But I have to confess that I have limited knowledge about it yet
>>>
>>> I aware that we ha a limited automated framework, at least thats what I
>>> found and played with.
>>>
>>> but, it is not integrated into our build, or our buildbot. Especially
>>> testing in buildbot gives better qa. An manually controlled automated test
>>> is not really an ideal solution.
>>
>> +1 and I think it was the intended idea behind this, have it run on a
>> regular basis ideally on the build bots. The work is not finished and
>> have to be done as so many open work items.
>> If thet run more stable and the results are in good shape we can
>> rpobably quite easy includ ethem in our build bots.
>> I know that hdu has some experience with this and can share some infos
>>
>
> A thought experiment: If we ran the existing test automation on
> 4.0.0, how many of the bugs that we're fixing in 4.0.1 do you think
> would be detected?
I don't think that we would have detected these problems.
The performance issue could have been detected if the correct reference
data would have been used. But the related changes are in the code for
~1 year. I don't know if we have a related test. But for sure a
candidate that could have been found in time if we would have a test and
good reference data.
All related problems with external extension are not detected.
Some of these problems are only visible when you see the screen.,
difficult to find.
The copy/paste problem with Punjabi? Not easy to detect.
Juergen
>
> -Rob
>
>>
>>>
>>> Take a look at the manual test cases, a lot could be automated, and free qa
>>> resources for more complex testing.
>>
>> sure but again the work have to be done and volunteers are welcome as
>> always. You have probably enough to do, the same for me or others ...
>>
>> Juergen
>>
>>>
>>> rgds
>>> jan I
>>>>
>>>> Juergen
>>>>
>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> -Rob
>>>>>
>>>>>> rgds
>>>>>> jan I.
>>>>>>
>>>>>>
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>>>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>>>>
>>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: qa-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Andrea Pescetti <pe...@apache.org>.
On 15/08/2013 Rob Weir wrote:
> A thought experiment: If we ran the existing test automation on
> 4.0.0, how many of the bugs that we're fixing in 4.0.1 do you think
> would be detected?
I would expect this 4.0.1 blocker issue (wrong results in certain Calc
functions) to be detectable by automated testing:
https://issues.apache.org/ooo/show_bug.cgi?id=122997
Regards,
Andrea.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Thu, Aug 15, 2013 at 6:58 AM, Jürgen Schmidt <jo...@gmail.com> wrote:
> On 8/15/13 12:19 PM, janI wrote:
>> On Aug 15, 2013 11:14 AM, "Jürgen Schmidt" <jo...@gmail.com> wrote:
>>>
>>> On 8/14/13 8:30 PM, Rob Weir wrote:
>>>> On Wed, Aug 14, 2013 at 1:55 PM, janI <ja...@apache.org> wrote:
>>>>> On 14 August 2013 19:36, Edwin Sharp <el...@mail-page.com> wrote:
>>>>>
>>>>>> Dear Rob
>>>>>> The 4.0 release was too ambitious - we should advance in smaller
>> steps.
>>>>>> Nothing compares to general public testing - betas and release
>> candidates
>>>>>> should not be avoided.
>>>>>> TestLink cases should be less comprehesive (in terms of feature
>> coverage)
>>>>>> and more stress testing oriented.
>>>>>> Regards,
>>>>>> Edwin
>>>>>>
>>>>>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>>>>>>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The
>> fact
>>>>>>> that we're doing this, and their are no arguments against it, shows
>>>>>>> that we value quality. I'd like to take this a step further, and
>> see
>>>>>>> what we can learn from the defects in AOO 4.0.0 and what we can do
>>>>>>> going forward to improve.
>>>>>>>
>>>>>>> Quality, in the end, is a process, not a state of grace. We improve
>>>>>>> by working smarter, not working harder. The goal should be to learn
>>>>>>> and improve, as individuals and as a community.
>>>>>>>
>>>>>>> Every regression that made it into 4.0.0 was added there by a
>>>>>>> programmer. And the defect went undetected by testers. This is not
>>>>>>> to blame. It just means that we're all human. We know that. We all
>>>>>>> make mistakes. I make mistakes. A quality process is not about
>>>>>>> becoming perfect, but about acknowledging that we make mistakes and
>>>>>>> that certain formal and informal practices are needed to prevent and
>>>>>>> detect these mistakes.
>>>>>>>
>>>>>>> But enough about generalities. I'm hoping you'll join with me in
>>>>>>> examining the 32 confirmed 4.0.0 regression defects and answering a
>>>>>>> few questions:
>>>>>>>
>>>>>>> 1) What caused the bug? What was the "root cause"? Note:
>>>>>>> "programmer error" is not really a cause. We should ask what caused
>>>>>>> the error.
>>>>>>>
>>>>>>> 2) What can we do to prevent bugs like this from being checked in?
>>>>>>>
>>>>>>> 3) Why wasn't the bug found during testing? Was it not covered by
>> any
>>>>>>> existing test case? Was a test case run but the defect was not
>>>>>>> recognized? Was the defect introduced into the software after the
>>>>>>> tests had already been executed?
>>>>>>>
>>>>>>> 4) What can we do to ensure that bugs like this are caught during
>>>>>> testing?
>>>>>>>
>>>>>>> So 2 basic questions -- what went wrong and how can we prevent it in
>>>>>>> the future, looked at from perspective of programmers and testers.
>> If
>>>>>>> we can keep these questions in mind, and try to answer them, we may
>> be
>>>>>>> able to find some patterns that can lead to some process changes for
>>>>>>> AOO 4.1.
>>>>>>>
>>>>>>> You can find the 4.0.0 regressions in Bugzilla here:
>>>>>>>
>>>>>>>
>>>>>>
>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> -Rob
>>>>>>
>>>>>
>>>>> I strongly believe that one of the things that went wrong is our
>> limited
>>>>> possibility to retest (due to resources), when I look at our current
>> manual
>>>>
>>>> I wonder about that as well. That's one reason it would be good to
>>>> know how many of the confirmed regressions were introduced late in the
>>>> release process, and thus missed coverage in our full test pass.
>>>>
>>>>> testcases, a lot of those could be automated, e.g. with a simple UI
>> macro,
>>>>> that would enable us to run these test cases with every build. It may
>> sound
>>>>> like a dream but where I come from, we did that every night, and it
>> caught
>>>>> a lot of regression bugs and sideeffects.
>>>>>
>>>>
>>>> This begs the question: Is the functionality of the regressions
>>>> covered by our test cases? Or are they covered but we didn't execute
>>>> them? Or we executed them but didn't recognize the defect? I don't
>>>> know (yet).
>>>>
>>>>> A simple start, if to request that every bug fix, is issued with at
>> least
>>>>> one test case (automated or manual).
>>>>>
>>>>
>>>> Often there is, though this information lives in Bugzilla. One thing
>>>> we did on another (non open source) project is to mark defects in our
>>>> bugtracking system that should become test cases. Not every bug did
>>>> that. For example, a defect report to update a mispelling in the UI
>>>> would not lead to a new test case. But many would.
>>>
>>> we have the automated test framework that needs some more attention and
>>> polishing. And of course the tests have to improved to get satisfying
>>> result.
>>>
>>> We have
>>>
>>> BVT - build verification test
>>> FVT - functional verification test
>>> PVT - performance verification test
>>> SVT - system verification test
>>>
>>> But I have to confess that I have limited knowledge about it yet
>>
>> I aware that we ha a limited automated framework, at least thats what I
>> found and played with.
>>
>> but, it is not integrated into our build, or our buildbot. Especially
>> testing in buildbot gives better qa. An manually controlled automated test
>> is not really an ideal solution.
>
> +1 and I think it was the intended idea behind this, have it run on a
> regular basis ideally on the build bots. The work is not finished and
> have to be done as so many open work items.
> If thet run more stable and the results are in good shape we can
> rpobably quite easy includ ethem in our build bots.
> I know that hdu has some experience with this and can share some infos
>
A thought experiment: If we ran the existing test automation on
4.0.0, how many of the bugs that we're fixing in 4.0.1 do you think
would be detected?
-Rob
>
>>
>> Take a look at the manual test cases, a lot could be automated, and free qa
>> resources for more complex testing.
>
> sure but again the work have to be done and volunteers are welcome as
> always. You have probably enough to do, the same for me or others ...
>
> Juergen
>
>>
>> rgds
>> jan I
>>>
>>> Juergen
>>>
>>>
>>>>
>>>> Regards,
>>>>
>>>> -Rob
>>>>
>>>>> rgds
>>>>> jan I.
>>>>>
>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>>>
>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Thu, Aug 15, 2013 at 6:58 AM, Jürgen Schmidt <jo...@gmail.com> wrote:
> On 8/15/13 12:19 PM, janI wrote:
>> On Aug 15, 2013 11:14 AM, "Jürgen Schmidt" <jo...@gmail.com> wrote:
>>>
>>> On 8/14/13 8:30 PM, Rob Weir wrote:
>>>> On Wed, Aug 14, 2013 at 1:55 PM, janI <ja...@apache.org> wrote:
>>>>> On 14 August 2013 19:36, Edwin Sharp <el...@mail-page.com> wrote:
>>>>>
>>>>>> Dear Rob
>>>>>> The 4.0 release was too ambitious - we should advance in smaller
>> steps.
>>>>>> Nothing compares to general public testing - betas and release
>> candidates
>>>>>> should not be avoided.
>>>>>> TestLink cases should be less comprehesive (in terms of feature
>> coverage)
>>>>>> and more stress testing oriented.
>>>>>> Regards,
>>>>>> Edwin
>>>>>>
>>>>>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>>>>>>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The
>> fact
>>>>>>> that we're doing this, and their are no arguments against it, shows
>>>>>>> that we value quality. I'd like to take this a step further, and
>> see
>>>>>>> what we can learn from the defects in AOO 4.0.0 and what we can do
>>>>>>> going forward to improve.
>>>>>>>
>>>>>>> Quality, in the end, is a process, not a state of grace. We improve
>>>>>>> by working smarter, not working harder. The goal should be to learn
>>>>>>> and improve, as individuals and as a community.
>>>>>>>
>>>>>>> Every regression that made it into 4.0.0 was added there by a
>>>>>>> programmer. And the defect went undetected by testers. This is not
>>>>>>> to blame. It just means that we're all human. We know that. We all
>>>>>>> make mistakes. I make mistakes. A quality process is not about
>>>>>>> becoming perfect, but about acknowledging that we make mistakes and
>>>>>>> that certain formal and informal practices are needed to prevent and
>>>>>>> detect these mistakes.
>>>>>>>
>>>>>>> But enough about generalities. I'm hoping you'll join with me in
>>>>>>> examining the 32 confirmed 4.0.0 regression defects and answering a
>>>>>>> few questions:
>>>>>>>
>>>>>>> 1) What caused the bug? What was the "root cause"? Note:
>>>>>>> "programmer error" is not really a cause. We should ask what caused
>>>>>>> the error.
>>>>>>>
>>>>>>> 2) What can we do to prevent bugs like this from being checked in?
>>>>>>>
>>>>>>> 3) Why wasn't the bug found during testing? Was it not covered by
>> any
>>>>>>> existing test case? Was a test case run but the defect was not
>>>>>>> recognized? Was the defect introduced into the software after the
>>>>>>> tests had already been executed?
>>>>>>>
>>>>>>> 4) What can we do to ensure that bugs like this are caught during
>>>>>> testing?
>>>>>>>
>>>>>>> So 2 basic questions -- what went wrong and how can we prevent it in
>>>>>>> the future, looked at from perspective of programmers and testers.
>> If
>>>>>>> we can keep these questions in mind, and try to answer them, we may
>> be
>>>>>>> able to find some patterns that can lead to some process changes for
>>>>>>> AOO 4.1.
>>>>>>>
>>>>>>> You can find the 4.0.0 regressions in Bugzilla here:
>>>>>>>
>>>>>>>
>>>>>>
>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>>>>>>
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> -Rob
>>>>>>
>>>>>
>>>>> I strongly believe that one of the things that went wrong is our
>> limited
>>>>> possibility to retest (due to resources), when I look at our current
>> manual
>>>>
>>>> I wonder about that as well. That's one reason it would be good to
>>>> know how many of the confirmed regressions were introduced late in the
>>>> release process, and thus missed coverage in our full test pass.
>>>>
>>>>> testcases, a lot of those could be automated, e.g. with a simple UI
>> macro,
>>>>> that would enable us to run these test cases with every build. It may
>> sound
>>>>> like a dream but where I come from, we did that every night, and it
>> caught
>>>>> a lot of regression bugs and sideeffects.
>>>>>
>>>>
>>>> This begs the question: Is the functionality of the regressions
>>>> covered by our test cases? Or are they covered but we didn't execute
>>>> them? Or we executed them but didn't recognize the defect? I don't
>>>> know (yet).
>>>>
>>>>> A simple start, if to request that every bug fix, is issued with at
>> least
>>>>> one test case (automated or manual).
>>>>>
>>>>
>>>> Often there is, though this information lives in Bugzilla. One thing
>>>> we did on another (non open source) project is to mark defects in our
>>>> bugtracking system that should become test cases. Not every bug did
>>>> that. For example, a defect report to update a mispelling in the UI
>>>> would not lead to a new test case. But many would.
>>>
>>> we have the automated test framework that needs some more attention and
>>> polishing. And of course the tests have to improved to get satisfying
>>> result.
>>>
>>> We have
>>>
>>> BVT - build verification test
>>> FVT - functional verification test
>>> PVT - performance verification test
>>> SVT - system verification test
>>>
>>> But I have to confess that I have limited knowledge about it yet
>>
>> I aware that we ha a limited automated framework, at least thats what I
>> found and played with.
>>
>> but, it is not integrated into our build, or our buildbot. Especially
>> testing in buildbot gives better qa. An manually controlled automated test
>> is not really an ideal solution.
>
> +1 and I think it was the intended idea behind this, have it run on a
> regular basis ideally on the build bots. The work is not finished and
> have to be done as so many open work items.
> If thet run more stable and the results are in good shape we can
> rpobably quite easy includ ethem in our build bots.
> I know that hdu has some experience with this and can share some infos
>
A thought experiment: If we ran the existing test automation on
4.0.0, how many of the bugs that we're fixing in 4.0.1 do you think
would be detected?
-Rob
>
>>
>> Take a look at the manual test cases, a lot could be automated, and free qa
>> resources for more complex testing.
>
> sure but again the work have to be done and volunteers are welcome as
> always. You have probably enough to do, the same for me or others ...
>
> Juergen
>
>>
>> rgds
>> jan I
>>>
>>> Juergen
>>>
>>>
>>>>
>>>> Regards,
>>>>
>>>> -Rob
>>>>
>>>>> rgds
>>>>> jan I.
>>>>>
>>>>>
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>>>
>>>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Jürgen Schmidt <jo...@gmail.com>.
On 8/15/13 12:19 PM, janI wrote:
> On Aug 15, 2013 11:14 AM, "Jürgen Schmidt" <jo...@gmail.com> wrote:
>>
>> On 8/14/13 8:30 PM, Rob Weir wrote:
>>> On Wed, Aug 14, 2013 at 1:55 PM, janI <ja...@apache.org> wrote:
>>>> On 14 August 2013 19:36, Edwin Sharp <el...@mail-page.com> wrote:
>>>>
>>>>> Dear Rob
>>>>> The 4.0 release was too ambitious - we should advance in smaller
> steps.
>>>>> Nothing compares to general public testing - betas and release
> candidates
>>>>> should not be avoided.
>>>>> TestLink cases should be less comprehesive (in terms of feature
> coverage)
>>>>> and more stress testing oriented.
>>>>> Regards,
>>>>> Edwin
>>>>>
>>>>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>>>>>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The
> fact
>>>>>> that we're doing this, and their are no arguments against it, shows
>>>>>> that we value quality. I'd like to take this a step further, and
> see
>>>>>> what we can learn from the defects in AOO 4.0.0 and what we can do
>>>>>> going forward to improve.
>>>>>>
>>>>>> Quality, in the end, is a process, not a state of grace. We improve
>>>>>> by working smarter, not working harder. The goal should be to learn
>>>>>> and improve, as individuals and as a community.
>>>>>>
>>>>>> Every regression that made it into 4.0.0 was added there by a
>>>>>> programmer. And the defect went undetected by testers. This is not
>>>>>> to blame. It just means that we're all human. We know that. We all
>>>>>> make mistakes. I make mistakes. A quality process is not about
>>>>>> becoming perfect, but about acknowledging that we make mistakes and
>>>>>> that certain formal and informal practices are needed to prevent and
>>>>>> detect these mistakes.
>>>>>>
>>>>>> But enough about generalities. I'm hoping you'll join with me in
>>>>>> examining the 32 confirmed 4.0.0 regression defects and answering a
>>>>>> few questions:
>>>>>>
>>>>>> 1) What caused the bug? What was the "root cause"? Note:
>>>>>> "programmer error" is not really a cause. We should ask what caused
>>>>>> the error.
>>>>>>
>>>>>> 2) What can we do to prevent bugs like this from being checked in?
>>>>>>
>>>>>> 3) Why wasn't the bug found during testing? Was it not covered by
> any
>>>>>> existing test case? Was a test case run but the defect was not
>>>>>> recognized? Was the defect introduced into the software after the
>>>>>> tests had already been executed?
>>>>>>
>>>>>> 4) What can we do to ensure that bugs like this are caught during
>>>>> testing?
>>>>>>
>>>>>> So 2 basic questions -- what went wrong and how can we prevent it in
>>>>>> the future, looked at from perspective of programmers and testers.
> If
>>>>>> we can keep these questions in mind, and try to answer them, we may
> be
>>>>>> able to find some patterns that can lead to some process changes for
>>>>>> AOO 4.1.
>>>>>>
>>>>>> You can find the 4.0.0 regressions in Bugzilla here:
>>>>>>
>>>>>>
>>>>>
> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> -Rob
>>>>>
>>>>
>>>> I strongly believe that one of the things that went wrong is our
> limited
>>>> possibility to retest (due to resources), when I look at our current
> manual
>>>
>>> I wonder about that as well. That's one reason it would be good to
>>> know how many of the confirmed regressions were introduced late in the
>>> release process, and thus missed coverage in our full test pass.
>>>
>>>> testcases, a lot of those could be automated, e.g. with a simple UI
> macro,
>>>> that would enable us to run these test cases with every build. It may
> sound
>>>> like a dream but where I come from, we did that every night, and it
> caught
>>>> a lot of regression bugs and sideeffects.
>>>>
>>>
>>> This begs the question: Is the functionality of the regressions
>>> covered by our test cases? Or are they covered but we didn't execute
>>> them? Or we executed them but didn't recognize the defect? I don't
>>> know (yet).
>>>
>>>> A simple start, if to request that every bug fix, is issued with at
> least
>>>> one test case (automated or manual).
>>>>
>>>
>>> Often there is, though this information lives in Bugzilla. One thing
>>> we did on another (non open source) project is to mark defects in our
>>> bugtracking system that should become test cases. Not every bug did
>>> that. For example, a defect report to update a mispelling in the UI
>>> would not lead to a new test case. But many would.
>>
>> we have the automated test framework that needs some more attention and
>> polishing. And of course the tests have to improved to get satisfying
>> result.
>>
>> We have
>>
>> BVT - build verification test
>> FVT - functional verification test
>> PVT - performance verification test
>> SVT - system verification test
>>
>> But I have to confess that I have limited knowledge about it yet
>
> I aware that we ha a limited automated framework, at least thats what I
> found and played with.
>
> but, it is not integrated into our build, or our buildbot. Especially
> testing in buildbot gives better qa. An manually controlled automated test
> is not really an ideal solution.
+1 and I think it was the intended idea behind this, have it run on a
regular basis ideally on the build bots. The work is not finished and
have to be done as so many open work items.
If thet run more stable and the results are in good shape we can
rpobably quite easy includ ethem in our build bots.
I know that hdu has some experience with this and can share some infos
>
> Take a look at the manual test cases, a lot could be automated, and free qa
> resources for more complex testing.
sure but again the work have to be done and volunteers are welcome as
always. You have probably enough to do, the same for me or others ...
Juergen
>
> rgds
> jan I
>>
>> Juergen
>>
>>
>>>
>>> Regards,
>>>
>>> -Rob
>>>
>>>> rgds
>>>> jan I.
>>>>
>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>>
>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by janI <ja...@apache.org>.
On Aug 15, 2013 11:14 AM, "Jürgen Schmidt" <jo...@gmail.com> wrote:
>
> On 8/14/13 8:30 PM, Rob Weir wrote:
> > On Wed, Aug 14, 2013 at 1:55 PM, janI <ja...@apache.org> wrote:
> >> On 14 August 2013 19:36, Edwin Sharp <el...@mail-page.com> wrote:
> >>
> >>> Dear Rob
> >>> The 4.0 release was too ambitious - we should advance in smaller
steps.
> >>> Nothing compares to general public testing - betas and release
candidates
> >>> should not be avoided.
> >>> TestLink cases should be less comprehesive (in terms of feature
coverage)
> >>> and more stress testing oriented.
> >>> Regards,
> >>> Edwin
> >>>
> >>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
> >>>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The
fact
> >>>> that we're doing this, and their are no arguments against it, shows
> >>>> that we value quality. I'd like to take this a step further, and
see
> >>>> what we can learn from the defects in AOO 4.0.0 and what we can do
> >>>> going forward to improve.
> >>>>
> >>>> Quality, in the end, is a process, not a state of grace. We improve
> >>>> by working smarter, not working harder. The goal should be to learn
> >>>> and improve, as individuals and as a community.
> >>>>
> >>>> Every regression that made it into 4.0.0 was added there by a
> >>>> programmer. And the defect went undetected by testers. This is not
> >>>> to blame. It just means that we're all human. We know that. We all
> >>>> make mistakes. I make mistakes. A quality process is not about
> >>>> becoming perfect, but about acknowledging that we make mistakes and
> >>>> that certain formal and informal practices are needed to prevent and
> >>>> detect these mistakes.
> >>>>
> >>>> But enough about generalities. I'm hoping you'll join with me in
> >>>> examining the 32 confirmed 4.0.0 regression defects and answering a
> >>>> few questions:
> >>>>
> >>>> 1) What caused the bug? What was the "root cause"? Note:
> >>>> "programmer error" is not really a cause. We should ask what caused
> >>>> the error.
> >>>>
> >>>> 2) What can we do to prevent bugs like this from being checked in?
> >>>>
> >>>> 3) Why wasn't the bug found during testing? Was it not covered by
any
> >>>> existing test case? Was a test case run but the defect was not
> >>>> recognized? Was the defect introduced into the software after the
> >>>> tests had already been executed?
> >>>>
> >>>> 4) What can we do to ensure that bugs like this are caught during
> >>> testing?
> >>>>
> >>>> So 2 basic questions -- what went wrong and how can we prevent it in
> >>>> the future, looked at from perspective of programmers and testers.
If
> >>>> we can keep these questions in mind, and try to answer them, we may
be
> >>>> able to find some patterns that can lead to some process changes for
> >>>> AOO 4.1.
> >>>>
> >>>> You can find the 4.0.0 regressions in Bugzilla here:
> >>>>
> >>>>
> >>>
https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> -Rob
> >>>
> >>
> >> I strongly believe that one of the things that went wrong is our
limited
> >> possibility to retest (due to resources), when I look at our current
manual
> >
> > I wonder about that as well. That's one reason it would be good to
> > know how many of the confirmed regressions were introduced late in the
> > release process, and thus missed coverage in our full test pass.
> >
> >> testcases, a lot of those could be automated, e.g. with a simple UI
macro,
> >> that would enable us to run these test cases with every build. It may
sound
> >> like a dream but where I come from, we did that every night, and it
caught
> >> a lot of regression bugs and sideeffects.
> >>
> >
> > This begs the question: Is the functionality of the regressions
> > covered by our test cases? Or are they covered but we didn't execute
> > them? Or we executed them but didn't recognize the defect? I don't
> > know (yet).
> >
> >> A simple start, if to request that every bug fix, is issued with at
least
> >> one test case (automated or manual).
> >>
> >
> > Often there is, though this information lives in Bugzilla. One thing
> > we did on another (non open source) project is to mark defects in our
> > bugtracking system that should become test cases. Not every bug did
> > that. For example, a defect report to update a mispelling in the UI
> > would not lead to a new test case. But many would.
>
> we have the automated test framework that needs some more attention and
> polishing. And of course the tests have to improved to get satisfying
> result.
>
> We have
>
> BVT - build verification test
> FVT - functional verification test
> PVT - performance verification test
> SVT - system verification test
>
> But I have to confess that I have limited knowledge about it yet
I aware that we ha a limited automated framework, at least thats what I
found and played with.
but, it is not integrated into our build, or our buildbot. Especially
testing in buildbot gives better qa. An manually controlled automated test
is not really an ideal solution.
Take a look at the manual test cases, a lot could be automated, and free qa
resources for more complex testing.
rgds
jan I
>
> Juergen
>
>
> >
> > Regards,
> >
> > -Rob
> >
> >> rgds
> >> jan I.
> >>
> >>
> >>>>
> >>>> ---------------------------------------------------------------------
> >>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
> >>>> For additional commands, e-mail: qa-help@openoffice.apache.org
> >>>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
> >>> For additional commands, e-mail: qa-help@openoffice.apache.org
> >>>
> >>>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> > For additional commands, e-mail: dev-help@openoffice.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
Re: Some thoughts on quality
Posted by Jürgen Schmidt <jo...@gmail.com>.
On 8/14/13 8:30 PM, Rob Weir wrote:
> On Wed, Aug 14, 2013 at 1:55 PM, janI <ja...@apache.org> wrote:
>> On 14 August 2013 19:36, Edwin Sharp <el...@mail-page.com> wrote:
>>
>>> Dear Rob
>>> The 4.0 release was too ambitious - we should advance in smaller steps.
>>> Nothing compares to general public testing - betas and release candidates
>>> should not be avoided.
>>> TestLink cases should be less comprehesive (in terms of feature coverage)
>>> and more stress testing oriented.
>>> Regards,
>>> Edwin
>>>
>>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>>>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
>>>> that we're doing this, and their are no arguments against it, shows
>>>> that we value quality. I'd like to take this a step further, and see
>>>> what we can learn from the defects in AOO 4.0.0 and what we can do
>>>> going forward to improve.
>>>>
>>>> Quality, in the end, is a process, not a state of grace. We improve
>>>> by working smarter, not working harder. The goal should be to learn
>>>> and improve, as individuals and as a community.
>>>>
>>>> Every regression that made it into 4.0.0 was added there by a
>>>> programmer. And the defect went undetected by testers. This is not
>>>> to blame. It just means that we're all human. We know that. We all
>>>> make mistakes. I make mistakes. A quality process is not about
>>>> becoming perfect, but about acknowledging that we make mistakes and
>>>> that certain formal and informal practices are needed to prevent and
>>>> detect these mistakes.
>>>>
>>>> But enough about generalities. I'm hoping you'll join with me in
>>>> examining the 32 confirmed 4.0.0 regression defects and answering a
>>>> few questions:
>>>>
>>>> 1) What caused the bug? What was the "root cause"? Note:
>>>> "programmer error" is not really a cause. We should ask what caused
>>>> the error.
>>>>
>>>> 2) What can we do to prevent bugs like this from being checked in?
>>>>
>>>> 3) Why wasn't the bug found during testing? Was it not covered by any
>>>> existing test case? Was a test case run but the defect was not
>>>> recognized? Was the defect introduced into the software after the
>>>> tests had already been executed?
>>>>
>>>> 4) What can we do to ensure that bugs like this are caught during
>>> testing?
>>>>
>>>> So 2 basic questions -- what went wrong and how can we prevent it in
>>>> the future, looked at from perspective of programmers and testers. If
>>>> we can keep these questions in mind, and try to answer them, we may be
>>>> able to find some patterns that can lead to some process changes for
>>>> AOO 4.1.
>>>>
>>>> You can find the 4.0.0 regressions in Bugzilla here:
>>>>
>>>>
>>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>>>
>>>>
>>>> Regards,
>>>>
>>>> -Rob
>>>
>>
>> I strongly believe that one of the things that went wrong is our limited
>> possibility to retest (due to resources), when I look at our current manual
>
> I wonder about that as well. That's one reason it would be good to
> know how many of the confirmed regressions were introduced late in the
> release process, and thus missed coverage in our full test pass.
>
>> testcases, a lot of those could be automated, e.g. with a simple UI macro,
>> that would enable us to run these test cases with every build. It may sound
>> like a dream but where I come from, we did that every night, and it caught
>> a lot of regression bugs and sideeffects.
>>
>
> This begs the question: Is the functionality of the regressions
> covered by our test cases? Or are they covered but we didn't execute
> them? Or we executed them but didn't recognize the defect? I don't
> know (yet).
>
>> A simple start, if to request that every bug fix, is issued with at least
>> one test case (automated or manual).
>>
>
> Often there is, though this information lives in Bugzilla. One thing
> we did on another (non open source) project is to mark defects in our
> bugtracking system that should become test cases. Not every bug did
> that. For example, a defect report to update a mispelling in the UI
> would not lead to a new test case. But many would.
we have the automated test framework that needs some more attention and
polishing. And of course the tests have to improved to get satisfying
result.
We have
BVT - build verification test
FVT - functional verification test
PVT - performance verification test
SVT - system verification test
But I have to confess that I have limited knowledge about it yet
Juergen
>
> Regards,
>
> -Rob
>
>> rgds
>> jan I.
>>
>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>>
>>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 14, 2013 at 1:55 PM, janI <ja...@apache.org> wrote:
> On 14 August 2013 19:36, Edwin Sharp <el...@mail-page.com> wrote:
>
>> Dear Rob
>> The 4.0 release was too ambitious - we should advance in smaller steps.
>> Nothing compares to general public testing - betas and release candidates
>> should not be avoided.
>> TestLink cases should be less comprehesive (in terms of feature coverage)
>> and more stress testing oriented.
>> Regards,
>> Edwin
>>
>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>> > We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
>> > that we're doing this, and their are no arguments against it, shows
>> > that we value quality. I'd like to take this a step further, and see
>> > what we can learn from the defects in AOO 4.0.0 and what we can do
>> > going forward to improve.
>> >
>> > Quality, in the end, is a process, not a state of grace. We improve
>> > by working smarter, not working harder. The goal should be to learn
>> > and improve, as individuals and as a community.
>> >
>> > Every regression that made it into 4.0.0 was added there by a
>> > programmer. And the defect went undetected by testers. This is not
>> > to blame. It just means that we're all human. We know that. We all
>> > make mistakes. I make mistakes. A quality process is not about
>> > becoming perfect, but about acknowledging that we make mistakes and
>> > that certain formal and informal practices are needed to prevent and
>> > detect these mistakes.
>> >
>> > But enough about generalities. I'm hoping you'll join with me in
>> > examining the 32 confirmed 4.0.0 regression defects and answering a
>> > few questions:
>> >
>> > 1) What caused the bug? What was the "root cause"? Note:
>> > "programmer error" is not really a cause. We should ask what caused
>> > the error.
>> >
>> > 2) What can we do to prevent bugs like this from being checked in?
>> >
>> > 3) Why wasn't the bug found during testing? Was it not covered by any
>> > existing test case? Was a test case run but the defect was not
>> > recognized? Was the defect introduced into the software after the
>> > tests had already been executed?
>> >
>> > 4) What can we do to ensure that bugs like this are caught during
>> testing?
>> >
>> > So 2 basic questions -- what went wrong and how can we prevent it in
>> > the future, looked at from perspective of programmers and testers. If
>> > we can keep these questions in mind, and try to answer them, we may be
>> > able to find some patterns that can lead to some process changes for
>> > AOO 4.1.
>> >
>> > You can find the 4.0.0 regressions in Bugzilla here:
>> >
>> >
>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>> >
>> >
>> > Regards,
>> >
>> > -Rob
>>
>
> I strongly believe that one of the things that went wrong is our limited
> possibility to retest (due to resources), when I look at our current manual
I wonder about that as well. That's one reason it would be good to
know how many of the confirmed regressions were introduced late in the
release process, and thus missed coverage in our full test pass.
> testcases, a lot of those could be automated, e.g. with a simple UI macro,
> that would enable us to run these test cases with every build. It may sound
> like a dream but where I come from, we did that every night, and it caught
> a lot of regression bugs and sideeffects.
>
This begs the question: Is the functionality of the regressions
covered by our test cases? Or are they covered but we didn't execute
them? Or we executed them but didn't recognize the defect? I don't
know (yet).
> A simple start, if to request that every bug fix, is issued with at least
> one test case (automated or manual).
>
Often there is, though this information lives in Bugzilla. One thing
we did on another (non open source) project is to mark defects in our
bugtracking system that should become test cases. Not every bug did
that. For example, a defect report to update a mispelling in the UI
would not lead to a new test case. But many would.
Regards,
-Rob
> rgds
> jan I.
>
>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>> > For additional commands, e-mail: qa-help@openoffice.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 14, 2013 at 1:55 PM, janI <ja...@apache.org> wrote:
> On 14 August 2013 19:36, Edwin Sharp <el...@mail-page.com> wrote:
>
>> Dear Rob
>> The 4.0 release was too ambitious - we should advance in smaller steps.
>> Nothing compares to general public testing - betas and release candidates
>> should not be avoided.
>> TestLink cases should be less comprehesive (in terms of feature coverage)
>> and more stress testing oriented.
>> Regards,
>> Edwin
>>
>> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>> > We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
>> > that we're doing this, and their are no arguments against it, shows
>> > that we value quality. I'd like to take this a step further, and see
>> > what we can learn from the defects in AOO 4.0.0 and what we can do
>> > going forward to improve.
>> >
>> > Quality, in the end, is a process, not a state of grace. We improve
>> > by working smarter, not working harder. The goal should be to learn
>> > and improve, as individuals and as a community.
>> >
>> > Every regression that made it into 4.0.0 was added there by a
>> > programmer. And the defect went undetected by testers. This is not
>> > to blame. It just means that we're all human. We know that. We all
>> > make mistakes. I make mistakes. A quality process is not about
>> > becoming perfect, but about acknowledging that we make mistakes and
>> > that certain formal and informal practices are needed to prevent and
>> > detect these mistakes.
>> >
>> > But enough about generalities. I'm hoping you'll join with me in
>> > examining the 32 confirmed 4.0.0 regression defects and answering a
>> > few questions:
>> >
>> > 1) What caused the bug? What was the "root cause"? Note:
>> > "programmer error" is not really a cause. We should ask what caused
>> > the error.
>> >
>> > 2) What can we do to prevent bugs like this from being checked in?
>> >
>> > 3) Why wasn't the bug found during testing? Was it not covered by any
>> > existing test case? Was a test case run but the defect was not
>> > recognized? Was the defect introduced into the software after the
>> > tests had already been executed?
>> >
>> > 4) What can we do to ensure that bugs like this are caught during
>> testing?
>> >
>> > So 2 basic questions -- what went wrong and how can we prevent it in
>> > the future, looked at from perspective of programmers and testers. If
>> > we can keep these questions in mind, and try to answer them, we may be
>> > able to find some patterns that can lead to some process changes for
>> > AOO 4.1.
>> >
>> > You can find the 4.0.0 regressions in Bugzilla here:
>> >
>> >
>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>> >
>> >
>> > Regards,
>> >
>> > -Rob
>>
>
> I strongly believe that one of the things that went wrong is our limited
> possibility to retest (due to resources), when I look at our current manual
I wonder about that as well. That's one reason it would be good to
know how many of the confirmed regressions were introduced late in the
release process, and thus missed coverage in our full test pass.
> testcases, a lot of those could be automated, e.g. with a simple UI macro,
> that would enable us to run these test cases with every build. It may sound
> like a dream but where I come from, we did that every night, and it caught
> a lot of regression bugs and sideeffects.
>
This begs the question: Is the functionality of the regressions
covered by our test cases? Or are they covered but we didn't execute
them? Or we executed them but didn't recognize the defect? I don't
know (yet).
> A simple start, if to request that every bug fix, is issued with at least
> one test case (automated or manual).
>
Often there is, though this information lives in Bugzilla. One thing
we did on another (non open source) project is to mark defects in our
bugtracking system that should become test cases. Not every bug did
that. For example, a defect report to update a mispelling in the UI
would not lead to a new test case. But many would.
Regards,
-Rob
> rgds
> jan I.
>
>
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>> > For additional commands, e-mail: qa-help@openoffice.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by janI <ja...@apache.org>.
On 14 August 2013 19:36, Edwin Sharp <el...@mail-page.com> wrote:
> Dear Rob
> The 4.0 release was too ambitious - we should advance in smaller steps.
> Nothing compares to general public testing - betas and release candidates
> should not be avoided.
> TestLink cases should be less comprehesive (in terms of feature coverage)
> and more stress testing oriented.
> Regards,
> Edwin
>
> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
> > We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
> > that we're doing this, and their are no arguments against it, shows
> > that we value quality. I'd like to take this a step further, and see
> > what we can learn from the defects in AOO 4.0.0 and what we can do
> > going forward to improve.
> >
> > Quality, in the end, is a process, not a state of grace. We improve
> > by working smarter, not working harder. The goal should be to learn
> > and improve, as individuals and as a community.
> >
> > Every regression that made it into 4.0.0 was added there by a
> > programmer. And the defect went undetected by testers. This is not
> > to blame. It just means that we're all human. We know that. We all
> > make mistakes. I make mistakes. A quality process is not about
> > becoming perfect, but about acknowledging that we make mistakes and
> > that certain formal and informal practices are needed to prevent and
> > detect these mistakes.
> >
> > But enough about generalities. I'm hoping you'll join with me in
> > examining the 32 confirmed 4.0.0 regression defects and answering a
> > few questions:
> >
> > 1) What caused the bug? What was the "root cause"? Note:
> > "programmer error" is not really a cause. We should ask what caused
> > the error.
> >
> > 2) What can we do to prevent bugs like this from being checked in?
> >
> > 3) Why wasn't the bug found during testing? Was it not covered by any
> > existing test case? Was a test case run but the defect was not
> > recognized? Was the defect introduced into the software after the
> > tests had already been executed?
> >
> > 4) What can we do to ensure that bugs like this are caught during
> testing?
> >
> > So 2 basic questions -- what went wrong and how can we prevent it in
> > the future, looked at from perspective of programmers and testers. If
> > we can keep these questions in mind, and try to answer them, we may be
> > able to find some patterns that can lead to some process changes for
> > AOO 4.1.
> >
> > You can find the 4.0.0 regressions in Bugzilla here:
> >
> >
> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
> >
> >
> > Regards,
> >
> > -Rob
>
I strongly believe that one of the things that went wrong is our limited
possibility to retest (due to resources), when I look at our current manual
testcases, a lot of those could be automated, e.g. with a simple UI macro,
that would enable us to run these test cases with every build. It may sound
like a dream but where I come from, we did that every night, and it caught
a lot of regression bugs and sideeffects.
A simple start, if to request that every bug fix, is issued with at least
one test case (automated or manual).
rgds
jan I.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
> > For additional commands, e-mail: qa-help@openoffice.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: qa-help@openoffice.apache.org
>
>
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Wed, Aug 14, 2013 at 1:36 PM, Edwin Sharp <el...@mail-page.com> wrote:
> Dear Rob
> The 4.0 release was too ambitious - we should advance in smaller steps.
> Nothing compares to general public testing - betas and release candidates should not be avoided.
> TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented.
The number to consider here is how many defects were found and fixed
during the 4.0.0 testing, before the general public users had access?
I assume it was quite substantial. If so, the TestLink usage was
effective. In other words, we might have found fewer bugs without
using it.
This is important to keep in mind: we want to prevent or find more
bugs, but we're not starting from zero. We're starting from a process
that does a lot of things right.
I like the idea of a public beta. But consider the numbers. The 40
or so regressions that were reported came from an install base (based
on download figures since 4.0.0 was released) of around 3 million
users. Realistically, can we expect anywhere near that number in a
public beta? Or is it more likely that a beta program has 10,000
users or fewer? I don't know the answer here. But certainly a
well-publicized and used beta will find more than a beta used by just
a few hundred users.
Regards,
-Rob
> Regards,
> Edwin
>
> On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
>> that we're doing this, and their are no arguments against it, shows
>> that we value quality. I'd like to take this a step further, and see
>> what we can learn from the defects in AOO 4.0.0 and what we can do
>> going forward to improve.
>>
>> Quality, in the end, is a process, not a state of grace. We improve
>> by working smarter, not working harder. The goal should be to learn
>> and improve, as individuals and as a community.
>>
>> Every regression that made it into 4.0.0 was added there by a
>> programmer. And the defect went undetected by testers. This is not
>> to blame. It just means that we're all human. We know that. We all
>> make mistakes. I make mistakes. A quality process is not about
>> becoming perfect, but about acknowledging that we make mistakes and
>> that certain formal and informal practices are needed to prevent and
>> detect these mistakes.
>>
>> But enough about generalities. I'm hoping you'll join with me in
>> examining the 32 confirmed 4.0.0 regression defects and answering a
>> few questions:
>>
>> 1) What caused the bug? What was the "root cause"? Note:
>> "programmer error" is not really a cause. We should ask what caused
>> the error.
>>
>> 2) What can we do to prevent bugs like this from being checked in?
>>
>> 3) Why wasn't the bug found during testing? Was it not covered by any
>> existing test case? Was a test case run but the defect was not
>> recognized? Was the defect introduced into the software after the
>> tests had already been executed?
>>
>> 4) What can we do to ensure that bugs like this are caught during testing?
>>
>> So 2 basic questions -- what went wrong and how can we prevent it in
>> the future, looked at from perspective of programmers and testers. If
>> we can keep these questions in mind, and try to answer them, we may be
>> able to find some patterns that can lead to some process changes for
>> AOO 4.1.
>>
>> You can find the 4.0.0 regressions in Bugzilla here:
>>
>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>
>>
>> Regards,
>>
>> -Rob
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: qa-help@openoffice.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: qa-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Edwin Sharp <el...@mail-page.com>.
Dear Rob
The 4.0 release was too ambitious - we should advance in smaller steps.
Nothing compares to general public testing - betas and release candidates should not be avoided.
TestLink cases should be less comprehesive (in terms of feature coverage) and more stress testing oriented.
Regards,
Edwin
On Wed, Aug 14, 2013, at 19:59, Rob Weir wrote:
> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
> that we're doing this, and their are no arguments against it, shows
> that we value quality. I'd like to take this a step further, and see
> what we can learn from the defects in AOO 4.0.0 and what we can do
> going forward to improve.
>
> Quality, in the end, is a process, not a state of grace. We improve
> by working smarter, not working harder. The goal should be to learn
> and improve, as individuals and as a community.
>
> Every regression that made it into 4.0.0 was added there by a
> programmer. And the defect went undetected by testers. This is not
> to blame. It just means that we're all human. We know that. We all
> make mistakes. I make mistakes. A quality process is not about
> becoming perfect, but about acknowledging that we make mistakes and
> that certain formal and informal practices are needed to prevent and
> detect these mistakes.
>
> But enough about generalities. I'm hoping you'll join with me in
> examining the 32 confirmed 4.0.0 regression defects and answering a
> few questions:
>
> 1) What caused the bug? What was the "root cause"? Note:
> "programmer error" is not really a cause. We should ask what caused
> the error.
>
> 2) What can we do to prevent bugs like this from being checked in?
>
> 3) Why wasn't the bug found during testing? Was it not covered by any
> existing test case? Was a test case run but the defect was not
> recognized? Was the defect introduced into the software after the
> tests had already been executed?
>
> 4) What can we do to ensure that bugs like this are caught during testing?
>
> So 2 basic questions -- what went wrong and how can we prevent it in
> the future, looked at from perspective of programmers and testers. If
> we can keep these questions in mind, and try to answer them, we may be
> able to find some patterns that can lead to some process changes for
> AOO 4.1.
>
> You can find the 4.0.0 regressions in Bugzilla here:
>
> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>
>
> Regards,
>
> -Rob
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: qa-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: qa-unsubscribe@openoffice.apache.org
For additional commands, e-mail: qa-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Rob Weir <ro...@apache.org>.
On Thu, Aug 15, 2013 at 4:08 AM, Oliver-Rainer Wittmann
<or...@googlemail.com> wrote:
> Hi,
>
>
> On 14.08.2013 18:59, Rob Weir wrote:
>>
>> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
>> that we're doing this, and their are no arguments against it, shows
>> that we value quality. I'd like to take this a step further, and see
>> what we can learn from the defects in AOO 4.0.0 and what we can do
>> going forward to improve.
>>
>> Quality, in the end, is a process, not a state of grace. We improve
>> by working smarter, not working harder. The goal should be to learn
>> and improve, as individuals and as a community.
>>
>> Every regression that made it into 4.0.0 was added there by a
>> programmer. And the defect went undetected by testers. This is not
>> to blame. It just means that we're all human. We know that. We all
>> make mistakes. I make mistakes. A quality process is not about
>> becoming perfect, but about acknowledging that we make mistakes and
>> that certain formal and informal practices are needed to prevent and
>> detect these mistakes.
>>
>> But enough about generalities. I'm hoping you'll join with me in
>> examining the 32 confirmed 4.0.0 regression defects and answering a
>> few questions:
>>
>> 1) What caused the bug? What was the "root cause"? Note:
>> "programmer error" is not really a cause. We should ask what caused
>> the error.
>>
>> 2) What can we do to prevent bugs like this from being checked in?
>>
>> 3) Why wasn't the bug found during testing? Was it not covered by any
>> existing test case? Was a test case run but the defect was not
>> recognized? Was the defect introduced into the software after the
>> tests had already been executed?
>>
>> 4) What can we do to ensure that bugs like this are caught during testing?
>>
>> So 2 basic questions -- what went wrong and how can we prevent it in
>> the future, looked at from perspective of programmers and testers. If
>> we can keep these questions in mind, and try to answer them, we may be
>> able to find some patterns that can lead to some process changes for
>> AOO 4.1.
>>
>> You can find the 4.0.0 regressions in Bugzilla here:
>>
>>
>> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>>
>
> Please include also issue with status ACCEPTED.
>
Done. That now gives us 36.
-Rob
> Best regards, Oliver.
>
>>
>> Regards,
>>
>> -Rob
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org
Re: Some thoughts on quality
Posted by Oliver-Rainer Wittmann <or...@googlemail.com>.
Hi,
On 14.08.2013 18:59, Rob Weir wrote:
> We're working now on AOO 4.0.1, to fix defects in AOO 4.0.0. The fact
> that we're doing this, and their are no arguments against it, shows
> that we value quality. I'd like to take this a step further, and see
> what we can learn from the defects in AOO 4.0.0 and what we can do
> going forward to improve.
>
> Quality, in the end, is a process, not a state of grace. We improve
> by working smarter, not working harder. The goal should be to learn
> and improve, as individuals and as a community.
>
> Every regression that made it into 4.0.0 was added there by a
> programmer. And the defect went undetected by testers. This is not
> to blame. It just means that we're all human. We know that. We all
> make mistakes. I make mistakes. A quality process is not about
> becoming perfect, but about acknowledging that we make mistakes and
> that certain formal and informal practices are needed to prevent and
> detect these mistakes.
>
> But enough about generalities. I'm hoping you'll join with me in
> examining the 32 confirmed 4.0.0 regression defects and answering a
> few questions:
>
> 1) What caused the bug? What was the "root cause"? Note:
> "programmer error" is not really a cause. We should ask what caused
> the error.
>
> 2) What can we do to prevent bugs like this from being checked in?
>
> 3) Why wasn't the bug found during testing? Was it not covered by any
> existing test case? Was a test case run but the defect was not
> recognized? Was the defect introduced into the software after the
> tests had already been executed?
>
> 4) What can we do to ensure that bugs like this are caught during testing?
>
> So 2 basic questions -- what went wrong and how can we prevent it in
> the future, looked at from perspective of programmers and testers. If
> we can keep these questions in mind, and try to answer them, we may be
> able to find some patterns that can lead to some process changes for
> AOO 4.1.
>
> You can find the 4.0.0 regressions in Bugzilla here:
>
> https://issues.apache.org/ooo/buglist.cgi?cmdtype=dorem&remaction=run&namedcmd=400_regressions&sharer_id=248521&list_id=80834
>
Please include also issue with status ACCEPTED.
Best regards, Oliver.
>
> Regards,
>
> -Rob
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org