You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Yi Xuan Liu <li...@gmail.com> on 2012/07/06 05:07:11 UTC
[QA] AOO 3.4.1 Performance Verification Test (PVT) Introduction
Hi, all:
I wrote a wiki
http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introductionabout
PVT project in AOO 3.4.1.
Any comment is welcomed!
Re: [QA] AOO 3.4.1 Performance Verification Test (PVT) Introduction
Posted by Chao Huang <ch...@gmail.com>.
hi, YiXuan
Thanks for your sharing. I think that it is better if add system memory
index in PVT report, which can help to track memory leak issue.
2012/7/6 Yi Xuan Liu <li...@gmail.com>
> Hi, all:
>
> I wrote a wiki
>
> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introductionabout
> PVT project in AOO 3.4.1.
> Any comment is welcomed!
>
--
Best regards,
Chao Huang
Re: [QA] AOO 3.4.1 Performance Verification Test (PVT) Introduction
Posted by Yi Xuan Liu <li...@gmail.com>.
"Standard deviation" shows the how much variation exists from the average.
The smaller the "standard deviation" is, the more stable the test is.
In the PVT, we want to use the "standard deviation" to exclude the case
that one extreme value changing the average when doing a build-to-build
comparison.
On Fri, Jul 6, 2012 at 4:13 PM, Andre Fischer <af...@a-w-f.de> wrote:
> On 06.07.2012 05:11, zhangjf wrote:
>
>> The right url is
>> http://wiki.services.**openoffice.org/wiki/**Performance/AOO3.4.1_PVT_**
>> Introduction<http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introduction>
>>
>> On Fri, Jul 6, 2012 at 11:07 AM, Yi Xuan Liu <li...@gmail.com>
>> wrote:
>>
>>> Hi, all:
>>>
>>> I wrote a wiki
>>> http://wiki.services.**openoffice.org/wiki/**Performance/AOO3.4.1_PVT_**
>>> Introductionabout<http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introductionabout>
>>> PVT project in AOO 3.4.1.
>>> Any comment is welcomed!
>>>
>>
> In section "Result computation Method" it says
>
> Note:the standard deviation is to reflect whether the 8 round
> results are stable.
>
> How exactly do you determine whether a result is stable?
>
> -Andre
>
>
Re: [QA] AOO 3.4.1 Performance Verification Test (PVT) Introduction
Posted by Andre Fischer <af...@a-w-f.de>.
On 06.07.2012 05:11, zhangjf wrote:
> The right url is
> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introduction
>
> On Fri, Jul 6, 2012 at 11:07 AM, Yi Xuan Liu <li...@gmail.com> wrote:
>> Hi, all:
>>
>> I wrote a wiki
>> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introductionabout
>> PVT project in AOO 3.4.1.
>> Any comment is welcomed!
In section "Result computation Method" it says
Note:the standard deviation is to reflect whether the 8 round
results are stable.
How exactly do you determine whether a result is stable?
-Andre
Re: [QA] AOO 3.4.1 Performance Verification Test (PVT) Introduction
Posted by Yi Xuan Liu <li...@gmail.com>.
Rob, thanks for your reply. My answer to your question is in below.
1) Not yet. I'll check in code in next week ASAP.
2) It takes about 2 hours to complete the whole text on my W500 laptop. (
CPU:2.53 GHz; Mem: 3GB; OS: XP SP3).
So, if running 100 times, it would cost about 1 day. Yes, it is also a
good method to check Memory leak :)
3) The order *b *is
4) No, don't restart OpenOffice during the whole PVT.
Also thanks for your advice about the trick mentioned. I'll try it.
On Sat, Jul 7, 2012 at 1:28 AM, Rob Weir <ro...@apache.org> wrote:
> On Thu, Jul 5, 2012 at 11:11 PM, zhangjf <zh...@apache.org> wrote:
> > The right url is
> >
> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introduction
> >
>
> Great. Thanks!
>
> A few questions:
>
> 1) Are the scripts and test documents checked into SVN?
>
> 2) How long does it take for a complete run of the tests?
>
> 3) What is the order of the tests? For example, are you doing:
>
> a) document 1 run 1, document 1 run 2, document 1 run 3...document 1
> run 8, document 2 run 1...document 2 run 8, etc.
>
> or
>
> b) document 1 run 1, document 2 run 1, document 3 run 1...document N
> run 1, then document 1 run 2, document 2 run 2, etc.
>
>
> 4) Within the test do you restart OpenOffice? If so, do you restart
> after every document? Or every measurement?
>
>
> If it is at all possible to take more measurements, I think we would
> get more high quality results. Right now, you take 8 measurements and
> throw away 3 of them (first run, highest time and lowest time). That
> throws away information and biases the results because the first run
> is probably also slower, so you toss out the two slowest runs but only
> toss out the single fastest run. But the fastest run is probably also
> the most accurate one, since there are many things in a test that can
> accidentally slow things down, but almost nothing can happen to make a
> test run faster. (assuming the test logic is accurate).
>
> In general, in an experiment, keep all the data you have, and get more
> accurate results by doing more repetitions.
>
> For example, what if we did 100 iterations of each test? How long
> would that take?
>
> If we did that, it would have some benefits:
>
> 1) We wouldn't need to worry about tossing out high and low values.
> Our error bounds would be good because of the number of runs we have.
> The impact of any one anomalous measurement will be much smaller.
>
> 2) We could at the same time look at the trend of the measurement over
> the test run. For example, compare the average of the first 10% of
> the runs with the average of the last 10%. Is there a difference? If
> a test slows down over time that might indicate a memory leak or
> other problem. You will never find this with only 8 measurements.
>
> 3) It would tell us the distribution of timings, as well as the average.
>
> (Another trick. If you are going to do N load measurements of the
> same document, maybe start the test run by creating N identical copies
> of the same document on disk. Then load each copy only once. That
> helps even out the disk cache and I/O environment compared to loading
> the exact same file N times)
>
> -Rob
>
>
> > On Fri, Jul 6, 2012 at 11:07 AM, Yi Xuan Liu <li...@gmail.com>
> wrote:
> >> Hi, all:
> >>
> >> I wrote a wiki
> >>
> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introductionabout
> >> PVT project in AOO 3.4.1.
> >> Any comment is welcomed!
>
Re: [QA] AOO 3.4.1 Performance Verification Test (PVT) Introduction
Posted by Rob Weir <ro...@apache.org>.
On Thu, Jul 5, 2012 at 11:11 PM, zhangjf <zh...@apache.org> wrote:
> The right url is
> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introduction
>
Great. Thanks!
A few questions:
1) Are the scripts and test documents checked into SVN?
2) How long does it take for a complete run of the tests?
3) What is the order of the tests? For example, are you doing:
a) document 1 run 1, document 1 run 2, document 1 run 3...document 1
run 8, document 2 run 1...document 2 run 8, etc.
or
b) document 1 run 1, document 2 run 1, document 3 run 1...document N
run 1, then document 1 run 2, document 2 run 2, etc.
4) Within the test do you restart OpenOffice? If so, do you restart
after every document? Or every measurement?
If it is at all possible to take more measurements, I think we would
get more high quality results. Right now, you take 8 measurements and
throw away 3 of them (first run, highest time and lowest time). That
throws away information and biases the results because the first run
is probably also slower, so you toss out the two slowest runs but only
toss out the single fastest run. But the fastest run is probably also
the most accurate one, since there are many things in a test that can
accidentally slow things down, but almost nothing can happen to make a
test run faster. (assuming the test logic is accurate).
In general, in an experiment, keep all the data you have, and get more
accurate results by doing more repetitions.
For example, what if we did 100 iterations of each test? How long
would that take?
If we did that, it would have some benefits:
1) We wouldn't need to worry about tossing out high and low values.
Our error bounds would be good because of the number of runs we have.
The impact of any one anomalous measurement will be much smaller.
2) We could at the same time look at the trend of the measurement over
the test run. For example, compare the average of the first 10% of
the runs with the average of the last 10%. Is there a difference? If
a test slows down over time that might indicate a memory leak or
other problem. You will never find this with only 8 measurements.
3) It would tell us the distribution of timings, as well as the average.
(Another trick. If you are going to do N load measurements of the
same document, maybe start the test run by creating N identical copies
of the same document on disk. Then load each copy only once. That
helps even out the disk cache and I/O environment compared to loading
the exact same file N times)
-Rob
> On Fri, Jul 6, 2012 at 11:07 AM, Yi Xuan Liu <li...@gmail.com> wrote:
>> Hi, all:
>>
>> I wrote a wiki
>> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introductionabout
>> PVT project in AOO 3.4.1.
>> Any comment is welcomed!
Re: [QA] AOO 3.4.1 Performance Verification Test (PVT) Introduction
Posted by zhangjf <zh...@apache.org>.
The right url is
http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introduction
On Fri, Jul 6, 2012 at 11:07 AM, Yi Xuan Liu <li...@gmail.com> wrote:
> Hi, all:
>
> I wrote a wiki
> http://wiki.services.openoffice.org/wiki/Performance/AOO3.4.1_PVT_Introductionabout
> PVT project in AOO 3.4.1.
> Any comment is welcomed!