You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@openoffice.apache.org by Linyi Li <li...@gmail.com> on 2012/11/16 09:34:00 UTC
[QA REPORT] - Milestone build r1400866 performance issue
Hi all,
I did GUI performance test on milestone build r1400866.
Comparing the performance result with trunk build result(r1383738), I found
most of the test cases have about 20% downgrade except xls/xlsx loading. I
manually checked some cases, the results are consistent with the automation
result.
The results are inconsistent with the trunk build comparing: r1408441 vs
r1383738. Build r1408441 has no huge downgrade comparing with r1383738
except for 3 issues[1][2][3].
I am not sure the milestone build has how many differences between the
trunk build. I put the result here[4], so we can analyze if there is any
other defect.
About the test result[4], I want to explain a little.
There are 5 platform result. You can choose "Ubuntu-12.04-amd64@aoopvt7" to
check the problem.
Milestone build is "350m1(Build:9611)-2012-10-23". Others are all trunk
build. If you want to compare the milestone buid and the trunk build "
350m1(Build:9610)-2012-09-12" result, just check "
350m1(Build:9611)-2012-10-23" and "350m1(Build:9610)-2012-09-12". The last
column is the baseline. Downgrade more than 10% are marked as pink or red.
[1]
*Bug 121200* <https://issues.apache.org/ooo/show_bug.cgi?id=121200> -
[Regression][Automation][Performance]Severe
downgrade to save xls sample files.
[2]
*Bug 121256* <https://issues.apache.org/ooo/show_bug.cgi?id=121256> - Crash
when saving ppt on Linux.
[3]
*Bug 121356* <https://issues.apache.org/ooo/show_bug.cgi?id=121356> - Crash
when loading pptx file second/third time after deleting user installation
directory.
[4] http://people.apache.org/~liuzhe/testdashboard/#pvt_gui_Benchmark
--
Best wishes.
Linyi Li
Re: [QA REPORT] - Milestone build r1400866 performance issue
Posted by li zhang <li...@gmail.com>.
Hi, all
I have checked pvt reports after r1400866, and it is becoming stable except
for 3 issues[1][2][3] refered by linyi. Maybe new code delivery has fix
this unstable problem. I suggest that we can wait for pvt results of
following revisions and check their status. If this problem still exists,
then we can go on with it.
On Tue, Nov 20, 2012 at 2:58 PM, Linyi Li <li...@gmail.com> wrote:
> Hi Herbert,
>
> Assuming you mean the officially released AOO341 as "milestone release
> build" and with "trunk build" you mean the snapshot builds [1] or the
> nightly builds [2] of trunk,
>
>
> Trunk build, I mean buildbot build:
> http://ci.apache.org/projects/openoffice/install/
> Milestone release build, I mean r1400866:
>
> https://cwiki.apache.org/confluence/display/OOOUSERS/Development+Snapshot+Builds
> 341 release build, I mean r1372282:
> http://www.openoffice.org/download/index.html
>
> then the biggest differences are probably:
>
> > - the nightly builds have no binfilter component
> > - most dev-snapshot and all nightly builds don't have system-integration
> > - new features / bugfixes / other changes in trunk
> >
> > [1] http://s.apache.org/aoo_**devsnaps <http://s.apache.org/aoo_devsnaps
> >
> > [2] http://ci.apache.org/projects/**openoffice<
> http://ci.apache.org/projects/openoffice>
> >
> >
> From the above differences, do you think there will be any performance
> difference between milestone release build and trunk build?
>
>
>
> > As for the numbers itself I'm not sure how to interpret them, e.g. the
> > dashboard indicates that the PVT test named "savePlainXLS" has improved
> by
> > 28% [3] on Linux32 between AOO341 and r1407366, whereas the individual
> runs
> > indicate a performance improvement from 104sec [4] -to 96sec [5], which
> are
> > about 7.7%. There are similar observations for other individual test
> > results.
> >
> > [3]
> http://people.apache.org/~**liuzhe/testdashboard/#pvt_gui_**Benchmark<
> http://people.apache.org/~liuzhe/testdashboard/#pvt_gui_Benchmark>
> > [4] http://people.apache.org/~**liuzhe/test/341m1%28Build:**
> > 9593%29@2012-08-13@1372282@**pvt.gui.Benchmark@Ubuntu-12.**
> > 04-i386@aoopvt2/result.html<
> http://people.apache.org/~liuzhe/test/341m1%28Build:9593%29@2012-08-13@1372282@pvt.gui.Benchmark@Ubuntu-12.04-i386@aoopvt2/result.html
> >
> > [5] http://people.apache.org/~**liuzhe/test/350m1%28Build:**
> > 9611%29@2012-11-10@1407366@**pvt.gui.Benchmark@Ubuntu-12.**
> > 04-i386@aoopvt2/result.html<
> http://people.apache.org/~liuzhe/test/350m1%28Build:9611%29@2012-11-10@1407366@pvt.gui.Benchmark@Ubuntu-12.04-i386@aoopvt2/result.html
> >
> >
> >
> The time in [3] is the average time of each test scenario, the unit is
> millisecond. I tested each scenario 8 times to get the average time in case
> one time is unstable. [4][5] is the total time of one test method, namely
> from starting OO to closing OO, 8 times of each scenario, the unit is
> second. The total time is not the accurate time of each test scenario.
>
>
> > Another question that worries me is whether the tests are run with the
> > test framework from the same revision they are testing? Or if they are
> all
> > run by the same test framework then the question is: which revision?
> >
> > Herbert
> >
> >
> All the tests are run with the same test framework. The revision number may
> be r1372282.
>
> --
> Best wishes.
> Linyi Li
>
Re: [QA REPORT] - Milestone build r1400866 performance issue
Posted by Linyi Li <li...@gmail.com>.
Hi Herbert,
Assuming you mean the officially released AOO341 as "milestone release
build" and with "trunk build" you mean the snapshot builds [1] or the
nightly builds [2] of trunk,
Trunk build, I mean buildbot build:
http://ci.apache.org/projects/openoffice/install/
Milestone release build, I mean r1400866:
https://cwiki.apache.org/confluence/display/OOOUSERS/Development+Snapshot+Builds
341 release build, I mean r1372282:
http://www.openoffice.org/download/index.html
then the biggest differences are probably:
> - the nightly builds have no binfilter component
> - most dev-snapshot and all nightly builds don't have system-integration
> - new features / bugfixes / other changes in trunk
>
> [1] http://s.apache.org/aoo_**devsnaps <http://s.apache.org/aoo_devsnaps>
> [2] http://ci.apache.org/projects/**openoffice<http://ci.apache.org/projects/openoffice>
>
>
>From the above differences, do you think there will be any performance
difference between milestone release build and trunk build?
> As for the numbers itself I'm not sure how to interpret them, e.g. the
> dashboard indicates that the PVT test named "savePlainXLS" has improved by
> 28% [3] on Linux32 between AOO341 and r1407366, whereas the individual runs
> indicate a performance improvement from 104sec [4] -to 96sec [5], which are
> about 7.7%. There are similar observations for other individual test
> results.
>
> [3] http://people.apache.org/~**liuzhe/testdashboard/#pvt_gui_**Benchmark<http://people.apache.org/~liuzhe/testdashboard/#pvt_gui_Benchmark>
> [4] http://people.apache.org/~**liuzhe/test/341m1%28Build:**
> 9593%29@2012-08-13@1372282@**pvt.gui.Benchmark@Ubuntu-12.**
> 04-i386@aoopvt2/result.html<http://people.apache.org/~liuzhe/test/341m1%28Build:9593%29@2012-08-13@1372282@pvt.gui.Benchmark@Ubuntu-12.04-i386@aoopvt2/result.html>
> [5] http://people.apache.org/~**liuzhe/test/350m1%28Build:**
> 9611%29@2012-11-10@1407366@**pvt.gui.Benchmark@Ubuntu-12.**
> 04-i386@aoopvt2/result.html<http://people.apache.org/~liuzhe/test/350m1%28Build:9611%29@2012-11-10@1407366@pvt.gui.Benchmark@Ubuntu-12.04-i386@aoopvt2/result.html>
>
>
The time in [3] is the average time of each test scenario, the unit is
millisecond. I tested each scenario 8 times to get the average time in case
one time is unstable. [4][5] is the total time of one test method, namely
from starting OO to closing OO, 8 times of each scenario, the unit is
second. The total time is not the accurate time of each test scenario.
> Another question that worries me is whether the tests are run with the
> test framework from the same revision they are testing? Or if they are all
> run by the same test framework then the question is: which revision?
>
> Herbert
>
>
All the tests are run with the same test framework. The revision number may
be r1372282.
--
Best wishes.
Linyi Li
Re: [QA REPORT] - Milestone build r1400866 performance issue
Posted by Herbert Duerr <hd...@apache.org>.
Hi Linyi,
On 19.11.2012 09:20, Linyi Li wrote:
> Analysed the whole results, I think the downgrade in ubuntu 32/64 bit may
> not be defect. I think it maybe the build difference.
> Because I tested trunk build r1400212 and 1401384 on ubuntu 32/64bit, no
> big downgrade.
>
> So does anybody can give some clue that there is any difference between
> trunk build and milestone release build?
Assuming you mean the officially released AOO341 as "milestone release
build" and with "trunk build" you mean the snapshot builds [1] or the
nightly builds [2] of trunk, then the biggest differences are probably:
- the nightly builds have no binfilter component
- most dev-snapshot and all nightly builds don't have system-integration
- new features / bugfixes / other changes in trunk
[1] http://s.apache.org/aoo_devsnaps
[2] http://ci.apache.org/projects/openoffice
As for the numbers itself I'm not sure how to interpret them, e.g. the
dashboard indicates that the PVT test named "savePlainXLS" has improved
by 28% [3] on Linux32 between AOO341 and r1407366, whereas the
individual runs indicate a performance improvement from 104sec [4] -to
96sec [5], which are about 7.7%. There are similar observations for
other individual test results.
[3] http://people.apache.org/~liuzhe/testdashboard/#pvt_gui_Benchmark
[4]
http://people.apache.org/~liuzhe/test/341m1%28Build:9593%29@2012-08-13@1372282@pvt.gui.Benchmark@Ubuntu-12.04-i386@aoopvt2/result.html
[5]
http://people.apache.org/~liuzhe/test/350m1%28Build:9611%29@2012-11-10@1407366@pvt.gui.Benchmark@Ubuntu-12.04-i386@aoopvt2/result.html
Another question that worries me is whether the tests are run with the
test framework from the same revision they are testing? Or if they are
all run by the same test framework then the question is: which revision?
Herbert
Re: [QA REPORT] - Milestone build r1400866 performance issue
Posted by Linyi Li <li...@gmail.com>.
Analysed the whole results, I think the downgrade in ubuntu 32/64 bit may
not be defect. I think it maybe the build difference.
Because I tested trunk build r1400212 and 1401384 on ubuntu 32/64bit, no
big downgrade.
So does anybody can give some clue that there is any difference between
trunk build and milestone release build?
On Mon, Nov 19, 2012 at 10:06 AM, Linyi Li <li...@gmail.com> wrote:
> I added 341 release build 1372282 result "341m1(Build:9593)-2012-08-13"
> to [1] and updated some platforms result.
>
> The results showed that there are downgrade in Ubuntu 12.04 32bit and
> 64bit, window xp and windows 7 is OK.
>
> [1] http://people.apache.org/~liuzhe/testdashboard/#pvt_gui_Benchmark
>
>
>
> On Fri, Nov 16, 2012 at 4:34 PM, Linyi Li <li...@gmail.com> wrote:
>
>> Hi all,
>>
>> I did GUI performance test on milestone build r1400866.
>> Comparing the performance result with trunk build result(r1383738), I
>> found most of the test cases have about 20% downgrade except xls/xlsx
>> loading. I manually checked some cases, the results are consistent with the
>> automation result.
>>
>> The results are inconsistent with the trunk build comparing: r1408441 vs
>> r1383738. Build r1408441 has no huge downgrade comparing with r1383738
>> except for 3 issues[1][2][3].
>>
>> I am not sure the milestone build has how many differences between the
>> trunk build. I put the result here[4], so we can analyze if there is any
>> other defect.
>>
>> About the test result[4], I want to explain a little.
>> There are 5 platform result. You can choose "Ubuntu-12.04-amd64@aoopvt7"
>> to check the problem.
>> Milestone build is "350m1(Build:9611)-2012-10-23". Others are all trunk
>> build. If you want to compare the milestone buid and the trunk build "
>> 350m1(Build:9610)-2012-09-12" result, just check "
>> 350m1(Build:9611)-2012-10-23" and "350m1(Build:9610)-2012-09-12". The
>> last column is the baseline. Downgrade more than 10% are marked as pink or
>> red.
>>
>>
>>
>> [1]
>> *Bug 121200* <https://issues.apache.org/ooo/show_bug.cgi?id=121200> - [Regression][Automation][Performance]Severe
>> downgrade to save xls sample files.
>> [2]
>> *Bug 121256* <https://issues.apache.org/ooo/show_bug.cgi?id=121256> - Crash
>> when saving ppt on Linux.
>> [3]
>> *Bug 121356* <https://issues.apache.org/ooo/show_bug.cgi?id=121356> - Crash
>> when loading pptx file second/third time after deleting user installation
>> directory.
>>
>> [4] http://people.apache.org/~liuzhe/testdashboard/#pvt_gui_Benchmark
>> --
>> Best wishes.
>> Linyi Li
>>
>>
>
>
> --
> Best wishes.
> Linyi Li
>
>
--
Best wishes.
Linyi Li
Re: [QA REPORT] - Milestone build r1400866 performance issue
Posted by Linyi Li <li...@gmail.com>.
I added 341 release build 1372282 result "341m1(Build:9593)-2012-08-13" to
[1] and updated some platforms result.
The results showed that there are downgrade in Ubuntu 12.04 32bit and
64bit, window xp and windows 7 is OK.
[1] http://people.apache.org/~liuzhe/testdashboard/#pvt_gui_Benchmark
On Fri, Nov 16, 2012 at 4:34 PM, Linyi Li <li...@gmail.com> wrote:
> Hi all,
>
> I did GUI performance test on milestone build r1400866.
> Comparing the performance result with trunk build result(r1383738), I
> found most of the test cases have about 20% downgrade except xls/xlsx
> loading. I manually checked some cases, the results are consistent with the
> automation result.
>
> The results are inconsistent with the trunk build comparing: r1408441 vs
> r1383738. Build r1408441 has no huge downgrade comparing with r1383738
> except for 3 issues[1][2][3].
>
> I am not sure the milestone build has how many differences between the
> trunk build. I put the result here[4], so we can analyze if there is any
> other defect.
>
> About the test result[4], I want to explain a little.
> There are 5 platform result. You can choose "Ubuntu-12.04-amd64@aoopvt7"
> to check the problem.
> Milestone build is "350m1(Build:9611)-2012-10-23". Others are all trunk
> build. If you want to compare the milestone buid and the trunk build "
> 350m1(Build:9610)-2012-09-12" result, just check "
> 350m1(Build:9611)-2012-10-23" and "350m1(Build:9610)-2012-09-12". The
> last column is the baseline. Downgrade more than 10% are marked as pink or
> red.
>
>
>
> [1]
> *Bug 121200* <https://issues.apache.org/ooo/show_bug.cgi?id=121200> - [Regression][Automation][Performance]Severe
> downgrade to save xls sample files.
> [2]
> *Bug 121256* <https://issues.apache.org/ooo/show_bug.cgi?id=121256> - Crash
> when saving ppt on Linux.
> [3]
> *Bug 121356* <https://issues.apache.org/ooo/show_bug.cgi?id=121356> - Crash
> when loading pptx file second/third time after deleting user installation
> directory.
>
> [4] http://people.apache.org/~liuzhe/testdashboard/#pvt_gui_Benchmark
> --
> Best wishes.
> Linyi Li
>
>
--
Best wishes.
Linyi Li