Regression tests?

[Patricia Shanahan <pa...@acm.org>]

I have a few simple changes I would like to check in. Before doing that, 
I would normally run regression tests against my working copy. In any 
case, as one of the few people who are building on Windows, I should 
test early, test often.

What tests do people normally run to check that changes do not have 
unintended consequences?

Thanks,

Patricia

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org

RE: Regression tests?

["Dennis E. Hamilton" <de...@acm.org>]

+1 Patricia.  Go for it.

> -----Original Message-----
> From: Patricia Shanahan [mailto:pats@acm.org]
> Sent: Tuesday, March 29, 2016 06:00
> To: dev@openoffice.apache.org
> Subject: Re: Regression tests?
> 
> Thanks for this extremely useful information - far too useful to only
> exist in a mail archive. Any objection to me creating a "Test Overview"
> Wiki page and populating it initially by pasting in the body of your
> message?
> 
> I am running with the unit tests disabled for two reasons. Currently,
> some of them fail in my environment for what I think are test artifact
> reasons rather than actual bugs. Additionally, given how slow builds
> are, I am reluctant to add anything to that process.
> 
> In general, I think we really need scripts or ant targets that just run
> various categories of tests.
> 
> Patricia
[ ... ]


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org

Re: Regression tests?

[Patricia Shanahan <pa...@acm.org>]

Thanks for this extremely useful information - far too useful to only 
exist in a mail archive. Any objection to me creating a "Test Overview" 
Wiki page and populating it initially by pasting in the body of your 
message?

I am running with the unit tests disabled for two reasons. Currently, 
some of them fail in my environment for what I think are test artifact 
reasons rather than actual bugs. Additionally, given how slow builds 
are, I am reluctant to add anything to that process.

In general, I think we really need scripts or ant targets that just run 
various categories of tests.

Patricia

On 3/29/2016 5:18 AM, Damjan Jovanovic wrote:
> We have many different types of tests. The documentation on them is
> sparse, scattered and outdated; a Wiki page is sorely needed.
>
> Unit tests are used by a few modules, run during the build (unless you
> pass --disable-unit-tests to ./configure), cause the build to fail if
> they fail (though I've noticed segmentation faults in tests are
> apparently ignored), but they cannot rely on the entire AOO
> environment to be available, only their module to have been built.
> Previously we used cppunit for this, but have since moved to Google
> Test. This is best for low-level self-contained functionality. They
> are always in C/C++.
>
> "Subsequent tests" are tests run after AOO is built. These use Junit
> and are written in Java. IIRC they're off by default, you have to
> define the environment variable OOO_SUBSEQUENT_TESTS when you run
> "build" on the module you want to run them on, but some extra Google
> Tests also run with that variable. I don't really know much about
> them.
>
> The qa tests are integration tests in the test/ directory (same
> directory level as main/). They do not run during the build, and have
> to be invoked separately, but can be invoked on an arbitrary AOO
> instance, so you can easily run the same tests against different AOO
> versions. You have to leave your computer unattended while they run,
> as the tests move and click the mouse, type buttons on the keyboard,
> and use the clipboard. There are several test categories: the bvt
> (basic verification test) is a default set of basic tests, fvt
> (functional verification test) has a lot more, pvt (performance
> verification tests) measures performance, etc. Because each test opens
> a new AOO instance and exits afterwards, they take ages: 10 minutes
> for bvt, 1 hour for fvt. You can read more on
> https://wiki.openoffice.org/wiki/QA/test_automation_guide#Getting_started_with_command_line
>
> In main/smoketestoo_native there is a "smoketest" written in
> StarBasic, that tests various functionality. It began to be refactored
> as a qa test, and has been copied there now and runs as a bvt test.
>
> We also have a spreadsheet test running as an fvt test, which can be
> used to verify things work by examining a specially formatted
> spreadsheet that uses spreadsheet functions or macros to obtain
> results. You can see examples in
> test/testuno/source/fvt/uno/sc/formula/TestFormulaDocs.java
>
> It's worth noting that these testing frameworks need some attention.
> The refactoring effort while IBM was still contributing left things
> unfinished. Some test code disappeared, breaking some unit tests and
> subsequent tests. Many tests (about 2 bvt tests and 25 fvt tests) have
> been failing for a while. The smoketest in test/ needs to use the
> document built in main/smoketestoo_native during the build instead of
> its private copy (but the test/ directory is designed to be separate
> from the build - should it build the document using Java functions
> instead?). Also these Java qa tests seem new and some are unfinished -
> were they ported from somewhere (smoketest?)?
>
> Damjan
>
> On Tue, Mar 29, 2016 at 5:53 AM, Patricia Shanahan <pa...@acm.org> wrote:
>> I have a few simple changes I would like to check in. Before doing that, I
>> would normally run regression tests against my working copy. In any case, as
>> one of the few people who are building on Windows, I should test early, test
>> often.
>>
>> What tests do people normally run to check that changes do not have
>> unintended consequences?
>>
>> Thanks,
>>
>> Patricia
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
>> For additional commands, e-mail: dev-help@openoffice.apache.org
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org

Re: Regression tests?

[Damjan Jovanovic <da...@apache.org>]

We have many different types of tests. The documentation on them is
sparse, scattered and outdated; a Wiki page is sorely needed.

Unit tests are used by a few modules, run during the build (unless you
pass --disable-unit-tests to ./configure), cause the build to fail if
they fail (though I've noticed segmentation faults in tests are
apparently ignored), but they cannot rely on the entire AOO
environment to be available, only their module to have been built.
Previously we used cppunit for this, but have since moved to Google
Test. This is best for low-level self-contained functionality. They
are always in C/C++.

"Subsequent tests" are tests run after AOO is built. These use Junit
and are written in Java. IIRC they're off by default, you have to
define the environment variable OOO_SUBSEQUENT_TESTS when you run
"build" on the module you want to run them on, but some extra Google
Tests also run with that variable. I don't really know much about
them.

The qa tests are integration tests in the test/ directory (same
directory level as main/). They do not run during the build, and have
to be invoked separately, but can be invoked on an arbitrary AOO
instance, so you can easily run the same tests against different AOO
versions. You have to leave your computer unattended while they run,
as the tests move and click the mouse, type buttons on the keyboard,
and use the clipboard. There are several test categories: the bvt
(basic verification test) is a default set of basic tests, fvt
(functional verification test) has a lot more, pvt (performance
verification tests) measures performance, etc. Because each test opens
a new AOO instance and exits afterwards, they take ages: 10 minutes
for bvt, 1 hour for fvt. You can read more on
https://wiki.openoffice.org/wiki/QA/test_automation_guide#Getting_started_with_command_line

In main/smoketestoo_native there is a "smoketest" written in
StarBasic, that tests various functionality. It began to be refactored
as a qa test, and has been copied there now and runs as a bvt test.

We also have a spreadsheet test running as an fvt test, which can be
used to verify things work by examining a specially formatted
spreadsheet that uses spreadsheet functions or macros to obtain
results. You can see examples in
test/testuno/source/fvt/uno/sc/formula/TestFormulaDocs.java

It's worth noting that these testing frameworks need some attention.
The refactoring effort while IBM was still contributing left things
unfinished. Some test code disappeared, breaking some unit tests and
subsequent tests. Many tests (about 2 bvt tests and 25 fvt tests) have
been failing for a while. The smoketest in test/ needs to use the
document built in main/smoketestoo_native during the build instead of
its private copy (but the test/ directory is designed to be separate
from the build - should it build the document using Java functions
instead?). Also these Java qa tests seem new and some are unfinished -
were they ported from somewhere (smoketest?)?

Damjan

On Tue, Mar 29, 2016 at 5:53 AM, Patricia Shanahan <pa...@acm.org> wrote:
> I have a few simple changes I would like to check in. Before doing that, I
> would normally run regression tests against my working copy. In any case, as
> one of the few people who are building on Windows, I should test early, test
> often.
>
> What tests do people normally run to check that changes do not have
> unintended consequences?
>
> Thanks,
>
> Patricia
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
> For additional commands, e-mail: dev-help@openoffice.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@openoffice.apache.org
For additional commands, e-mail: dev-help@openoffice.apache.org