You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Mikhail Loenko <ml...@gmail.com> on 2006/01/26 12:38:37 UTC

[testing] code for exotic configurations

Well let's start a new thread as this is more general problem.

So if we have some code designed for some exotic configurations
and we have tests that verify that exotic code.

The test when run in usual configuration (not exotic one) should
report something that would not scare people. But if one
wants to test that specific exotic configuration that he should be
able to easily verify that he successfully made required conf and
the test worked well.

The following options I see here:
1) introduce a new test status (like skipped) to mark those tests that
did not actually run
2) agree on exact wording that the skipped tests would print to allow
grep logs later
3) introduce tests-indicators that would fail when current
configuration disallow
running certain tests

Please let me know what you think

Thanks,
Mikhail Loenko
Intel Middleware Products Division

Re: [testing] code for exotic configurations

Posted by Geir Magnusson Jr <ge...@pobox.com>.

Mikhail Loenko wrote:
> Do you mean that for a single test that verifies 10 lines of code
> working on very specific configuration I have to create a parallel test tree?

The model I had in my head was based on two things - making it easy for 
the casual browser to figure out what is what, and making it simple to 
maintain our infrastructure.

I imagine that we have a big pool of unit tests that work on the boring, 
everyday, non-exotic situations.  For ease of comprehension and use, 
that's in

    test/java/....

so then our test framework can be configure to just run every test it 
finds in there.  With ant driving junit, it's easy - just write and drop 
  in that tree.  it will get run.

Then, if we have odd tests, like "test that can only be run on tuesday 
on sparc when moon is full and I had a cheese sandwhich the day before", 
  we can have a 'odd test' tree

    oddtest/java/....

and then because you have all those conditions to setup (tuesday, 
sandwhich, moon...) you'll have to have a more exotic config for the 
test harness anyway, so you will have to call out that test by name 
anyway in the config, and thus when you are browsing through 
oddtest/java tree, you'll know that in order to undertand the context of 
any test you find in there, you need to look back at the test harness to 
know what's going on.

See what I mean?

> 
> What about tests that work in two different exotic configurations? Should
> we duplicate them?

No.

geir

> 
> Thanks,
> Mikhail
> 
> On 1/26/06, Geir Magnusson Jr <ge...@pobox.com> wrote:
>> one solution is to simply group the "exotic" tests separately from the
>> main tests, so they can be run optionally when you are in that exotic
>> configuration.
>>
>> You can do this in several ways, including a naming convention, or
>> another parallel code tree of the tests...
>>
>> I like the latter, as it makes it easier to "see"
>>
>> geir
>>
>>
>> Mikhail Loenko wrote:
>>> Well let's start a new thread as this is more general problem.
>>>
>>> So if we have some code designed for some exotic configurations
>>> and we have tests that verify that exotic code.
>>>
>>> The test when run in usual configuration (not exotic one) should
>>> report something that would not scare people. But if one
>>> wants to test that specific exotic configuration that he should be
>>> able to easily verify that he successfully made required conf and
>>> the test worked well.
>>>
>>> The following options I see here:
>>> 1) introduce a new test status (like skipped) to mark those tests that
>>> did not actually run
>>> 2) agree on exact wording that the skipped tests would print to allow
>>> grep logs later
>>> 3) introduce tests-indicators that would fail when current
>>> configuration disallow
>>> running certain tests
>>>
>>> Please let me know what you think
>>>
>>> Thanks,
>>> Mikhail Loenko
>>> Intel Middleware Products Division
>>>
>>>
> 
> 

Re: [testing] code for exotic configurations

Posted by Anton Avtamonov <an...@gmail.com>.
> This ties us hard to JUnit.  That's my only worry.  I'd like to also
> keep in consideration how ant, maven, etc can drive this aspect of testing..

Definitely. However 'system-related' scenarious are usually quite
complicated and can hardly perform the required environment settings
(tools installation, platform configuration, etc) from JUnit test. We
should be prepared to have test cases which are only partially in
JUnit format or even not at all. That's why we can benefit from having
separate tree for those tests (and test runners driven by ant or what
ever).
Btw, I agree with a comment regarding the package visibility when
using different trees: more packages - more problems. But I think it
can be easily solved (by using some common place, for instance:
test/common).

--
Anton Avtamonov,
Intel Middleware Products Division

Re: [testing] code for exotic configurations

Posted by Geir Magnusson Jr <ge...@pobox.com>.

George Harley wrote:

> 
> OK. What I am driving at is extending the basic JUnit test runner with a 
> JUnit decorator that could be employed to decide - at runtime - on 
> whether or not certain test cases in the test suite should be run. The 
> decorator could also write out what tests got skipped.

That would be useful.

> 
> How the "run this test/exclude this test" decision is reached could be 
> done in a number of ways. One approach would be to capture the 
> identities of all of the tests you want to skip inside an "exclusions 
> list" text file. The file could also be used to hold information about 
> why each test is being excluded and whatever else might be useful for 
> reporting purposes or perhaps to help with the runtime decision making.

Do both.  Have a inclusion list and an inclusion list, so I can tell you

include :

    org/apache/harmony/foo/bar/*Test

exclude

    **/bar/Certificate*Test.java

or ish.

> 
> As discussed a while ago, using the existing JUnit capabilities like 
> the  junit.extensions.TestDecorator class means that we have the 
> flexibility to run *the same* test suite with whatever exclusions we 
> like according to whatever configuration we want to put in place. All 
> that would change is the text file that we use as input to the decorator.

This ties us hard to JUnit.  That's my only worry.  I'd like to also 
keep in consideration how ant, maven, etc can drive this aspect of testing..

> 
> I have some code that does just this. I hope to be able to contribute it 
> very soon now.

Cool.

> 
> 
>>  
>>>> Thanks,
>>>> Mikhail
>>>>
>>>>
>>>>
>>>>      
>>>>> Alternatively, they could be
>>>>> included as part of a general test suite but be purposely skipped 
>>>>> over at
>>>>> test execution time using a
>>>>> test exclusion list understood by the test runner.
>>>>>
>>>>>
>>>>> Best regards,
>>>>> George
>>>>> ________________________________________
>>>>> George C. Harley
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Tim Ellison <t....@gmail.com>
>>>>> 27/01/2006 08:53
>>>>> Please respond to
>>>>> harmony-dev@incubator.apache.org
>>>>>
>>>>>
>>>>> To
>>>>> harmony-dev@incubator.apache.org
>>>>> cc
>>>>>
>>>>> Subject
>>>>> Re: [testing] code for exotic configurations
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Anton Avtamonov wrote:
>>>>>
>>>>>        
>>>>>>> Note that I could create my own provider and test with it, but 
>>>>>>> what I
>>>>>>>
>>>>>>>             
>>>>> would
>>>>>
>>>>>        
>>>>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>>>>> AlgorithmParameters from real provider as well as how my other 
>>>>>>> classes
>>>>>>>
>>>>>>>             
>>>>> work
>>>>>
>>>>>        
>>>>>>> with real implementations of crypto Engines.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mikhail.
>>>>>>>
>>>>>>>
>>>>>>>             
>>>>>> Hi Mikhail,
>>>>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>>>>> developer-level. Each unit test is intended to test just a limited
>>>>>> piece of functionality separately from other sub-systems (test for 
>>>>>> one
>>>>>> fucntion, test for one class, etc). Such tests must create a desired
>>>>>> environment over the testing fucntionality and run the scenario in 
>>>>>> the
>>>>>> predefined conditions. Unit tests usually able to cover all scenarios
>>>>>> (execution paths) for the tested parts of fucntionality.
>>>>>>
>>>>>> What are you talking about looks like 'system' testing. Such tests
>>>>>> usually run on the real environment and test the most often 
>>>>>> scenarious
>>>>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>>>>> testing is not concentrated on the particular fucntionality, but
>>>>>> covers the work of the whole system.
>>>>>> A sample is: "run some demo application on some particular platform,
>>>>>> with some particular providers installed and perform some 
>>>>>> operations".
>>>>>>
>>>>>> I think currently we should focus on 'unit' test approach since it is
>>>>>> more applicable during the development (so my advise is to revert 
>>>>>> your
>>>>>> tests to install 'test' providers with the desired behavior as George
>>>>>> proposed).
>>>>>> However we should think about 'system' scenarios which can be run on
>>>>>> the later stage and act as 'verification' of proper work of the 
>>>>>> entire
>>>>>> system.
>>>>>>
>>>>>>           
>>>>> I agree with all this.  The unit tests are one style of test for
>>>>> establishing the correctness of the code.  As you point out the unit
>>>>> tests typically require a well-defined environment in which to run, 
>>>>> and
>>>>> it becomes a judgment-call as to whether a particular test's
>>>>> environmental requirements are 'reasonable' or not.
>>>>>
>>>>> For example, you can reasonably expect all developers to have an
>>>>> environment to run unit tests that has enough RAM and a writable disk
>>>>> etc. such that if those things do not exist the tests will simply 
>>>>> fail.
>>>>>  However, you may decide it is unreasonable to expect the 
>>>>> environment to
>>>>> include a populated LDAP server, or a carefully configured RMI server.
>>>>> If you were to call that environment unreasonable then testing JNDI 
>>>>> and
>>>>> RMI would likely involve mock objects etc. to get good unit tests.
>>>>>
>>>>> Of course, as you point out, once you are passing the unit tests you
>>>>> also need the 'system' tests to ensure the code works in a real
>>>>> environment.  Usage scenarios based on the bigger system are good, 
>>>>> as is
>>>>> running the bigger system's test suite on our runtime.
>>>>>
>>>>> Regards,
>>>>> Tim
>>>>>
>>>>>
>>>>>
>>>>>        
>>>>>> -- 
>>>>>> Anton Avtamonov,
>>>>>> Intel Middleware Products Division
>>>>>>
>>>>>>
>>>>>>           
>>>>> -- 
>>>>>
>>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>>> IBM Java technology centre, UK.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>         
>>>>       
>>>     
>>
>>   
> 
> 

Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
On 1/30/06, Anton Avtamonov <an...@gmail.com> wrote:
> On 1/29/06, Mikhail Loenko <ml...@gmail.com> wrote:
> > P.S. There are also 'integration' tests that verify how different 'units'
> > work together.
> >
> > I think the areas of various tests intersect and it would be good to define
> > them on this early stage
> >
>
> Well, agree. And I think our discussion is exactly about to define
> what is right unit, integration and system testing.
>
> Talking about how real unit test should be prepared I'd like to give
> some example:

I like examples :)


>
> There is a method java.awt.Toolkit.getDefaultToolkit().getScreenSize()
> which returns the current screen size.
> Let's assume there is some functionality which works differently
> depending on actual screen size. You are going to test that
> functionality.
> The first approach (as I understand that is what you are using) is to:
> - if the screen resolution is as expected - do testing
> - otherwise skip the test.
>
> Another approach (which I think to be more appropriate) is:
> - override getScreenSize() method of some 'TestToolkit' and install it
> as 'default toolkit' in the test setUp, so that it returns the desired
> value for the particular test.
> This gives me an ability to write all necessary tests and cover all
> the paths of my functionality which depend on screen size.

Well, getScreenSize() returns a number, and it is possible to loop
through valid (and some invalid but possible) values and test
with them.

But what if the method returns an AlgorithmParameters *implementation* or
a certificate in encoded form (which is byte array)?

Thanks,
Mikhail.

>
> Note that I'm not going to re-create all other classes of JSEE :-). I
> need to control getScreenSize() only and have to create the required
> mock-object to achieve that.
>
> --
> Anton Avtamonov,
> Intel Middleware Products Division
>

Re: [testing] code for exotic configurations

Posted by Anton Avtamonov <an...@gmail.com>.
On 1/29/06, Mikhail Loenko <ml...@gmail.com> wrote:
> P.S. There are also 'integration' tests that verify how different 'units'
> work together.
>
> I think the areas of various tests intersect and it would be good to define
> them on this early stage
>

Well, agree. And I think our discussion is exactly about to define
what is right unit, integration and system testing.

Talking about how real unit test should be prepared I'd like to give
some example:

There is a method java.awt.Toolkit.getDefaultToolkit().getScreenSize()
which returns the current screen size.
Let's assume there is some functionality which works differently
depending on actual screen size. You are going to test that
functionality.
The first approach (as I understand that is what you are using) is to:
- if the screen resolution is as expected - do testing
- otherwise skip the test.

Another approach (which I think to be more appropriate) is:
- override getScreenSize() method of some 'TestToolkit' and install it
as 'default toolkit' in the test setUp, so that it returns the desired
value for the particular test.
This gives me an ability to write all necessary tests and cover all
the paths of my functionality which depend on screen size.

Note that I'm not going to re-create all other classes of JSEE :-). I
need to control getScreenSize() only and have to create the required
mock-object to achieve that.

--
Anton Avtamonov,
Intel Middleware Products Division

Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
P.S. There are also 'integration' tests that verify how different 'units'
work together.

I think the areas of various tests intersect and it would be good to define
them on this early stage

Thanks,
Mikhail

On 1/29/06, Mikhail Loenko <ml...@gmail.com> wrote:
> Hi George,
>
> another example: implementation of SecureRandom might depend on
> file:/dev/random
>
> So, depending on OS, the bytes produced by my implementation of
> SecureRandom might be not too random or too secure.
>
> How do you consider  the test that verifies behavior of *my* code
> that depends on OS? Is it unit or system?
>
> Thanks,
> Mikhail
>
> On 1/29/06, George Harley <ge...@googlemail.com> wrote:
> > Hi Mikhail,
> >
> > Comments inlined below.
> >
> > Best regards,
> > George
> > IBM UK
> >
> >
> > Mikhail Loenko wrote:
> > > Hi Anton,
> > >
> > > On 1/28/06, Mikhail Loenko <ml...@gmail.com> wrote:
> > >
> > >> On 1/27/06, Anton Avtamonov <an...@gmail.com> wrote:
> > >>
> > >>> Sorry for being away during the major part of your discussion. Hope
> > >>> I'm still not too late.
> > >>>
> > >>> As I can see currently we have only one 'exotic' situation - some
> > >>> tests which are based on providers and use so many provider's
> > >>> fucntionality that cannot be replaced with mock objects in a
> > >>> reasonable time.
> > >>>
> > >
> > > You have only one example, not one situation - there is a difference... <g>
> > >
> >
> > Would you like to give us some more examples ?
> >
> > > Harmony has a modular structure and final product may be assembled from
> > > the modules we can not even think of yet. And the only thing we know about
> > > those modules is that they follow the spec.
> > >
> > > You likely know, security module that does not have any provider or any
> > > algorithm implementation follows the spec :). How do you think, will
> > > java.util or java.crypto work without those implementations? No.
> > >
> > > So, following your logic, all unit tests should either do not rely on
> > > other modules
> > > or include their own implementation of those modules (and better - full J2SE
> > > implementation :).
> >
> > That was not my interpretation of what Anton wrote. What do you consider
> > is actually being tested by a unit test ? Is it *just your code* or is
> > it *your code plus the system around it* ? I am really keen to hear your
> > answer as I think it lies at the very heart of this discussion.
> >
> >
> > >  Tests that do not include their own J2SE are system ones,
> > > as they rely on environment.
> > >
> >
> > Could you elaborate a bit more on this point please ?
> >
> > > What I think is we have to be ready that some functionality would be unusable
> > > or untestable in some situations and our tests/framework would better
> > > be ready for it.
> > >
> >
> > That is a system test issue not a unit test issue.
> >
> > > Thanks,
> > > Mikhail
> > >
> > >
> >
> >
>

Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
Hi George,

another example: implementation of SecureRandom might depend on
file:/dev/random

So, depending on OS, the bytes produced by my implementation of
SecureRandom might be not too random or too secure.

How do you consider  the test that verifies behavior of *my* code
that depends on OS? Is it unit or system?

Thanks,
Mikhail

On 1/29/06, George Harley <ge...@googlemail.com> wrote:
> Hi Mikhail,
>
> Comments inlined below.
>
> Best regards,
> George
> IBM UK
>
>
> Mikhail Loenko wrote:
> > Hi Anton,
> >
> > On 1/28/06, Mikhail Loenko <ml...@gmail.com> wrote:
> >
> >> On 1/27/06, Anton Avtamonov <an...@gmail.com> wrote:
> >>
> >>> Sorry for being away during the major part of your discussion. Hope
> >>> I'm still not too late.
> >>>
> >>> As I can see currently we have only one 'exotic' situation - some
> >>> tests which are based on providers and use so many provider's
> >>> fucntionality that cannot be replaced with mock objects in a
> >>> reasonable time.
> >>>
> >
> > You have only one example, not one situation - there is a difference... <g>
> >
>
> Would you like to give us some more examples ?
>
> > Harmony has a modular structure and final product may be assembled from
> > the modules we can not even think of yet. And the only thing we know about
> > those modules is that they follow the spec.
> >
> > You likely know, security module that does not have any provider or any
> > algorithm implementation follows the spec :). How do you think, will
> > java.util or java.crypto work without those implementations? No.
> >
> > So, following your logic, all unit tests should either do not rely on
> > other modules
> > or include their own implementation of those modules (and better - full J2SE
> > implementation :).
>
> That was not my interpretation of what Anton wrote. What do you consider
> is actually being tested by a unit test ? Is it *just your code* or is
> it *your code plus the system around it* ? I am really keen to hear your
> answer as I think it lies at the very heart of this discussion.
>
>
> >  Tests that do not include their own J2SE are system ones,
> > as they rely on environment.
> >
>
> Could you elaborate a bit more on this point please ?
>
> > What I think is we have to be ready that some functionality would be unusable
> > or untestable in some situations and our tests/framework would better
> > be ready for it.
> >
>
> That is a system test issue not a unit test issue.
>
> > Thanks,
> > Mikhail
> >
> >
>
>

Re: [testing] code for exotic configurations

Posted by George Harley <ge...@googlemail.com>.
Hi Mikhail,

Comments inlined below.

Best regards,
George
IBM UK


Mikhail Loenko wrote:
> Hi Anton,
>
> On 1/28/06, Mikhail Loenko <ml...@gmail.com> wrote:
>   
>> On 1/27/06, Anton Avtamonov <an...@gmail.com> wrote:
>>     
>>> Sorry for being away during the major part of your discussion. Hope
>>> I'm still not too late.
>>>
>>> As I can see currently we have only one 'exotic' situation - some
>>> tests which are based on providers and use so many provider's
>>> fucntionality that cannot be replaced with mock objects in a
>>> reasonable time.
>>>       
>
> You have only one example, not one situation - there is a difference... <g>
>   

Would you like to give us some more examples ?

> Harmony has a modular structure and final product may be assembled from
> the modules we can not even think of yet. And the only thing we know about
> those modules is that they follow the spec.
>
> You likely know, security module that does not have any provider or any
> algorithm implementation follows the spec :). How do you think, will
> java.util or java.crypto work without those implementations? No.
>
> So, following your logic, all unit tests should either do not rely on
> other modules
> or include their own implementation of those modules (and better - full J2SE
> implementation :).

That was not my interpretation of what Anton wrote. What do you consider 
is actually being tested by a unit test ? Is it *just your code* or is 
it *your code plus the system around it* ? I am really keen to hear your 
answer as I think it lies at the very heart of this discussion.


>  Tests that do not include their own J2SE are system ones,
> as they rely on environment.
>   

Could you elaborate a bit more on this point please ?

> What I think is we have to be ready that some functionality would be unusable
> or untestable in some situations and our tests/framework would better
> be ready for it.
>   

That is a system test issue not a unit test issue.

> Thanks,
> Mikhail
>
>   


Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
Hi Anton,

On 1/28/06, Mikhail Loenko <ml...@gmail.com> wrote:
> On 1/27/06, Anton Avtamonov <an...@gmail.com> wrote:
> > Sorry for being away during the major part of your discussion. Hope
> > I'm still not too late.
> >
> > As I can see currently we have only one 'exotic' situation - some
> > tests which are based on providers and use so many provider's
> > fucntionality that cannot be replaced with mock objects in a
> > reasonable time.

You have only one example, not one situation - there is a difference... <g>

Harmony has a modular structure and final product may be assembled from
the modules we can not even think of yet. And the only thing we know about
those modules is that they follow the spec.

You likely know, security module that does not have any provider or any
algorithm implementation follows the spec :). How do you think, will
java.util or java.crypto work without those implementations? No.

So, following your logic, all unit tests should either do not rely on
other modules
or include their own implementation of those modules (and better - full J2SE
implementation :). Tests that do not include their own J2SE are system ones,
as they rely on environment.

What I think is we have to be ready that some functionality would be unusable
or untestable in some situations and our tests/framework would better
be ready for it.

Thanks,
Mikhail

Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
On 1/27/06, Anton Avtamonov <an...@gmail.com> wrote:
> Sorry for being away during the major part of your discussion. Hope
> I'm still not too late.
>
> As I can see currently we have only one 'exotic' situation - some
> tests which are based on providers and use so many provider's
> fucntionality that cannot be replaced with mock objects in a
> reasonable time. Since such tests look like 'system' (use real
> environment) I believe the idea of having separate trees is better.
> Something like:
>
> test/unit
> and
> test/system.

As I wrote earlier in this thread that cause duplication of tests:
we have hundreds tests that might serve both as 'unit' and 'system'


>
> It will be clear that *all* unit tests are run regularly during the
> development and don't need any specifical
> settings/logging/filtering/etc.
>
> --
> Anton Avtamonov,
> Intel Middleware Products Division
>

Re: [testing] code for exotic configurations

Posted by Anton Avtamonov <an...@gmail.com>.
Sorry for being away during the major part of your discussion. Hope
I'm still not too late.

As I can see currently we have only one 'exotic' situation - some
tests which are based on providers and use so many provider's
fucntionality that cannot be replaced with mock objects in a
reasonable time. Since such tests look like 'system' (use real
environment) I believe the idea of having separate trees is better.
Something like:

test/unit
and
test/system.

It will be clear that *all* unit tests are run regularly during the
development and don't need any specifical
settings/logging/filtering/etc.

--
Anton Avtamonov,
Intel Middleware Products Division

Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
OK, let's see your contribution.

Meanwhile I've gone through the security2 unit tests; there are some of them
that *can not* do verification in some 'exotic' configurations, I've made them
reporting failure in this case.

And there are a few tests that *can* verify in some 'exotic' configuration only.
But this configuration is possible in only case that is legacy
providers are installed
in the system. Normally these tests do not verify anything. I've made this tests
reporting pass if configuration is not 'exotic'

So I'm going to return to both these types of tests when we have framework
good enough to work with things like we discussed

Thanks,
Mikhail.

On 1/27/06, George Harley <ge...@googlemail.com> wrote:
> Hi Mikhail,
>
> Comments inlined below ...
>
> Best regards,
> George
> IBM UK
>
>
> Mikhail Loenko wrote:
> > Hi George,
> >
> > On 1/27/06, George Harley <ge...@googlemail.com> wrote:
> >
> >> Sure. And it should be the role of the test framework to inform users
> >> about which tests got run, passed, failed, got skipped etc.
> >>
> >
> > see below
> >
> >
> >>> It would not disturb most of the people because the test will pass in 'bad'
> >>> environment. But those, who know about these tests will sometimes grep
> >>> logs to validate configuration.
> >>>
> >>>
> >> Why use logs to tell us information that the unit test framework can already
> >> provide ? Who wants to have to grep through log files when the test runner
> >> can provide us with what we need ?
> >>
> >
> > Sorry but I did not quite understand. I do not see anything related to
> > the 'skipped' status in the junit.framework.TestResult
> >
> > If there is such a status then it would be excellent, it is the option #1 in the
> > first mail in the thread
> >
> > Thanks,
> > Mikhail
> >
>
> OK. What I am driving at is extending the basic JUnit test runner with a
> JUnit decorator that could be employed to decide - at runtime - on
> whether or not certain test cases in the test suite should be run. The
> decorator could also write out what tests got skipped.
>
> How the "run this test/exclude this test" decision is reached could be
> done in a number of ways. One approach would be to capture the
> identities of all of the tests you want to skip inside an "exclusions
> list" text file. The file could also be used to hold information about
> why each test is being excluded and whatever else might be useful for
> reporting purposes or perhaps to help with the runtime decision making.
>
> As discussed a while ago, using the existing JUnit capabilities like
> the  junit.extensions.TestDecorator class means that we have the
> flexibility to run *the same* test suite with whatever exclusions we
> like according to whatever configuration we want to put in place. All
> that would change is the text file that we use as input to the decorator.
>
> I have some code that does just this. I hope to be able to contribute it
> very soon now.
>
>
> >
> >>> Thanks,
> >>> Mikhail
> >>>
> >>>
> >>>
> >>>
> >>>> Alternatively, they could be
> >>>> included as part of a general test suite but be purposely skipped over at
> >>>> test execution time using a
> >>>> test exclusion list understood by the test runner.
> >>>>
> >>>>
> >>>> Best regards,
> >>>> George
> >>>> ________________________________________
> >>>> George C. Harley
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Tim Ellison <t....@gmail.com>
> >>>> 27/01/2006 08:53
> >>>> Please respond to
> >>>> harmony-dev@incubator.apache.org
> >>>>
> >>>>
> >>>> To
> >>>> harmony-dev@incubator.apache.org
> >>>> cc
> >>>>
> >>>> Subject
> >>>> Re: [testing] code for exotic configurations
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Anton Avtamonov wrote:
> >>>>
> >>>>
> >>>>>> Note that I could create my own provider and test with it, but what I
> >>>>>>
> >>>>>>
> >>>> would
> >>>>
> >>>>
> >>>>>> really want is to test how my EncryptedPrivateKeyInfo works with
> >>>>>> AlgorithmParameters from real provider as well as how my other classes
> >>>>>>
> >>>>>>
> >>>> work
> >>>>
> >>>>
> >>>>>> with real implementations of crypto Engines.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Mikhail.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>> Hi Mikhail,
> >>>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
> >>>>> developer-level. Each unit test is intended to test just a limited
> >>>>> piece of functionality separately from other sub-systems (test for one
> >>>>> fucntion, test for one class, etc). Such tests must create a desired
> >>>>> environment over the testing fucntionality and run the scenario in the
> >>>>> predefined conditions. Unit tests usually able to cover all scenarios
> >>>>> (execution paths) for the tested parts of fucntionality.
> >>>>>
> >>>>> What are you talking about looks like 'system' testing. Such tests
> >>>>> usually run on the real environment and test the most often scenarious
> >>>>> (the reduntant set, all scenarios usually cannot be covered). Such
> >>>>> testing is not concentrated on the particular fucntionality, but
> >>>>> covers the work of the whole system.
> >>>>> A sample is: "run some demo application on some particular platform,
> >>>>> with some particular providers installed and perform some operations".
> >>>>>
> >>>>> I think currently we should focus on 'unit' test approach since it is
> >>>>> more applicable during the development (so my advise is to revert your
> >>>>> tests to install 'test' providers with the desired behavior as George
> >>>>> proposed).
> >>>>> However we should think about 'system' scenarios which can be run on
> >>>>> the later stage and act as 'verification' of proper work of the entire
> >>>>> system.
> >>>>>
> >>>>>
> >>>> I agree with all this.  The unit tests are one style of test for
> >>>> establishing the correctness of the code.  As you point out the unit
> >>>> tests typically require a well-defined environment in which to run, and
> >>>> it becomes a judgment-call as to whether a particular test's
> >>>> environmental requirements are 'reasonable' or not.
> >>>>
> >>>> For example, you can reasonably expect all developers to have an
> >>>> environment to run unit tests that has enough RAM and a writable disk
> >>>> etc. such that if those things do not exist the tests will simply fail.
> >>>>  However, you may decide it is unreasonable to expect the environment to
> >>>> include a populated LDAP server, or a carefully configured RMI server.
> >>>> If you were to call that environment unreasonable then testing JNDI and
> >>>> RMI would likely involve mock objects etc. to get good unit tests.
> >>>>
> >>>> Of course, as you point out, once you are passing the unit tests you
> >>>> also need the 'system' tests to ensure the code works in a real
> >>>> environment.  Usage scenarios based on the bigger system are good, as is
> >>>> running the bigger system's test suite on our runtime.
> >>>>
> >>>> Regards,
> >>>> Tim
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> --
> >>>>> Anton Avtamonov,
> >>>>> Intel Middleware Products Division
> >>>>>
> >>>>>
> >>>>>
> >>>> --
> >>>>
> >>>> Tim Ellison (t.p.ellison@gmail.com)
> >>>> IBM Java technology centre, UK.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >
> >
>
>

Re: [testing] code for exotic configurations

Posted by George Harley <ge...@googlemail.com>.
Hi Mikhail,

Comments inlined below ...

Best regards,
George
IBM UK


Mikhail Loenko wrote:
> Hi George,
>
> On 1/27/06, George Harley <ge...@googlemail.com> wrote:
>   
>> Sure. And it should be the role of the test framework to inform users
>> about which tests got run, passed, failed, got skipped etc.
>>     
>
> see below
>
>   
>>> It would not disturb most of the people because the test will pass in 'bad'
>>> environment. But those, who know about these tests will sometimes grep
>>> logs to validate configuration.
>>>
>>>       
>> Why use logs to tell us information that the unit test framework can already
>> provide ? Who wants to have to grep through log files when the test runner
>> can provide us with what we need ?
>>     
>
> Sorry but I did not quite understand. I do not see anything related to
> the 'skipped' status in the junit.framework.TestResult
>
> If there is such a status then it would be excellent, it is the option #1 in the
> first mail in the thread
>
> Thanks,
> Mikhail
>   

OK. What I am driving at is extending the basic JUnit test runner with a 
JUnit decorator that could be employed to decide - at runtime - on 
whether or not certain test cases in the test suite should be run. The 
decorator could also write out what tests got skipped.

How the "run this test/exclude this test" decision is reached could be 
done in a number of ways. One approach would be to capture the 
identities of all of the tests you want to skip inside an "exclusions 
list" text file. The file could also be used to hold information about 
why each test is being excluded and whatever else might be useful for 
reporting purposes or perhaps to help with the runtime decision making.

As discussed a while ago, using the existing JUnit capabilities like 
the  junit.extensions.TestDecorator class means that we have the 
flexibility to run *the same* test suite with whatever exclusions we 
like according to whatever configuration we want to put in place. All 
that would change is the text file that we use as input to the decorator.

I have some code that does just this. I hope to be able to contribute it 
very soon now.


>   
>>> Thanks,
>>> Mikhail
>>>
>>>
>>>
>>>       
>>>> Alternatively, they could be
>>>> included as part of a general test suite but be purposely skipped over at
>>>> test execution time using a
>>>> test exclusion list understood by the test runner.
>>>>
>>>>
>>>> Best regards,
>>>> George
>>>> ________________________________________
>>>> George C. Harley
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Tim Ellison <t....@gmail.com>
>>>> 27/01/2006 08:53
>>>> Please respond to
>>>> harmony-dev@incubator.apache.org
>>>>
>>>>
>>>> To
>>>> harmony-dev@incubator.apache.org
>>>> cc
>>>>
>>>> Subject
>>>> Re: [testing] code for exotic configurations
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Anton Avtamonov wrote:
>>>>
>>>>         
>>>>>> Note that I could create my own provider and test with it, but what I
>>>>>>
>>>>>>             
>>>> would
>>>>
>>>>         
>>>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>>>> AlgorithmParameters from real provider as well as how my other classes
>>>>>>
>>>>>>             
>>>> work
>>>>
>>>>         
>>>>>> with real implementations of crypto Engines.
>>>>>>
>>>>>> Thanks,
>>>>>> Mikhail.
>>>>>>
>>>>>>
>>>>>>             
>>>>> Hi Mikhail,
>>>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>>>> developer-level. Each unit test is intended to test just a limited
>>>>> piece of functionality separately from other sub-systems (test for one
>>>>> fucntion, test for one class, etc). Such tests must create a desired
>>>>> environment over the testing fucntionality and run the scenario in the
>>>>> predefined conditions. Unit tests usually able to cover all scenarios
>>>>> (execution paths) for the tested parts of fucntionality.
>>>>>
>>>>> What are you talking about looks like 'system' testing. Such tests
>>>>> usually run on the real environment and test the most often scenarious
>>>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>>>> testing is not concentrated on the particular fucntionality, but
>>>>> covers the work of the whole system.
>>>>> A sample is: "run some demo application on some particular platform,
>>>>> with some particular providers installed and perform some operations".
>>>>>
>>>>> I think currently we should focus on 'unit' test approach since it is
>>>>> more applicable during the development (so my advise is to revert your
>>>>> tests to install 'test' providers with the desired behavior as George
>>>>> proposed).
>>>>> However we should think about 'system' scenarios which can be run on
>>>>> the later stage and act as 'verification' of proper work of the entire
>>>>> system.
>>>>>
>>>>>           
>>>> I agree with all this.  The unit tests are one style of test for
>>>> establishing the correctness of the code.  As you point out the unit
>>>> tests typically require a well-defined environment in which to run, and
>>>> it becomes a judgment-call as to whether a particular test's
>>>> environmental requirements are 'reasonable' or not.
>>>>
>>>> For example, you can reasonably expect all developers to have an
>>>> environment to run unit tests that has enough RAM and a writable disk
>>>> etc. such that if those things do not exist the tests will simply fail.
>>>>  However, you may decide it is unreasonable to expect the environment to
>>>> include a populated LDAP server, or a carefully configured RMI server.
>>>> If you were to call that environment unreasonable then testing JNDI and
>>>> RMI would likely involve mock objects etc. to get good unit tests.
>>>>
>>>> Of course, as you point out, once you are passing the unit tests you
>>>> also need the 'system' tests to ensure the code works in a real
>>>> environment.  Usage scenarios based on the bigger system are good, as is
>>>> running the bigger system's test suite on our runtime.
>>>>
>>>> Regards,
>>>> Tim
>>>>
>>>>
>>>>
>>>>         
>>>>> --
>>>>> Anton Avtamonov,
>>>>> Intel Middleware Products Division
>>>>>
>>>>>
>>>>>           
>>>> --
>>>>
>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>> IBM Java technology centre, UK.
>>>>
>>>>
>>>>
>>>>
>>>>         
>>>       
>>     
>
>   


Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
Hi George,

On 1/27/06, George Harley <ge...@googlemail.com> wrote:
> Sure. And it should be the role of the test framework to inform users
> about which tests got run, passed, failed, got skipped etc.

see below

> > It would not disturb most of the people because the test will pass in 'bad'
> > environment. But those, who know about these tests will sometimes grep
> > logs to validate configuration.
> >
> Why use logs to tell us information that the unit test framework can already
> provide ? Who wants to have to grep through log files when the test runner
> can provide us with what we need ?

Sorry but I did not quite understand. I do not see anything related to
the 'skipped' status in the junit.framework.TestResult

If there is such a status then it would be excellent, it is the option #1 in the
first mail in the thread

Thanks,
Mikhail

>
> > Thanks,
> > Mikhail
> >
> >
> >
> >> Alternatively, they could be
> >> included as part of a general test suite but be purposely skipped over at
> >> test execution time using a
> >> test exclusion list understood by the test runner.
> >>
> >>
> >> Best regards,
> >> George
> >> ________________________________________
> >> George C. Harley
> >>
> >>
> >>
> >>
> >>
> >> Tim Ellison <t....@gmail.com>
> >> 27/01/2006 08:53
> >> Please respond to
> >> harmony-dev@incubator.apache.org
> >>
> >>
> >> To
> >> harmony-dev@incubator.apache.org
> >> cc
> >>
> >> Subject
> >> Re: [testing] code for exotic configurations
> >>
> >>
> >>
> >>
> >>
> >>
> >> Anton Avtamonov wrote:
> >>
> >>>> Note that I could create my own provider and test with it, but what I
> >>>>
> >> would
> >>
> >>>> really want is to test how my EncryptedPrivateKeyInfo works with
> >>>> AlgorithmParameters from real provider as well as how my other classes
> >>>>
> >> work
> >>
> >>>> with real implementations of crypto Engines.
> >>>>
> >>>> Thanks,
> >>>> Mikhail.
> >>>>
> >>>>
> >>> Hi Mikhail,
> >>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
> >>> developer-level. Each unit test is intended to test just a limited
> >>> piece of functionality separately from other sub-systems (test for one
> >>> fucntion, test for one class, etc). Such tests must create a desired
> >>> environment over the testing fucntionality and run the scenario in the
> >>> predefined conditions. Unit tests usually able to cover all scenarios
> >>> (execution paths) for the tested parts of fucntionality.
> >>>
> >>> What are you talking about looks like 'system' testing. Such tests
> >>> usually run on the real environment and test the most often scenarious
> >>> (the reduntant set, all scenarios usually cannot be covered). Such
> >>> testing is not concentrated on the particular fucntionality, but
> >>> covers the work of the whole system.
> >>> A sample is: "run some demo application on some particular platform,
> >>> with some particular providers installed and perform some operations".
> >>>
> >>> I think currently we should focus on 'unit' test approach since it is
> >>> more applicable during the development (so my advise is to revert your
> >>> tests to install 'test' providers with the desired behavior as George
> >>> proposed).
> >>> However we should think about 'system' scenarios which can be run on
> >>> the later stage and act as 'verification' of proper work of the entire
> >>> system.
> >>>
> >> I agree with all this.  The unit tests are one style of test for
> >> establishing the correctness of the code.  As you point out the unit
> >> tests typically require a well-defined environment in which to run, and
> >> it becomes a judgment-call as to whether a particular test's
> >> environmental requirements are 'reasonable' or not.
> >>
> >> For example, you can reasonably expect all developers to have an
> >> environment to run unit tests that has enough RAM and a writable disk
> >> etc. such that if those things do not exist the tests will simply fail.
> >>  However, you may decide it is unreasonable to expect the environment to
> >> include a populated LDAP server, or a carefully configured RMI server.
> >> If you were to call that environment unreasonable then testing JNDI and
> >> RMI would likely involve mock objects etc. to get good unit tests.
> >>
> >> Of course, as you point out, once you are passing the unit tests you
> >> also need the 'system' tests to ensure the code works in a real
> >> environment.  Usage scenarios based on the bigger system are good, as is
> >> running the bigger system's test suite on our runtime.
> >>
> >> Regards,
> >> Tim
> >>
> >>
> >>
> >>> --
> >>> Anton Avtamonov,
> >>> Intel Middleware Products Division
> >>>
> >>>
> >> --
> >>
> >> Tim Ellison (t.p.ellison@gmail.com)
> >> IBM Java technology centre, UK.
> >>
> >>
> >>
> >>
> >
> >
>
>

Re: [testing] code for exotic configurations

Posted by George Harley <ge...@googlemail.com>.
Hi Mikhail,

None of my test messages to the list from this account seem to have reached
their destination. I'll keep my fingers crossed for this response though...


Mikhail Loenko wrote:
> On 1/27/06, George Harley1 <GH...@uk.ibm.com> wrote:
>   
>> But because we live in a less than ideal world there will, no doubt, be
>> some tests that will demand an
>> environment that is impossible or at the very least difficult to mock up
>> for the majority of developers/testers.
>>     
>
> I'm absolutely agree that we are neither living in the ideal world nor trying
> to make it ideal :)
>   

C'mon. We are all trying to make it ideal. :-)

> So until we got a 'system' test suite why should we weaken existing tests?
>
>   
Uh ? I don't think that is what I was proposing.

>> One solution could be to segregate those tests into a separate test suite
>> (available for all but primarily
>> for those working in the niche area that demands the special environment).
>>     
>
> Moving this kind of tests would affect many people: they will see
> separate suites,
> try, ask questions...
>   
It would affect people in as much as they would see a number of different
test suites that could be potentially run. Some test suites would need some
more configuration up-front than others. The information about that would
be captured either in text files alongside the test code and/or in other
project documentation residing elsewhere (Harmony web pages, wiki pages 
etc).
It is a communication thing.

> If the test can be configured by a few people only who works on that
> specific area and those people are aware of those tests why not just
> print a log when the test is skipped?
>   
Sure. And it should be the role of the test framework to inform users
about which tests got run, passed, failed, got skipped etc.

> It would not disturb most of the people because the test will pass in 'bad'
> environment. But those, who know about these tests will sometimes grep
> logs to validate configuration.
>   
Why use logs to tell us information that the unit test framework can already
provide ? Who wants to have to grep through log files when the test runner
can provide us with what we need ?

> Thanks,
> Mikhail
>
>
>   
>> Alternatively, they could be
>> included as part of a general test suite but be purposely skipped over at
>> test execution time using a
>> test exclusion list understood by the test runner.
>>
>>
>> Best regards,
>> George
>> ________________________________________
>> George C. Harley
>>
>>
>>
>>
>>
>> Tim Ellison <t....@gmail.com>
>> 27/01/2006 08:53
>> Please respond to
>> harmony-dev@incubator.apache.org
>>
>>
>> To
>> harmony-dev@incubator.apache.org
>> cc
>>
>> Subject
>> Re: [testing] code for exotic configurations
>>
>>
>>
>>
>>
>>
>> Anton Avtamonov wrote:
>>     
>>>> Note that I could create my own provider and test with it, but what I
>>>>         
>> would
>>     
>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>> AlgorithmParameters from real provider as well as how my other classes
>>>>         
>> work
>>     
>>>> with real implementations of crypto Engines.
>>>>
>>>> Thanks,
>>>> Mikhail.
>>>>
>>>>         
>>> Hi Mikhail,
>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>> developer-level. Each unit test is intended to test just a limited
>>> piece of functionality separately from other sub-systems (test for one
>>> fucntion, test for one class, etc). Such tests must create a desired
>>> environment over the testing fucntionality and run the scenario in the
>>> predefined conditions. Unit tests usually able to cover all scenarios
>>> (execution paths) for the tested parts of fucntionality.
>>>
>>> What are you talking about looks like 'system' testing. Such tests
>>> usually run on the real environment and test the most often scenarious
>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>> testing is not concentrated on the particular fucntionality, but
>>> covers the work of the whole system.
>>> A sample is: "run some demo application on some particular platform,
>>> with some particular providers installed and perform some operations".
>>>
>>> I think currently we should focus on 'unit' test approach since it is
>>> more applicable during the development (so my advise is to revert your
>>> tests to install 'test' providers with the desired behavior as George
>>> proposed).
>>> However we should think about 'system' scenarios which can be run on
>>> the later stage and act as 'verification' of proper work of the entire
>>> system.
>>>       
>> I agree with all this.  The unit tests are one style of test for
>> establishing the correctness of the code.  As you point out the unit
>> tests typically require a well-defined environment in which to run, and
>> it becomes a judgment-call as to whether a particular test's
>> environmental requirements are 'reasonable' or not.
>>
>> For example, you can reasonably expect all developers to have an
>> environment to run unit tests that has enough RAM and a writable disk
>> etc. such that if those things do not exist the tests will simply fail.
>>  However, you may decide it is unreasonable to expect the environment to
>> include a populated LDAP server, or a carefully configured RMI server.
>> If you were to call that environment unreasonable then testing JNDI and
>> RMI would likely involve mock objects etc. to get good unit tests.
>>
>> Of course, as you point out, once you are passing the unit tests you
>> also need the 'system' tests to ensure the code works in a real
>> environment.  Usage scenarios based on the bigger system are good, as is
>> running the bigger system's test suite on our runtime.
>>
>> Regards,
>> Tim
>>
>>
>>     
>>> --
>>> Anton Avtamonov,
>>> Intel Middleware Products Division
>>>
>>>       
>> --
>>
>> Tim Ellison (t.p.ellison@gmail.com)
>> IBM Java technology centre, UK.
>>
>>
>>
>>     
>
>   


Re: [testing] code for exotic configurations

Posted by Geir Magnusson Jr <ge...@pobox.com>.

George Harley wrote:
> Hi Geir,
> 
>  > This is why I advocate making a "separate tree" for the system tests 
> - make it clear that they are not the general unit tests...
> 
> Right. But rather than being a separate directory tree in the source 
> repository that separate tree could be realised as a logical JUnit test 
> suite grouping.

for certain cases, yes.  Please don't extrapolate to the absurd - I 
wouldn't advocate one directory tree per test suite.

What I don't want to have is that any testing, especially unit testing, 
requires configuration burden when not ncessary.

For example, I'm guessing that the majority of unit tests need nothing 
more than being dropped into the test tree and that should be that.  No 
AllTests.java needed, nothing in build.xml, etc.

Simply put, I want to avoid  maintenance of simple unit tests going 
beyond the test code itself.  More complicated things can require 
whatever they need, but those more complicated things shouldn't have any 
impact on the standard "light and get away"* unit tests.

> 
> So, for instance, say we have a bunch of "general" test cases. We would 
> add them into an executable JUnit test suite called say, 
> TestSuiteGeneral. When TestSuiteGeneral gets run then only the test 
> cases that we specifically added to that suite are executed. There is no 
> "skipping" of tests because we only ran those tests that we had 
> previously configured (in the TestSuiteGeneral code) to run.

Sure - as long as when I add a test, it just works.  I don't want to 
modify anything else when I drop a new testcase into the tree.  There's 
no reason to.

I might have subsections to save time - like BeansTestSuite that would 
be groups of unit tests that are appropriate for beans (for example), 
and then I can see having the extra step (but it would be nice to find a 
way around that as well...  I suspect that pcakage visibility 
considerations will limit our freedom there, tho.

> 
> Any "special" tests that either need specialist configuration or else 
> should not be run by just anyone could be programatically grouped into a 
> separate executable JUnit test suite. When that suite gets run it *only* 
> executes our group of special tests.

Yep

> 
> Just like we can let Junit tell us about what passed, failed etc, so we 
> can use existing JUnit  practices to put our test cases into whatever 
> runtime grouping we like. That's clean and simple.

Yep

> 
> Best regards,
> George
> 
> 
> 
> Geir Magnusson Jr wrote:
>>
>>
>> Mikhail Loenko wrote:
>>> On 1/27/06, George Harley1 <GH...@uk.ibm.com> wrote:
>>>> But because we live in a less than ideal world there will, no doubt, be
>>>> some tests that will demand an
>>>> environment that is impossible or at the very least difficult to 
>>>> mock up
>>>> for the majority of developers/testers.
>>>
>>> I'm absolutely agree that we are neither living in the ideal world 
>>> nor trying
>>> to make it ideal :)
>>>
>>> So until we got a 'system' test suite why should we weaken existing 
>>> tests?
>>>
>>>> One solution could be to segregate those tests into a separate test 
>>>> suite
>>>> (available for all but primarily
>>>> for those working in the niche area that demands the special 
>>>> environment).
>>>
>>> Moving this kind of tests would affect many people: they will see
>>> separate suites,
>>> try, ask questions...
>>>
>>> If the test can be configured by a few people only who works on that
>>> specific area and those people are aware of those tests why not just
>>> print a log when the test is skipped?
>>
>> Because the same set of people that will be bothered by separate 
>> suites will have the same reaction to skipped tests.
>>
>> This is why I advocate making a "separate tree" for the system tests - 
>> make it clear that they are not the general unit tests... Geir 
>> Magnusson Jr wrote:
>>>
>>>
>>> Mikhail Loenko wrote:
>>>> On 1/27/06, George Harley1 <GH...@uk.ibm.com> wrote:
>>>>> But because we live in a less than ideal world there will, no 
>>>>> doubt, be
>>>>> some tests that will demand an
>>>>> environment that is impossible or at the very least difficult to 
>>>>> mock up
>>>>> for the majority of developers/testers.
>>>>
>>>> I'm absolutely agree that we are neither living in the ideal world 
>>>> nor trying
>>>> to make it ideal :)
>>>>
>>>> So until we got a 'system' test suite why should we weaken existing 
>>>> tests?
>>>>
>>>>> One solution could be to segregate those tests into a separate test 
>>>>> suite
>>>>> (available for all but primarily
>>>>> for those working in the niche area that demands the special 
>>>>> environment).
>>>>
>>>> Moving this kind of tests would affect many people: they will see
>>>> separate suites,
>>>> try, ask questions...
>>>>
>>>> If the test can be configured by a few people only who works on that
>>>> specific area and those people are aware of those tests why not just
>>>> print a log when the test is skipped?
>>>
>>> Because the same set of people that will be bothered by separate 
>>> suites will have the same reaction to skipped tests.
>>>
>>> This is why I advocate making a "separate tree" for the system tests 
>>> - make it clear that they are not the general unit tests...
>>>
>>>>
>>>> It would not disturb most of the people because the test will pass 
>>>> in 'bad'
>>>> environment. But those, who know about these tests will sometimes grep
>>>> logs to validate configuration.
>>>
>>> IMO, there's too much special information there, too much config.  
>>> I'm a simple person, and like things clean and simple.  I don't like 
>>> to mix concerns when possible, and here's a place where it's 
>>> definitely possible to separate cleanly.
>>>
>>> I don't see the downside.
>>>
>>> geir
>>>
>>>>
>>>> Thanks,
>>>> Mikhail
>>>>
>>>>
>>>>> Alternatively, they could be
>>>>> included as part of a general test suite but be purposely skipped 
>>>>> over at
>>>>> test execution time using a
>>>>> test exclusion list understood by the test runner.
>>>>>
>>>>>
>>>>> Best regards,
>>>>> George
>>>>> ________________________________________
>>>>> George C. Harley
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Tim Ellison <t....@gmail.com>
>>>>> 27/01/2006 08:53
>>>>> Please respond to
>>>>> harmony-dev@incubator.apache.org
>>>>>
>>>>>
>>>>> To
>>>>> harmony-dev@incubator.apache.org
>>>>> cc
>>>>>
>>>>> Subject
>>>>> Re: [testing] code for exotic configurations
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Anton Avtamonov wrote:
>>>>>>> Note that I could create my own provider and test with it, but 
>>>>>>> what I
>>>>> would
>>>>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>>>>> AlgorithmParameters from real provider as well as how my other 
>>>>>>> classes
>>>>> work
>>>>>>> with real implementations of crypto Engines.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mikhail.
>>>>>>>
>>>>>> Hi Mikhail,
>>>>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>>>>> developer-level. Each unit test is intended to test just a limited
>>>>>> piece of functionality separately from other sub-systems (test for 
>>>>>> one
>>>>>> fucntion, test for one class, etc). Such tests must create a desired
>>>>>> environment over the testing fucntionality and run the scenario in 
>>>>>> the
>>>>>> predefined conditions. Unit tests usually able to cover all scenarios
>>>>>> (execution paths) for the tested parts of fucntionality.
>>>>>>
>>>>>> What are you talking about looks like 'system' testing. Such tests
>>>>>> usually run on the real environment and test the most often 
>>>>>> scenarious
>>>>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>>>>> testing is not concentrated on the particular fucntionality, but
>>>>>> covers the work of the whole system.
>>>>>> A sample is: "run some demo application on some particular platform,
>>>>>> with some particular providers installed and perform some 
>>>>>> operations".
>>>>>>
>>>>>> I think currently we should focus on 'unit' test approach since it is
>>>>>> more applicable during the development (so my advise is to revert 
>>>>>> your
>>>>>> tests to install 'test' providers with the desired behavior as George
>>>>>> proposed).
>>>>>> However we should think about 'system' scenarios which can be run on
>>>>>> the later stage and act as 'verification' of proper work of the 
>>>>>> entire
>>>>>> system.
>>>>> I agree with all this.  The unit tests are one style of test for
>>>>> establishing the correctness of the code.  As you point out the unit
>>>>> tests typically require a well-defined environment in which to run, 
>>>>> and
>>>>> it becomes a judgment-call as to whether a particular test's
>>>>> environmental requirements are 'reasonable' or not.
>>>>>
>>>>> For example, you can reasonably expect all developers to have an
>>>>> environment to run unit tests that has enough RAM and a writable disk
>>>>> etc. such that if those things do not exist the tests will simply 
>>>>> fail.
>>>>>  However, you may decide it is unreasonable to expect the 
>>>>> environment to
>>>>> include a populated LDAP server, or a carefully configured RMI server.
>>>>> If you were to call that environment unreasonable then testing JNDI 
>>>>> and
>>>>> RMI would likely involve mock objects etc. to get good unit tests.
>>>>>
>>>>> Of course, as you point out, once you are passing the unit tests you
>>>>> also need the 'system' tests to ensure the code works in a real
>>>>> environment.  Usage scenarios based on the bigger system are good, 
>>>>> as is
>>>>> running the bigger system's test suite on our runtime.
>>>>>
>>>>> Regards,
>>>>> Tim
>>>>>
>>>>>
>>>>>> -- 
>>>>>> Anton Avtamonov,
>>>>>> Intel Middleware Products Division
>>>>>>
>>>>> -- 
>>>>>
>>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>>> IBM Java technology centre, UK.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
> 
>>
>>>
>>> It would not disturb most of the people because the test will pass in 
>>> 'bad'
>>> environment. But those, who know about these tests will sometimes grep
>>> logs to validate configuration.
>>
>> IMO, there's too much special information there, too much config.  I'm 
>> a simple person, and like things clean and simple.  I don't like to 
>> mix concerns when possible, and here's a place where it's definitely 
>> possible to separate cleanly.
>>
>> I don't see the downside.
>>
>> geir
>> Geir Magnusson Jr wrote:
>>>
>>>
>>> Mikhail Loenko wrote:
>>>> On 1/27/06, George Harley1 <GH...@uk.ibm.com> wrote:
>>>>> But because we live in a less than ideal world there will, no 
>>>>> doubt, be
>>>>> some tests that will demand an
>>>>> environment that is impossible or at the very least difficult to 
>>>>> mock up
>>>>> for the majority of developers/testers.
>>>>
>>>> I'm absolutely agree that we are neither living in the ideal world 
>>>> nor trying
>>>> to make it ideal :)
>>>>
>>>> So until we got a 'system' test suite why should we weaken existing 
>>>> tests?
>>>>
>>>>> One solution could be to segregate those tests into a separate test 
>>>>> suite
>>>>> (available for all but primarily
>>>>> for those working in the niche area that demands the special 
>>>>> environment).
>>>>
>>>> Moving this kind of tests would affect many people: they will see
>>>> separate suites,
>>>> try, ask questions...
>>>>
>>>> If the test can be configured by a few people only who works on that
>>>> specific area and those people are aware of those tests why not just
>>>> print a log when the test is skipped?
>>>
>>> Because the same set of people that will be bothered by separate 
>>> suites will have the same reaction to skipped tests.
>>>
>>> This is why I advocate making a "separate tree" for the system tests 
>>> - make it clear that they are not the general unit tests...
>>>
>>>>
>>>> It would not disturb most of the people because the test will pass 
>>>> in 'bad'
>>>> environment. But those, who know about these tests will sometimes grep
>>>> logs to validate configuration.
>>>
>>> IMO, there's too much special information there, too much config.  
>>> I'm a simple person, and like things clean and simple.  I don't like 
>>> to mix concerns when possible, and here's a place where it's 
>>> definitely possible to separate cleanly.
>>>
>>> I don't see the downside.
>>>
>>> geir
>>>
>>>>
>>>> Thanks,
>>>> Mikhail
>>>>
>>>>
>>>>> Alternatively, they could be
>>>>> included as part of a general test suite but be purposely skipped 
>>>>> over at
>>>>> test execution time using a
>>>>> test exclusion list understood by the test runner.
>>>>>
>>>>>
>>>>> Best regards,
>>>>> George
>>>>> ________________________________________
>>>>> George C. Harley
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Tim Ellison <t....@gmail.com>
>>>>> 27/01/2006 08:53
>>>>> Please respond to
>>>>> harmony-dev@incubator.apache.org
>>>>>
>>>>>
>>>>> To
>>>>> harmony-dev@incubator.apache.org
>>>>> cc
>>>>>
>>>>> Subject
>>>>> Re: [testing] code for exotic configurations
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Anton Avtamonov wrote:
>>>>>>> Note that I could create my own provider and test with it, but 
>>>>>>> what I
>>>>> would
>>>>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>>>>> AlgorithmParameters from real provider as well as how my other 
>>>>>>> classes
>>>>> work
>>>>>>> with real implementations of crypto Engines.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Mikhail.
>>>>>>>
>>>>>> Hi Mikhail,
>>>>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>>>>> developer-level. Each unit test is intended to test just a limited
>>>>>> piece of functionality separately from other sub-systems (test for 
>>>>>> one
>>>>>> fucntion, test for one class, etc). Such tests must create a desired
>>>>>> environment over the testing fucntionality and run the scenario in 
>>>>>> the
>>>>>> predefined conditions. Unit tests usually able to cover all scenarios
>>>>>> (execution paths) for the tested parts of fucntionality.
>>>>>>
>>>>>> What are you talking about looks like 'system' testing. Such tests
>>>>>> usually run on the real environment and test the most often 
>>>>>> scenarious
>>>>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>>>>> testing is not concentrated on the particular fucntionality, but
>>>>>> covers the work of the whole system.
>>>>>> A sample is: "run some demo application on some particular platform,
>>>>>> with some particular providers installed and perform some 
>>>>>> operations".
>>>>>>
>>>>>> I think currently we should focus on 'unit' test approach since it is
>>>>>> more applicable during the development (so my advise is to revert 
>>>>>> your
>>>>>> tests to install 'test' providers with the desired behavior as George
>>>>>> proposed).
>>>>>> However we should think about 'system' scenarios which can be run on
>>>>>> the later stage and act as 'verification' of proper work of the 
>>>>>> entire
>>>>>> system.
>>>>> I agree with all this.  The unit tests are one style of test for
>>>>> establishing the correctness of the code.  As you point out the unit
>>>>> tests typically require a well-defined environment in which to run, 
>>>>> and
>>>>> it becomes a judgment-call as to whether a particular test's
>>>>> environmental requirements are 'reasonable' or not.
>>>>>
>>>>> For example, you can reasonably expect all developers to have an
>>>>> environment to run unit tests that has enough RAM and a writable disk
>>>>> etc. such that if those things do not exist the tests will simply 
>>>>> fail.
>>>>>  However, you may decide it is unreasonable to expect the 
>>>>> environment to
>>>>> include a populated LDAP server, or a carefully configured RMI server.
>>>>> If you were to call that environment unreasonable then testing JNDI 
>>>>> and
>>>>> RMI would likely involve mock objects etc. to get good unit tests.
>>>>>
>>>>> Of course, as you point out, once you are passing the unit tests you
>>>>> also need the 'system' tests to ensure the code works in a real
>>>>> environment.  Usage scenarios based on the bigger system are good, 
>>>>> as is
>>>>> running the bigger system's test suite on our runtime.
>>>>>
>>>>> Regards,
>>>>> Tim
>>>>>
>>>>>
>>>>>> -- 
>>>>>> Anton Avtamonov,
>>>>>> Intel Middleware Products Division
>>>>>>
>>>>> -- 
>>>>>
>>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>>> IBM Java technology centre, UK.
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
> 
>>>
>>> Thanks,
>>> Mikhail
>>>
>>>
>>>> Alternatively, they could be
>>>> included as part of a general test suite but be purposely skipped 
>>>> over at
>>>> test execution time using a
>>>> test exclusion list understood by the test runner.
>>>>
>>>>
>>>> Best regards,
>>>> George
>>>> ________________________________________
>>>> George C. Harley
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Tim Ellison <t....@gmail.com>
>>>> 27/01/2006 08:53
>>>> Please respond to
>>>> harmony-dev@incubator.apache.org
>>>>
>>>>
>>>> To
>>>> harmony-dev@incubator.apache.org
>>>> cc
>>>>
>>>> Subject
>>>> Re: [testing] code for exotic configurations
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Anton Avtamonov wrote:
>>>>>> Note that I could create my own provider and test with it, but what I
>>>> would
>>>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>>>> AlgorithmParameters from real provider as well as how my other 
>>>>>> classes
>>>> work
>>>>>> with real implementations of crypto Engines.
>>>>>>
>>>>>> Thanks,
>>>>>> Mikhail.
>>>>>>
>>>>> Hi Mikhail,
>>>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>>>> developer-level. Each unit test is intended to test just a limited
>>>>> piece of functionality separately from other sub-systems (test for one
>>>>> fucntion, test for one class, etc). Such tests must create a desired
>>>>> environment over the testing fucntionality and run the scenario in the
>>>>> predefined conditions. Unit tests usually able to cover all scenarios
>>>>> (execution paths) for the tested parts of fucntionality.
>>>>>
>>>>> What are you talking about looks like 'system' testing. Such tests
>>>>> usually run on the real environment and test the most often scenarious
>>>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>>>> testing is not concentrated on the particular fucntionality, but
>>>>> covers the work of the whole system.
>>>>> A sample is: "run some demo application on some particular platform,
>>>>> with some particular providers installed and perform some operations".
>>>>>
>>>>> I think currently we should focus on 'unit' test approach since it is
>>>>> more applicable during the development (so my advise is to revert your
>>>>> tests to install 'test' providers with the desired behavior as George
>>>>> proposed).
>>>>> However we should think about 'system' scenarios which can be run on
>>>>> the later stage and act as 'verification' of proper work of the entire
>>>>> system.
>>>> I agree with all this.  The unit tests are one style of test for
>>>> establishing the correctness of the code.  As you point out the unit
>>>> tests typically require a well-defined environment in which to run, and
>>>> it becomes a judgment-call as to whether a particular test's
>>>> environmental requirements are 'reasonable' or not.
>>>>
>>>> For example, you can reasonably expect all developers to have an
>>>> environment to run unit tests that has enough RAM and a writable disk
>>>> etc. such that if those things do not exist the tests will simply fail.
>>>>  However, you may decide it is unreasonable to expect the 
>>>> environment to
>>>> include a populated LDAP server, or a carefully configured RMI server.
>>>> If you were to call that environment unreasonable then testing JNDI and
>>>> RMI would likely involve mock objects etc. to get good unit tests.
>>>>
>>>> Of course, as you point out, once you are passing the unit tests you
>>>> also need the 'system' tests to ensure the code works in a real
>>>> environment.  Usage scenarios based on the bigger system are good, 
>>>> as is
>>>> running the bigger system's test suite on our runtime.
>>>>
>>>> Regards,
>>>> Tim
>>>>
>>>>
>>>>> -- 
>>>>> Anton Avtamonov,
>>>>> Intel Middleware Products Division
>>>>>
>>>> -- 
>>>>
>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>> IBM Java technology centre, UK.
>>>>
>>>>
>>>>
>>>
>>>
>>
> 
> 


Re: [testing] code for exotic configurations

Posted by George Harley <ge...@googlemail.com>.
Hi Geir,

 > This is why I advocate making a "separate tree" for the system tests 
- make it clear that they are not the general unit tests...

Right. But rather than being a separate directory tree in the source 
repository that separate tree could be realised as a logical JUnit test 
suite grouping.

So, for instance, say we have a bunch of "general" test cases. We would 
add them into an executable JUnit test suite called say, 
TestSuiteGeneral. When TestSuiteGeneral gets run then only the test 
cases that we specifically added to that suite are executed. There is no 
"skipping" of tests because we only ran those tests that we had 
previously configured (in the TestSuiteGeneral code) to run.

Any "special" tests that either need specialist configuration or else 
should not be run by just anyone could be programatically grouped into a 
separate executable JUnit test suite. When that suite gets run it *only* 
executes our group of special tests.

Just like we can let Junit tell us about what passed, failed etc, so we 
can use existing JUnit  practices to put our test cases into whatever 
runtime grouping we like. That's clean and simple.

Best regards,
George



Geir Magnusson Jr wrote:
>
>
> Mikhail Loenko wrote:
>> On 1/27/06, George Harley1 <GH...@uk.ibm.com> wrote:
>>> But because we live in a less than ideal world there will, no doubt, be
>>> some tests that will demand an
>>> environment that is impossible or at the very least difficult to 
>>> mock up
>>> for the majority of developers/testers.
>>
>> I'm absolutely agree that we are neither living in the ideal world 
>> nor trying
>> to make it ideal :)
>>
>> So until we got a 'system' test suite why should we weaken existing 
>> tests?
>>
>>> One solution could be to segregate those tests into a separate test 
>>> suite
>>> (available for all but primarily
>>> for those working in the niche area that demands the special 
>>> environment).
>>
>> Moving this kind of tests would affect many people: they will see
>> separate suites,
>> try, ask questions...
>>
>> If the test can be configured by a few people only who works on that
>> specific area and those people are aware of those tests why not just
>> print a log when the test is skipped?
>
> Because the same set of people that will be bothered by separate 
> suites will have the same reaction to skipped tests.
>
> This is why I advocate making a "separate tree" for the system tests - 
> make it clear that they are not the general unit tests... Geir 
> Magnusson Jr wrote:
>>
>>
>> Mikhail Loenko wrote:
>>> On 1/27/06, George Harley1 <GH...@uk.ibm.com> wrote:
>>>> But because we live in a less than ideal world there will, no 
>>>> doubt, be
>>>> some tests that will demand an
>>>> environment that is impossible or at the very least difficult to 
>>>> mock up
>>>> for the majority of developers/testers.
>>>
>>> I'm absolutely agree that we are neither living in the ideal world 
>>> nor trying
>>> to make it ideal :)
>>>
>>> So until we got a 'system' test suite why should we weaken existing 
>>> tests?
>>>
>>>> One solution could be to segregate those tests into a separate test 
>>>> suite
>>>> (available for all but primarily
>>>> for those working in the niche area that demands the special 
>>>> environment).
>>>
>>> Moving this kind of tests would affect many people: they will see
>>> separate suites,
>>> try, ask questions...
>>>
>>> If the test can be configured by a few people only who works on that
>>> specific area and those people are aware of those tests why not just
>>> print a log when the test is skipped?
>>
>> Because the same set of people that will be bothered by separate 
>> suites will have the same reaction to skipped tests.
>>
>> This is why I advocate making a "separate tree" for the system tests 
>> - make it clear that they are not the general unit tests...
>>
>>>
>>> It would not disturb most of the people because the test will pass 
>>> in 'bad'
>>> environment. But those, who know about these tests will sometimes grep
>>> logs to validate configuration.
>>
>> IMO, there's too much special information there, too much config.  
>> I'm a simple person, and like things clean and simple.  I don't like 
>> to mix concerns when possible, and here's a place where it's 
>> definitely possible to separate cleanly.
>>
>> I don't see the downside.
>>
>> geir
>>
>>>
>>> Thanks,
>>> Mikhail
>>>
>>>
>>>> Alternatively, they could be
>>>> included as part of a general test suite but be purposely skipped 
>>>> over at
>>>> test execution time using a
>>>> test exclusion list understood by the test runner.
>>>>
>>>>
>>>> Best regards,
>>>> George
>>>> ________________________________________
>>>> George C. Harley
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Tim Ellison <t....@gmail.com>
>>>> 27/01/2006 08:53
>>>> Please respond to
>>>> harmony-dev@incubator.apache.org
>>>>
>>>>
>>>> To
>>>> harmony-dev@incubator.apache.org
>>>> cc
>>>>
>>>> Subject
>>>> Re: [testing] code for exotic configurations
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Anton Avtamonov wrote:
>>>>>> Note that I could create my own provider and test with it, but 
>>>>>> what I
>>>> would
>>>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>>>> AlgorithmParameters from real provider as well as how my other 
>>>>>> classes
>>>> work
>>>>>> with real implementations of crypto Engines.
>>>>>>
>>>>>> Thanks,
>>>>>> Mikhail.
>>>>>>
>>>>> Hi Mikhail,
>>>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>>>> developer-level. Each unit test is intended to test just a limited
>>>>> piece of functionality separately from other sub-systems (test for 
>>>>> one
>>>>> fucntion, test for one class, etc). Such tests must create a desired
>>>>> environment over the testing fucntionality and run the scenario in 
>>>>> the
>>>>> predefined conditions. Unit tests usually able to cover all scenarios
>>>>> (execution paths) for the tested parts of fucntionality.
>>>>>
>>>>> What are you talking about looks like 'system' testing. Such tests
>>>>> usually run on the real environment and test the most often 
>>>>> scenarious
>>>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>>>> testing is not concentrated on the particular fucntionality, but
>>>>> covers the work of the whole system.
>>>>> A sample is: "run some demo application on some particular platform,
>>>>> with some particular providers installed and perform some 
>>>>> operations".
>>>>>
>>>>> I think currently we should focus on 'unit' test approach since it is
>>>>> more applicable during the development (so my advise is to revert 
>>>>> your
>>>>> tests to install 'test' providers with the desired behavior as George
>>>>> proposed).
>>>>> However we should think about 'system' scenarios which can be run on
>>>>> the later stage and act as 'verification' of proper work of the 
>>>>> entire
>>>>> system.
>>>> I agree with all this.  The unit tests are one style of test for
>>>> establishing the correctness of the code.  As you point out the unit
>>>> tests typically require a well-defined environment in which to run, 
>>>> and
>>>> it becomes a judgment-call as to whether a particular test's
>>>> environmental requirements are 'reasonable' or not.
>>>>
>>>> For example, you can reasonably expect all developers to have an
>>>> environment to run unit tests that has enough RAM and a writable disk
>>>> etc. such that if those things do not exist the tests will simply 
>>>> fail.
>>>>  However, you may decide it is unreasonable to expect the 
>>>> environment to
>>>> include a populated LDAP server, or a carefully configured RMI server.
>>>> If you were to call that environment unreasonable then testing JNDI 
>>>> and
>>>> RMI would likely involve mock objects etc. to get good unit tests.
>>>>
>>>> Of course, as you point out, once you are passing the unit tests you
>>>> also need the 'system' tests to ensure the code works in a real
>>>> environment.  Usage scenarios based on the bigger system are good, 
>>>> as is
>>>> running the bigger system's test suite on our runtime.
>>>>
>>>> Regards,
>>>> Tim
>>>>
>>>>
>>>>> -- 
>>>>> Anton Avtamonov,
>>>>> Intel Middleware Products Division
>>>>>
>>>> -- 
>>>>
>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>> IBM Java technology centre, UK.
>>>>
>>>>
>>>>
>>>
>>>
>>
>

>
>>
>> It would not disturb most of the people because the test will pass in 
>> 'bad'
>> environment. But those, who know about these tests will sometimes grep
>> logs to validate configuration.
>
> IMO, there's too much special information there, too much config.  I'm 
> a simple person, and like things clean and simple.  I don't like to 
> mix concerns when possible, and here's a place where it's definitely 
> possible to separate cleanly.
>
> I don't see the downside.
>
> geir
> Geir Magnusson Jr wrote:
>>
>>
>> Mikhail Loenko wrote:
>>> On 1/27/06, George Harley1 <GH...@uk.ibm.com> wrote:
>>>> But because we live in a less than ideal world there will, no 
>>>> doubt, be
>>>> some tests that will demand an
>>>> environment that is impossible or at the very least difficult to 
>>>> mock up
>>>> for the majority of developers/testers.
>>>
>>> I'm absolutely agree that we are neither living in the ideal world 
>>> nor trying
>>> to make it ideal :)
>>>
>>> So until we got a 'system' test suite why should we weaken existing 
>>> tests?
>>>
>>>> One solution could be to segregate those tests into a separate test 
>>>> suite
>>>> (available for all but primarily
>>>> for those working in the niche area that demands the special 
>>>> environment).
>>>
>>> Moving this kind of tests would affect many people: they will see
>>> separate suites,
>>> try, ask questions...
>>>
>>> If the test can be configured by a few people only who works on that
>>> specific area and those people are aware of those tests why not just
>>> print a log when the test is skipped?
>>
>> Because the same set of people that will be bothered by separate 
>> suites will have the same reaction to skipped tests.
>>
>> This is why I advocate making a "separate tree" for the system tests 
>> - make it clear that they are not the general unit tests...
>>
>>>
>>> It would not disturb most of the people because the test will pass 
>>> in 'bad'
>>> environment. But those, who know about these tests will sometimes grep
>>> logs to validate configuration.
>>
>> IMO, there's too much special information there, too much config.  
>> I'm a simple person, and like things clean and simple.  I don't like 
>> to mix concerns when possible, and here's a place where it's 
>> definitely possible to separate cleanly.
>>
>> I don't see the downside.
>>
>> geir
>>
>>>
>>> Thanks,
>>> Mikhail
>>>
>>>
>>>> Alternatively, they could be
>>>> included as part of a general test suite but be purposely skipped 
>>>> over at
>>>> test execution time using a
>>>> test exclusion list understood by the test runner.
>>>>
>>>>
>>>> Best regards,
>>>> George
>>>> ________________________________________
>>>> George C. Harley
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Tim Ellison <t....@gmail.com>
>>>> 27/01/2006 08:53
>>>> Please respond to
>>>> harmony-dev@incubator.apache.org
>>>>
>>>>
>>>> To
>>>> harmony-dev@incubator.apache.org
>>>> cc
>>>>
>>>> Subject
>>>> Re: [testing] code for exotic configurations
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Anton Avtamonov wrote:
>>>>>> Note that I could create my own provider and test with it, but 
>>>>>> what I
>>>> would
>>>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>>>> AlgorithmParameters from real provider as well as how my other 
>>>>>> classes
>>>> work
>>>>>> with real implementations of crypto Engines.
>>>>>>
>>>>>> Thanks,
>>>>>> Mikhail.
>>>>>>
>>>>> Hi Mikhail,
>>>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>>>> developer-level. Each unit test is intended to test just a limited
>>>>> piece of functionality separately from other sub-systems (test for 
>>>>> one
>>>>> fucntion, test for one class, etc). Such tests must create a desired
>>>>> environment over the testing fucntionality and run the scenario in 
>>>>> the
>>>>> predefined conditions. Unit tests usually able to cover all scenarios
>>>>> (execution paths) for the tested parts of fucntionality.
>>>>>
>>>>> What are you talking about looks like 'system' testing. Such tests
>>>>> usually run on the real environment and test the most often 
>>>>> scenarious
>>>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>>>> testing is not concentrated on the particular fucntionality, but
>>>>> covers the work of the whole system.
>>>>> A sample is: "run some demo application on some particular platform,
>>>>> with some particular providers installed and perform some 
>>>>> operations".
>>>>>
>>>>> I think currently we should focus on 'unit' test approach since it is
>>>>> more applicable during the development (so my advise is to revert 
>>>>> your
>>>>> tests to install 'test' providers with the desired behavior as George
>>>>> proposed).
>>>>> However we should think about 'system' scenarios which can be run on
>>>>> the later stage and act as 'verification' of proper work of the 
>>>>> entire
>>>>> system.
>>>> I agree with all this.  The unit tests are one style of test for
>>>> establishing the correctness of the code.  As you point out the unit
>>>> tests typically require a well-defined environment in which to run, 
>>>> and
>>>> it becomes a judgment-call as to whether a particular test's
>>>> environmental requirements are 'reasonable' or not.
>>>>
>>>> For example, you can reasonably expect all developers to have an
>>>> environment to run unit tests that has enough RAM and a writable disk
>>>> etc. such that if those things do not exist the tests will simply 
>>>> fail.
>>>>  However, you may decide it is unreasonable to expect the 
>>>> environment to
>>>> include a populated LDAP server, or a carefully configured RMI server.
>>>> If you were to call that environment unreasonable then testing JNDI 
>>>> and
>>>> RMI would likely involve mock objects etc. to get good unit tests.
>>>>
>>>> Of course, as you point out, once you are passing the unit tests you
>>>> also need the 'system' tests to ensure the code works in a real
>>>> environment.  Usage scenarios based on the bigger system are good, 
>>>> as is
>>>> running the bigger system's test suite on our runtime.
>>>>
>>>> Regards,
>>>> Tim
>>>>
>>>>
>>>>> -- 
>>>>> Anton Avtamonov,
>>>>> Intel Middleware Products Division
>>>>>
>>>> -- 
>>>>
>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>> IBM Java technology centre, UK.
>>>>
>>>>
>>>>
>>>
>>>
>>
>

>>
>> Thanks,
>> Mikhail
>>
>>
>>> Alternatively, they could be
>>> included as part of a general test suite but be purposely skipped 
>>> over at
>>> test execution time using a
>>> test exclusion list understood by the test runner.
>>>
>>>
>>> Best regards,
>>> George
>>> ________________________________________
>>> George C. Harley
>>>
>>>
>>>
>>>
>>>
>>> Tim Ellison <t....@gmail.com>
>>> 27/01/2006 08:53
>>> Please respond to
>>> harmony-dev@incubator.apache.org
>>>
>>>
>>> To
>>> harmony-dev@incubator.apache.org
>>> cc
>>>
>>> Subject
>>> Re: [testing] code for exotic configurations
>>>
>>>
>>>
>>>
>>>
>>>
>>> Anton Avtamonov wrote:
>>>>> Note that I could create my own provider and test with it, but what I
>>> would
>>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>>> AlgorithmParameters from real provider as well as how my other 
>>>>> classes
>>> work
>>>>> with real implementations of crypto Engines.
>>>>>
>>>>> Thanks,
>>>>> Mikhail.
>>>>>
>>>> Hi Mikhail,
>>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>>> developer-level. Each unit test is intended to test just a limited
>>>> piece of functionality separately from other sub-systems (test for one
>>>> fucntion, test for one class, etc). Such tests must create a desired
>>>> environment over the testing fucntionality and run the scenario in the
>>>> predefined conditions. Unit tests usually able to cover all scenarios
>>>> (execution paths) for the tested parts of fucntionality.
>>>>
>>>> What are you talking about looks like 'system' testing. Such tests
>>>> usually run on the real environment and test the most often scenarious
>>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>>> testing is not concentrated on the particular fucntionality, but
>>>> covers the work of the whole system.
>>>> A sample is: "run some demo application on some particular platform,
>>>> with some particular providers installed and perform some operations".
>>>>
>>>> I think currently we should focus on 'unit' test approach since it is
>>>> more applicable during the development (so my advise is to revert your
>>>> tests to install 'test' providers with the desired behavior as George
>>>> proposed).
>>>> However we should think about 'system' scenarios which can be run on
>>>> the later stage and act as 'verification' of proper work of the entire
>>>> system.
>>> I agree with all this.  The unit tests are one style of test for
>>> establishing the correctness of the code.  As you point out the unit
>>> tests typically require a well-defined environment in which to run, and
>>> it becomes a judgment-call as to whether a particular test's
>>> environmental requirements are 'reasonable' or not.
>>>
>>> For example, you can reasonably expect all developers to have an
>>> environment to run unit tests that has enough RAM and a writable disk
>>> etc. such that if those things do not exist the tests will simply fail.
>>>  However, you may decide it is unreasonable to expect the 
>>> environment to
>>> include a populated LDAP server, or a carefully configured RMI server.
>>> If you were to call that environment unreasonable then testing JNDI and
>>> RMI would likely involve mock objects etc. to get good unit tests.
>>>
>>> Of course, as you point out, once you are passing the unit tests you
>>> also need the 'system' tests to ensure the code works in a real
>>> environment.  Usage scenarios based on the bigger system are good, 
>>> as is
>>> running the bigger system's test suite on our runtime.
>>>
>>> Regards,
>>> Tim
>>>
>>>
>>>> -- 
>>>> Anton Avtamonov,
>>>> Intel Middleware Products Division
>>>>
>>> -- 
>>>
>>> Tim Ellison (t.p.ellison@gmail.com)
>>> IBM Java technology centre, UK.
>>>
>>>
>>>
>>
>>
>


Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
On 1/27/06, Geir Magnusson Jr <ge...@pobox.com> wrote:
>
>
> Mikhail Loenko wrote:
> > On 1/27/06, Geir Magnusson Jr <ge...@pobox.com> wrote:
> >>> If the test can be configured by a few people only who works on that
> >>> specific area and those people are aware of those tests why not just
> >>> print a log when the test is skipped?
> >> Because the same set of people that will be bothered by separate suites
> >> will have the same reaction to skipped tests.
> >
> > By 'skipped' in this context I meant that the test did not verify the code but
> > reported the 'passed' status. That set of people will not even notice
> > these tests.
>
> I don't think you want that either.  I think you want the test to not be
> run.

Sorry that my message was not quite clear.

What I did mean is:

The test is always launched. If it detects that the real config does not allow
do all the checks, then the test either returns special status (i.e.
'passed but did
not do all the checks') to the framework or prints a log message like
'warning: the test was skipped'

Then a 'regular user'  will see that the test passed, while 'advanced' one
will be able to check what happen.

Thanks,
Mikhail

>
> We're going to automate this whole thing, and we may want to know when a
> given test broke.  If it's falsely reporting "pass", we'll get confused.
>
> geir
>
> >
> > Thanks,
> > Mikhail
> >
> >> This is why I advocate making a "separate tree" for the system tests -
> >> make it clear that they are not the general unit tests...
> >>
> >>> It would not disturb most of the people because the test will pass in 'bad'
> >>> environment. But those, who know about these tests will sometimes grep
> >>> logs to validate configuration.
> >> IMO, there's too much special information there, too much config.  I'm a
> >> simple person, and like things clean and simple.  I don't like to mix
> >> concerns when possible, and here's a place where it's definitely
> >> possible to separate cleanly.
> >>
> >> I don't see the downside.
> >>
> >> geir
> >>
> >>> Thanks,
> >>> Mikhail
> >>>
> >>>
> >>>> Alternatively, they could be
> >>>> included as part of a general test suite but be purposely skipped over at
> >>>> test execution time using a
> >>>> test exclusion list understood by the test runner.
> >>>>
> >>>>
> >>>> Best regards,
> >>>> George
> >>>> ________________________________________
> >>>> George C. Harley
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Tim Ellison <t....@gmail.com>
> >>>> 27/01/2006 08:53
> >>>> Please respond to
> >>>> harmony-dev@incubator.apache.org
> >>>>
> >>>>
> >>>> To
> >>>> harmony-dev@incubator.apache.org
> >>>> cc
> >>>>
> >>>> Subject
> >>>> Re: [testing] code for exotic configurations
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Anton Avtamonov wrote:
> >>>>>> Note that I could create my own provider and test with it, but what I
> >>>> would
> >>>>>> really want is to test how my EncryptedPrivateKeyInfo works with
> >>>>>> AlgorithmParameters from real provider as well as how my other classes
> >>>> work
> >>>>>> with real implementations of crypto Engines.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Mikhail.
> >>>>>>
> >>>>> Hi Mikhail,
> >>>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
> >>>>> developer-level. Each unit test is intended to test just a limited
> >>>>> piece of functionality separately from other sub-systems (test for one
> >>>>> fucntion, test for one class, etc). Such tests must create a desired
> >>>>> environment over the testing fucntionality and run the scenario in the
> >>>>> predefined conditions. Unit tests usually able to cover all scenarios
> >>>>> (execution paths) for the tested parts of fucntionality.
> >>>>>
> >>>>> What are you talking about looks like 'system' testing. Such tests
> >>>>> usually run on the real environment and test the most often scenarious
> >>>>> (the reduntant set, all scenarios usually cannot be covered). Such
> >>>>> testing is not concentrated on the particular fucntionality, but
> >>>>> covers the work of the whole system.
> >>>>> A sample is: "run some demo application on some particular platform,
> >>>>> with some particular providers installed and perform some operations".
> >>>>>
> >>>>> I think currently we should focus on 'unit' test approach since it is
> >>>>> more applicable during the development (so my advise is to revert your
> >>>>> tests to install 'test' providers with the desired behavior as George
> >>>>> proposed).
> >>>>> However we should think about 'system' scenarios which can be run on
> >>>>> the later stage and act as 'verification' of proper work of the entire
> >>>>> system.
> >>>> I agree with all this.  The unit tests are one style of test for
> >>>> establishing the correctness of the code.  As you point out the unit
> >>>> tests typically require a well-defined environment in which to run, and
> >>>> it becomes a judgment-call as to whether a particular test's
> >>>> environmental requirements are 'reasonable' or not.
> >>>>
> >>>> For example, you can reasonably expect all developers to have an
> >>>> environment to run unit tests that has enough RAM and a writable disk
> >>>> etc. such that if those things do not exist the tests will simply fail.
> >>>>  However, you may decide it is unreasonable to expect the environment to
> >>>> include a populated LDAP server, or a carefully configured RMI server.
> >>>> If you were to call that environment unreasonable then testing JNDI and
> >>>> RMI would likely involve mock objects etc. to get good unit tests.
> >>>>
> >>>> Of course, as you point out, once you are passing the unit tests you
> >>>> also need the 'system' tests to ensure the code works in a real
> >>>> environment.  Usage scenarios based on the bigger system are good, as is
> >>>> running the bigger system's test suite on our runtime.
> >>>>
> >>>> Regards,
> >>>> Tim
> >>>>
> >>>>
> >>>>> --
> >>>>> Anton Avtamonov,
> >>>>> Intel Middleware Products Division
> >>>>>
> >>>> --
> >>>>
> >>>> Tim Ellison (t.p.ellison@gmail.com)
> >>>> IBM Java technology centre, UK.
> >>>>
> >>>>
> >>>>
> >>>
> >
> >
>

Re: [testing] code for exotic configurations

Posted by Geir Magnusson Jr <ge...@pobox.com>.

Mikhail Loenko wrote:
> On 1/27/06, Geir Magnusson Jr <ge...@pobox.com> wrote:
>>> If the test can be configured by a few people only who works on that
>>> specific area and those people are aware of those tests why not just
>>> print a log when the test is skipped?
>> Because the same set of people that will be bothered by separate suites
>> will have the same reaction to skipped tests.
> 
> By 'skipped' in this context I meant that the test did not verify the code but
> reported the 'passed' status. That set of people will not even notice
> these tests.

I don't think you want that either.  I think you want the test to not be 
run.

We're going to automate this whole thing, and we may want to know when a 
given test broke.  If it's falsely reporting "pass", we'll get confused.

geir

> 
> Thanks,
> Mikhail
> 
>> This is why I advocate making a "separate tree" for the system tests -
>> make it clear that they are not the general unit tests...
>>
>>> It would not disturb most of the people because the test will pass in 'bad'
>>> environment. But those, who know about these tests will sometimes grep
>>> logs to validate configuration.
>> IMO, there's too much special information there, too much config.  I'm a
>> simple person, and like things clean and simple.  I don't like to mix
>> concerns when possible, and here's a place where it's definitely
>> possible to separate cleanly.
>>
>> I don't see the downside.
>>
>> geir
>>
>>> Thanks,
>>> Mikhail
>>>
>>>
>>>> Alternatively, they could be
>>>> included as part of a general test suite but be purposely skipped over at
>>>> test execution time using a
>>>> test exclusion list understood by the test runner.
>>>>
>>>>
>>>> Best regards,
>>>> George
>>>> ________________________________________
>>>> George C. Harley
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Tim Ellison <t....@gmail.com>
>>>> 27/01/2006 08:53
>>>> Please respond to
>>>> harmony-dev@incubator.apache.org
>>>>
>>>>
>>>> To
>>>> harmony-dev@incubator.apache.org
>>>> cc
>>>>
>>>> Subject
>>>> Re: [testing] code for exotic configurations
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Anton Avtamonov wrote:
>>>>>> Note that I could create my own provider and test with it, but what I
>>>> would
>>>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>>>> AlgorithmParameters from real provider as well as how my other classes
>>>> work
>>>>>> with real implementations of crypto Engines.
>>>>>>
>>>>>> Thanks,
>>>>>> Mikhail.
>>>>>>
>>>>> Hi Mikhail,
>>>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>>>> developer-level. Each unit test is intended to test just a limited
>>>>> piece of functionality separately from other sub-systems (test for one
>>>>> fucntion, test for one class, etc). Such tests must create a desired
>>>>> environment over the testing fucntionality and run the scenario in the
>>>>> predefined conditions. Unit tests usually able to cover all scenarios
>>>>> (execution paths) for the tested parts of fucntionality.
>>>>>
>>>>> What are you talking about looks like 'system' testing. Such tests
>>>>> usually run on the real environment and test the most often scenarious
>>>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>>>> testing is not concentrated on the particular fucntionality, but
>>>>> covers the work of the whole system.
>>>>> A sample is: "run some demo application on some particular platform,
>>>>> with some particular providers installed and perform some operations".
>>>>>
>>>>> I think currently we should focus on 'unit' test approach since it is
>>>>> more applicable during the development (so my advise is to revert your
>>>>> tests to install 'test' providers with the desired behavior as George
>>>>> proposed).
>>>>> However we should think about 'system' scenarios which can be run on
>>>>> the later stage and act as 'verification' of proper work of the entire
>>>>> system.
>>>> I agree with all this.  The unit tests are one style of test for
>>>> establishing the correctness of the code.  As you point out the unit
>>>> tests typically require a well-defined environment in which to run, and
>>>> it becomes a judgment-call as to whether a particular test's
>>>> environmental requirements are 'reasonable' or not.
>>>>
>>>> For example, you can reasonably expect all developers to have an
>>>> environment to run unit tests that has enough RAM and a writable disk
>>>> etc. such that if those things do not exist the tests will simply fail.
>>>>  However, you may decide it is unreasonable to expect the environment to
>>>> include a populated LDAP server, or a carefully configured RMI server.
>>>> If you were to call that environment unreasonable then testing JNDI and
>>>> RMI would likely involve mock objects etc. to get good unit tests.
>>>>
>>>> Of course, as you point out, once you are passing the unit tests you
>>>> also need the 'system' tests to ensure the code works in a real
>>>> environment.  Usage scenarios based on the bigger system are good, as is
>>>> running the bigger system's test suite on our runtime.
>>>>
>>>> Regards,
>>>> Tim
>>>>
>>>>
>>>>> --
>>>>> Anton Avtamonov,
>>>>> Intel Middleware Products Division
>>>>>
>>>> --
>>>>
>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>> IBM Java technology centre, UK.
>>>>
>>>>
>>>>
>>>
> 
> 

Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
On 1/27/06, Geir Magnusson Jr <ge...@pobox.com> wrote:
> > If the test can be configured by a few people only who works on that
> > specific area and those people are aware of those tests why not just
> > print a log when the test is skipped?
>
> Because the same set of people that will be bothered by separate suites
> will have the same reaction to skipped tests.

By 'skipped' in this context I meant that the test did not verify the code but
reported the 'passed' status. That set of people will not even notice
these tests.

Thanks,
Mikhail

>
> This is why I advocate making a "separate tree" for the system tests -
> make it clear that they are not the general unit tests...
>
> >
> > It would not disturb most of the people because the test will pass in 'bad'
> > environment. But those, who know about these tests will sometimes grep
> > logs to validate configuration.
>
> IMO, there's too much special information there, too much config.  I'm a
> simple person, and like things clean and simple.  I don't like to mix
> concerns when possible, and here's a place where it's definitely
> possible to separate cleanly.
>
> I don't see the downside.
>
> geir
>
> >
> > Thanks,
> > Mikhail
> >
> >
> >> Alternatively, they could be
> >> included as part of a general test suite but be purposely skipped over at
> >> test execution time using a
> >> test exclusion list understood by the test runner.
> >>
> >>
> >> Best regards,
> >> George
> >> ________________________________________
> >> George C. Harley
> >>
> >>
> >>
> >>
> >>
> >> Tim Ellison <t....@gmail.com>
> >> 27/01/2006 08:53
> >> Please respond to
> >> harmony-dev@incubator.apache.org
> >>
> >>
> >> To
> >> harmony-dev@incubator.apache.org
> >> cc
> >>
> >> Subject
> >> Re: [testing] code for exotic configurations
> >>
> >>
> >>
> >>
> >>
> >>
> >> Anton Avtamonov wrote:
> >>>> Note that I could create my own provider and test with it, but what I
> >> would
> >>>> really want is to test how my EncryptedPrivateKeyInfo works with
> >>>> AlgorithmParameters from real provider as well as how my other classes
> >> work
> >>>> with real implementations of crypto Engines.
> >>>>
> >>>> Thanks,
> >>>> Mikhail.
> >>>>
> >>> Hi Mikhail,
> >>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
> >>> developer-level. Each unit test is intended to test just a limited
> >>> piece of functionality separately from other sub-systems (test for one
> >>> fucntion, test for one class, etc). Such tests must create a desired
> >>> environment over the testing fucntionality and run the scenario in the
> >>> predefined conditions. Unit tests usually able to cover all scenarios
> >>> (execution paths) for the tested parts of fucntionality.
> >>>
> >>> What are you talking about looks like 'system' testing. Such tests
> >>> usually run on the real environment and test the most often scenarious
> >>> (the reduntant set, all scenarios usually cannot be covered). Such
> >>> testing is not concentrated on the particular fucntionality, but
> >>> covers the work of the whole system.
> >>> A sample is: "run some demo application on some particular platform,
> >>> with some particular providers installed and perform some operations".
> >>>
> >>> I think currently we should focus on 'unit' test approach since it is
> >>> more applicable during the development (so my advise is to revert your
> >>> tests to install 'test' providers with the desired behavior as George
> >>> proposed).
> >>> However we should think about 'system' scenarios which can be run on
> >>> the later stage and act as 'verification' of proper work of the entire
> >>> system.
> >> I agree with all this.  The unit tests are one style of test for
> >> establishing the correctness of the code.  As you point out the unit
> >> tests typically require a well-defined environment in which to run, and
> >> it becomes a judgment-call as to whether a particular test's
> >> environmental requirements are 'reasonable' or not.
> >>
> >> For example, you can reasonably expect all developers to have an
> >> environment to run unit tests that has enough RAM and a writable disk
> >> etc. such that if those things do not exist the tests will simply fail.
> >>  However, you may decide it is unreasonable to expect the environment to
> >> include a populated LDAP server, or a carefully configured RMI server.
> >> If you were to call that environment unreasonable then testing JNDI and
> >> RMI would likely involve mock objects etc. to get good unit tests.
> >>
> >> Of course, as you point out, once you are passing the unit tests you
> >> also need the 'system' tests to ensure the code works in a real
> >> environment.  Usage scenarios based on the bigger system are good, as is
> >> running the bigger system's test suite on our runtime.
> >>
> >> Regards,
> >> Tim
> >>
> >>
> >>> --
> >>> Anton Avtamonov,
> >>> Intel Middleware Products Division
> >>>
> >> --
> >>
> >> Tim Ellison (t.p.ellison@gmail.com)
> >> IBM Java technology centre, UK.
> >>
> >>
> >>
> >
> >
>

Re: [testing] code for exotic configurations

Posted by Geir Magnusson Jr <ge...@pobox.com>.

Mikhail Loenko wrote:
> Well, I like things clean and simple too :)
> 
> We could develop our own implementations for unit tests, it would take
> the same efforts as develop our own provider. It's a huge amount of work.
> Now we use BouncyCastle provider for the purpose of unit testing:
> it's open source, everyone can download it etc.
> 
> But our unit tests without any modification could serve as 'system' tests
> as Anton called them: Different providers might be installed to the system
> and the tests could be run in that configuration.
> 
> Does it mean that we are going to copy unit tests to a parallel 'system' test
> suite?

No - if you reuse something, that's cool.

What I'm trying to avoid is having a big bag of tests that require extra 
  things to be maintained in order for simple, common things to happen. 
  Case in point would be our current security2 testbed, where there's no 
naming convention (yet) and there has to be exclusion lists of tests by 
package in order for non-test code that is in the same packages to not 
be run.

(the solution is a *Test.java naming convention that gives full freedom 
for co-located support resources w/o the need for unncessary maintenance...)

Similarly, for things like system tests where there's going to be 
special setup and configuration anyway (like modifying the classpath, 
setting system properties, etc) then defining test cases from wherever 
is less of a burden.

See what I'm getting at?

geir

> 
> Thanks,
> Mikhail
> 
> On 1/27/06, Geir Magnusson Jr <ge...@pobox.com> wrote:
>>
>> Mikhail Loenko wrote:
>>> On 1/27/06, George Harley1 <GH...@uk.ibm.com> wrote:
>>>> But because we live in a less than ideal world there will, no doubt, be
>>>> some tests that will demand an
>>>> environment that is impossible or at the very least difficult to mock up
>>>> for the majority of developers/testers.
>>> I'm absolutely agree that we are neither living in the ideal world nor trying
>>> to make it ideal :)
>>>
>>> So until we got a 'system' test suite why should we weaken existing tests?
>>>
>>>> One solution could be to segregate those tests into a separate test suite
>>>> (available for all but primarily
>>>> for those working in the niche area that demands the special environment).
>>> Moving this kind of tests would affect many people: they will see
>>> separate suites,
>>> try, ask questions...
>>>
>>> If the test can be configured by a few people only who works on that
>>> specific area and those people are aware of those tests why not just
>>> print a log when the test is skipped?
>> Because the same set of people that will be bothered by separate suites
>> will have the same reaction to skipped tests.
>>
>> This is why I advocate making a "separate tree" for the system tests -
>> make it clear that they are not the general unit tests...
>>
>>> It would not disturb most of the people because the test will pass in 'bad'
>>> environment. But those, who know about these tests will sometimes grep
>>> logs to validate configuration.
>> IMO, there's too much special information there, too much config.  I'm a
>> simple person, and like things clean and simple.  I don't like to mix
>> concerns when possible, and here's a place where it's definitely
>> possible to separate cleanly.
>>
>> I don't see the downside.
>>
>> geir
>>
>>> Thanks,
>>> Mikhail
>>>
>>>
>>>> Alternatively, they could be
>>>> included as part of a general test suite but be purposely skipped over at
>>>> test execution time using a
>>>> test exclusion list understood by the test runner.
>>>>
>>>>
>>>> Best regards,
>>>> George
>>>> ________________________________________
>>>> George C. Harley
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Tim Ellison <t....@gmail.com>
>>>> 27/01/2006 08:53
>>>> Please respond to
>>>> harmony-dev@incubator.apache.org
>>>>
>>>>
>>>> To
>>>> harmony-dev@incubator.apache.org
>>>> cc
>>>>
>>>> Subject
>>>> Re: [testing] code for exotic configurations
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Anton Avtamonov wrote:
>>>>>> Note that I could create my own provider and test with it, but what I
>>>> would
>>>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>>>> AlgorithmParameters from real provider as well as how my other classes
>>>> work
>>>>>> with real implementations of crypto Engines.
>>>>>>
>>>>>> Thanks,
>>>>>> Mikhail.
>>>>>>
>>>>> Hi Mikhail,
>>>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>>>> developer-level. Each unit test is intended to test just a limited
>>>>> piece of functionality separately from other sub-systems (test for one
>>>>> fucntion, test for one class, etc). Such tests must create a desired
>>>>> environment over the testing fucntionality and run the scenario in the
>>>>> predefined conditions. Unit tests usually able to cover all scenarios
>>>>> (execution paths) for the tested parts of fucntionality.
>>>>>
>>>>> What are you talking about looks like 'system' testing. Such tests
>>>>> usually run on the real environment and test the most often scenarious
>>>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>>>> testing is not concentrated on the particular fucntionality, but
>>>>> covers the work of the whole system.
>>>>> A sample is: "run some demo application on some particular platform,
>>>>> with some particular providers installed and perform some operations".
>>>>>
>>>>> I think currently we should focus on 'unit' test approach since it is
>>>>> more applicable during the development (so my advise is to revert your
>>>>> tests to install 'test' providers with the desired behavior as George
>>>>> proposed).
>>>>> However we should think about 'system' scenarios which can be run on
>>>>> the later stage and act as 'verification' of proper work of the entire
>>>>> system.
>>>> I agree with all this.  The unit tests are one style of test for
>>>> establishing the correctness of the code.  As you point out the unit
>>>> tests typically require a well-defined environment in which to run, and
>>>> it becomes a judgment-call as to whether a particular test's
>>>> environmental requirements are 'reasonable' or not.
>>>>
>>>> For example, you can reasonably expect all developers to have an
>>>> environment to run unit tests that has enough RAM and a writable disk
>>>> etc. such that if those things do not exist the tests will simply fail.
>>>>  However, you may decide it is unreasonable to expect the environment to
>>>> include a populated LDAP server, or a carefully configured RMI server.
>>>> If you were to call that environment unreasonable then testing JNDI and
>>>> RMI would likely involve mock objects etc. to get good unit tests.
>>>>
>>>> Of course, as you point out, once you are passing the unit tests you
>>>> also need the 'system' tests to ensure the code works in a real
>>>> environment.  Usage scenarios based on the bigger system are good, as is
>>>> running the bigger system's test suite on our runtime.
>>>>
>>>> Regards,
>>>> Tim
>>>>
>>>>
>>>>> --
>>>>> Anton Avtamonov,
>>>>> Intel Middleware Products Division
>>>>>
>>>> --
>>>>
>>>> Tim Ellison (t.p.ellison@gmail.com)
>>>> IBM Java technology centre, UK.
>>>>
>>>>
>>>>
>>>
> 
> 


Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
Well, I like things clean and simple too :)

We could develop our own implementations for unit tests, it would take
the same efforts as develop our own provider. It's a huge amount of work.
Now we use BouncyCastle provider for the purpose of unit testing:
it's open source, everyone can download it etc.

But our unit tests without any modification could serve as 'system' tests
as Anton called them: Different providers might be installed to the system
and the tests could be run in that configuration.

Does it mean that we are going to copy unit tests to a parallel 'system' test
suite?

Thanks,
Mikhail

On 1/27/06, Geir Magnusson Jr <ge...@pobox.com> wrote:
>
>
> Mikhail Loenko wrote:
> > On 1/27/06, George Harley1 <GH...@uk.ibm.com> wrote:
> >> But because we live in a less than ideal world there will, no doubt, be
> >> some tests that will demand an
> >> environment that is impossible or at the very least difficult to mock up
> >> for the majority of developers/testers.
> >
> > I'm absolutely agree that we are neither living in the ideal world nor trying
> > to make it ideal :)
> >
> > So until we got a 'system' test suite why should we weaken existing tests?
> >
> >> One solution could be to segregate those tests into a separate test suite
> >> (available for all but primarily
> >> for those working in the niche area that demands the special environment).
> >
> > Moving this kind of tests would affect many people: they will see
> > separate suites,
> > try, ask questions...
> >
> > If the test can be configured by a few people only who works on that
> > specific area and those people are aware of those tests why not just
> > print a log when the test is skipped?
>
> Because the same set of people that will be bothered by separate suites
> will have the same reaction to skipped tests.
>
> This is why I advocate making a "separate tree" for the system tests -
> make it clear that they are not the general unit tests...
>
> >
> > It would not disturb most of the people because the test will pass in 'bad'
> > environment. But those, who know about these tests will sometimes grep
> > logs to validate configuration.
>
> IMO, there's too much special information there, too much config.  I'm a
> simple person, and like things clean and simple.  I don't like to mix
> concerns when possible, and here's a place where it's definitely
> possible to separate cleanly.
>
> I don't see the downside.
>
> geir
>
> >
> > Thanks,
> > Mikhail
> >
> >
> >> Alternatively, they could be
> >> included as part of a general test suite but be purposely skipped over at
> >> test execution time using a
> >> test exclusion list understood by the test runner.
> >>
> >>
> >> Best regards,
> >> George
> >> ________________________________________
> >> George C. Harley
> >>
> >>
> >>
> >>
> >>
> >> Tim Ellison <t....@gmail.com>
> >> 27/01/2006 08:53
> >> Please respond to
> >> harmony-dev@incubator.apache.org
> >>
> >>
> >> To
> >> harmony-dev@incubator.apache.org
> >> cc
> >>
> >> Subject
> >> Re: [testing] code for exotic configurations
> >>
> >>
> >>
> >>
> >>
> >>
> >> Anton Avtamonov wrote:
> >>>> Note that I could create my own provider and test with it, but what I
> >> would
> >>>> really want is to test how my EncryptedPrivateKeyInfo works with
> >>>> AlgorithmParameters from real provider as well as how my other classes
> >> work
> >>>> with real implementations of crypto Engines.
> >>>>
> >>>> Thanks,
> >>>> Mikhail.
> >>>>
> >>> Hi Mikhail,
> >>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
> >>> developer-level. Each unit test is intended to test just a limited
> >>> piece of functionality separately from other sub-systems (test for one
> >>> fucntion, test for one class, etc). Such tests must create a desired
> >>> environment over the testing fucntionality and run the scenario in the
> >>> predefined conditions. Unit tests usually able to cover all scenarios
> >>> (execution paths) for the tested parts of fucntionality.
> >>>
> >>> What are you talking about looks like 'system' testing. Such tests
> >>> usually run on the real environment and test the most often scenarious
> >>> (the reduntant set, all scenarios usually cannot be covered). Such
> >>> testing is not concentrated on the particular fucntionality, but
> >>> covers the work of the whole system.
> >>> A sample is: "run some demo application on some particular platform,
> >>> with some particular providers installed and perform some operations".
> >>>
> >>> I think currently we should focus on 'unit' test approach since it is
> >>> more applicable during the development (so my advise is to revert your
> >>> tests to install 'test' providers with the desired behavior as George
> >>> proposed).
> >>> However we should think about 'system' scenarios which can be run on
> >>> the later stage and act as 'verification' of proper work of the entire
> >>> system.
> >> I agree with all this.  The unit tests are one style of test for
> >> establishing the correctness of the code.  As you point out the unit
> >> tests typically require a well-defined environment in which to run, and
> >> it becomes a judgment-call as to whether a particular test's
> >> environmental requirements are 'reasonable' or not.
> >>
> >> For example, you can reasonably expect all developers to have an
> >> environment to run unit tests that has enough RAM and a writable disk
> >> etc. such that if those things do not exist the tests will simply fail.
> >>  However, you may decide it is unreasonable to expect the environment to
> >> include a populated LDAP server, or a carefully configured RMI server.
> >> If you were to call that environment unreasonable then testing JNDI and
> >> RMI would likely involve mock objects etc. to get good unit tests.
> >>
> >> Of course, as you point out, once you are passing the unit tests you
> >> also need the 'system' tests to ensure the code works in a real
> >> environment.  Usage scenarios based on the bigger system are good, as is
> >> running the bigger system's test suite on our runtime.
> >>
> >> Regards,
> >> Tim
> >>
> >>
> >>> --
> >>> Anton Avtamonov,
> >>> Intel Middleware Products Division
> >>>
> >> --
> >>
> >> Tim Ellison (t.p.ellison@gmail.com)
> >> IBM Java technology centre, UK.
> >>
> >>
> >>
> >
> >
>

Re: [testing] code for exotic configurations

Posted by Geir Magnusson Jr <ge...@pobox.com>.

Mikhail Loenko wrote:
> On 1/27/06, George Harley1 <GH...@uk.ibm.com> wrote:
>> But because we live in a less than ideal world there will, no doubt, be
>> some tests that will demand an
>> environment that is impossible or at the very least difficult to mock up
>> for the majority of developers/testers.
> 
> I'm absolutely agree that we are neither living in the ideal world nor trying
> to make it ideal :)
> 
> So until we got a 'system' test suite why should we weaken existing tests?
> 
>> One solution could be to segregate those tests into a separate test suite
>> (available for all but primarily
>> for those working in the niche area that demands the special environment).
> 
> Moving this kind of tests would affect many people: they will see
> separate suites,
> try, ask questions...
> 
> If the test can be configured by a few people only who works on that
> specific area and those people are aware of those tests why not just
> print a log when the test is skipped?

Because the same set of people that will be bothered by separate suites 
will have the same reaction to skipped tests.

This is why I advocate making a "separate tree" for the system tests - 
make it clear that they are not the general unit tests...

> 
> It would not disturb most of the people because the test will pass in 'bad'
> environment. But those, who know about these tests will sometimes grep
> logs to validate configuration.

IMO, there's too much special information there, too much config.  I'm a 
simple person, and like things clean and simple.  I don't like to mix 
concerns when possible, and here's a place where it's definitely 
possible to separate cleanly.

I don't see the downside.

geir

> 
> Thanks,
> Mikhail
> 
> 
>> Alternatively, they could be
>> included as part of a general test suite but be purposely skipped over at
>> test execution time using a
>> test exclusion list understood by the test runner.
>>
>>
>> Best regards,
>> George
>> ________________________________________
>> George C. Harley
>>
>>
>>
>>
>>
>> Tim Ellison <t....@gmail.com>
>> 27/01/2006 08:53
>> Please respond to
>> harmony-dev@incubator.apache.org
>>
>>
>> To
>> harmony-dev@incubator.apache.org
>> cc
>>
>> Subject
>> Re: [testing] code for exotic configurations
>>
>>
>>
>>
>>
>>
>> Anton Avtamonov wrote:
>>>> Note that I could create my own provider and test with it, but what I
>> would
>>>> really want is to test how my EncryptedPrivateKeyInfo works with
>>>> AlgorithmParameters from real provider as well as how my other classes
>> work
>>>> with real implementations of crypto Engines.
>>>>
>>>> Thanks,
>>>> Mikhail.
>>>>
>>> Hi Mikhail,
>>> There are 'system' and 'unit' tests. Traditionally, unit tests are of
>>> developer-level. Each unit test is intended to test just a limited
>>> piece of functionality separately from other sub-systems (test for one
>>> fucntion, test for one class, etc). Such tests must create a desired
>>> environment over the testing fucntionality and run the scenario in the
>>> predefined conditions. Unit tests usually able to cover all scenarios
>>> (execution paths) for the tested parts of fucntionality.
>>>
>>> What are you talking about looks like 'system' testing. Such tests
>>> usually run on the real environment and test the most often scenarious
>>> (the reduntant set, all scenarios usually cannot be covered). Such
>>> testing is not concentrated on the particular fucntionality, but
>>> covers the work of the whole system.
>>> A sample is: "run some demo application on some particular platform,
>>> with some particular providers installed and perform some operations".
>>>
>>> I think currently we should focus on 'unit' test approach since it is
>>> more applicable during the development (so my advise is to revert your
>>> tests to install 'test' providers with the desired behavior as George
>>> proposed).
>>> However we should think about 'system' scenarios which can be run on
>>> the later stage and act as 'verification' of proper work of the entire
>>> system.
>> I agree with all this.  The unit tests are one style of test for
>> establishing the correctness of the code.  As you point out the unit
>> tests typically require a well-defined environment in which to run, and
>> it becomes a judgment-call as to whether a particular test's
>> environmental requirements are 'reasonable' or not.
>>
>> For example, you can reasonably expect all developers to have an
>> environment to run unit tests that has enough RAM and a writable disk
>> etc. such that if those things do not exist the tests will simply fail.
>>  However, you may decide it is unreasonable to expect the environment to
>> include a populated LDAP server, or a carefully configured RMI server.
>> If you were to call that environment unreasonable then testing JNDI and
>> RMI would likely involve mock objects etc. to get good unit tests.
>>
>> Of course, as you point out, once you are passing the unit tests you
>> also need the 'system' tests to ensure the code works in a real
>> environment.  Usage scenarios based on the bigger system are good, as is
>> running the bigger system's test suite on our runtime.
>>
>> Regards,
>> Tim
>>
>>
>>> --
>>> Anton Avtamonov,
>>> Intel Middleware Products Division
>>>
>> --
>>
>> Tim Ellison (t.p.ellison@gmail.com)
>> IBM Java technology centre, UK.
>>
>>
>>
> 
> 

Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
On 1/27/06, George Harley1 <GH...@uk.ibm.com> wrote:
> But because we live in a less than ideal world there will, no doubt, be
> some tests that will demand an
> environment that is impossible or at the very least difficult to mock up
> for the majority of developers/testers.

I'm absolutely agree that we are neither living in the ideal world nor trying
to make it ideal :)

So until we got a 'system' test suite why should we weaken existing tests?

> One solution could be to segregate those tests into a separate test suite
> (available for all but primarily
> for those working in the niche area that demands the special environment).

Moving this kind of tests would affect many people: they will see
separate suites,
try, ask questions...

If the test can be configured by a few people only who works on that
specific area and those people are aware of those tests why not just
print a log when the test is skipped?

It would not disturb most of the people because the test will pass in 'bad'
environment. But those, who know about these tests will sometimes grep
logs to validate configuration.

Thanks,
Mikhail


> Alternatively, they could be
> included as part of a general test suite but be purposely skipped over at
> test execution time using a
> test exclusion list understood by the test runner.
>
>
> Best regards,
> George
> ________________________________________
> George C. Harley
>
>
>
>
>
> Tim Ellison <t....@gmail.com>
> 27/01/2006 08:53
> Please respond to
> harmony-dev@incubator.apache.org
>
>
> To
> harmony-dev@incubator.apache.org
> cc
>
> Subject
> Re: [testing] code for exotic configurations
>
>
>
>
>
>
> Anton Avtamonov wrote:
> >> Note that I could create my own provider and test with it, but what I
> would
> >> really want is to test how my EncryptedPrivateKeyInfo works with
> >> AlgorithmParameters from real provider as well as how my other classes
> work
> >> with real implementations of crypto Engines.
> >>
> >> Thanks,
> >> Mikhail.
> >>
> >
> > Hi Mikhail,
> > There are 'system' and 'unit' tests. Traditionally, unit tests are of
> > developer-level. Each unit test is intended to test just a limited
> > piece of functionality separately from other sub-systems (test for one
> > fucntion, test for one class, etc). Such tests must create a desired
> > environment over the testing fucntionality and run the scenario in the
> > predefined conditions. Unit tests usually able to cover all scenarios
> > (execution paths) for the tested parts of fucntionality.
> >
> > What are you talking about looks like 'system' testing. Such tests
> > usually run on the real environment and test the most often scenarious
> > (the reduntant set, all scenarios usually cannot be covered). Such
> > testing is not concentrated on the particular fucntionality, but
> > covers the work of the whole system.
> > A sample is: "run some demo application on some particular platform,
> > with some particular providers installed and perform some operations".
> >
> > I think currently we should focus on 'unit' test approach since it is
> > more applicable during the development (so my advise is to revert your
> > tests to install 'test' providers with the desired behavior as George
> > proposed).
> > However we should think about 'system' scenarios which can be run on
> > the later stage and act as 'verification' of proper work of the entire
> > system.
>
> I agree with all this.  The unit tests are one style of test for
> establishing the correctness of the code.  As you point out the unit
> tests typically require a well-defined environment in which to run, and
> it becomes a judgment-call as to whether a particular test's
> environmental requirements are 'reasonable' or not.
>
> For example, you can reasonably expect all developers to have an
> environment to run unit tests that has enough RAM and a writable disk
> etc. such that if those things do not exist the tests will simply fail.
>  However, you may decide it is unreasonable to expect the environment to
> include a populated LDAP server, or a carefully configured RMI server.
> If you were to call that environment unreasonable then testing JNDI and
> RMI would likely involve mock objects etc. to get good unit tests.
>
> Of course, as you point out, once you are passing the unit tests you
> also need the 'system' tests to ensure the code works in a real
> environment.  Usage scenarios based on the bigger system are good, as is
> running the bigger system's test suite on our runtime.
>
> Regards,
> Tim
>
>
> > --
> > Anton Avtamonov,
> > Intel Middleware Products Division
> >
>
> --
>
> Tim Ellison (t.p.ellison@gmail.com)
> IBM Java technology centre, UK.
>
>
>

Re: [testing] code for exotic configurations

Posted by George Harley1 <GH...@uk.ibm.com>.
Hi, 


> I agree with all this.  The unit tests are one style of test for
> establishing the correctness of the code.  As you point out the unit
> tests typically require a well-defined environment in which to run, and
> it becomes a judgment-call as to whether a particular test's
> environmental requirements are 'reasonable' or not.
> 

Yes, and that's something that needs to be carefully weighed up by the 
developer before committing any such unit test code into the test suite. 
If the environment required to run a test case cannot be easily recreated
by other developers then that case should not be checked in.


> For example, you can reasonably expect all developers to have an
> environment to run unit tests that has enough RAM and a writable disk
> etc. such that if those things do not exist the tests will simply fail.
> However, you may decide it is unreasonable to expect the environment to
> include a populated LDAP server, or a carefully configured RMI server.
> If you were to call that environment unreasonable then testing JNDI and
> RMI would likely involve mock objects etc. to get good unit tests.

Right, and it should be the responsibility of the test creator to ensure 
that those mocks and stubs (and
whatever other artifacts are needed) are checked in as part of the test. 
It should not be left to other
developers to spend a lot of time setting up the required environment for 
a test. Tests that are a pain 
to configure will just end up not being run.

But because we live in a less than ideal world there will, no doubt, be 
some tests that will demand an 
environment that is impossible or at the very least difficult to mock up 
for the majority of developers/testers. 
One solution could be to segregate those tests into a separate test suite 
(available for all but primarily
for those working in the niche area that demands the special environment). 
Alternatively, they could be
included as part of a general test suite but be purposely skipped over at 
test execution time using a 
test exclusion list understood by the test runner. 


Best regards, 
George
________________________________________
George C. Harley





Tim Ellison <t....@gmail.com> 
27/01/2006 08:53
Please respond to
harmony-dev@incubator.apache.org


To
harmony-dev@incubator.apache.org
cc

Subject
Re: [testing] code for exotic configurations






Anton Avtamonov wrote:
>> Note that I could create my own provider and test with it, but what I 
would
>> really want is to test how my EncryptedPrivateKeyInfo works with
>> AlgorithmParameters from real provider as well as how my other classes 
work
>> with real implementations of crypto Engines.
>>
>> Thanks,
>> Mikhail.
>>
> 
> Hi Mikhail,
> There are 'system' and 'unit' tests. Traditionally, unit tests are of
> developer-level. Each unit test is intended to test just a limited
> piece of functionality separately from other sub-systems (test for one
> fucntion, test for one class, etc). Such tests must create a desired
> environment over the testing fucntionality and run the scenario in the
> predefined conditions. Unit tests usually able to cover all scenarios
> (execution paths) for the tested parts of fucntionality.
> 
> What are you talking about looks like 'system' testing. Such tests
> usually run on the real environment and test the most often scenarious
> (the reduntant set, all scenarios usually cannot be covered). Such
> testing is not concentrated on the particular fucntionality, but
> covers the work of the whole system.
> A sample is: "run some demo application on some particular platform,
> with some particular providers installed and perform some operations".
> 
> I think currently we should focus on 'unit' test approach since it is
> more applicable during the development (so my advise is to revert your
> tests to install 'test' providers with the desired behavior as George
> proposed).
> However we should think about 'system' scenarios which can be run on
> the later stage and act as 'verification' of proper work of the entire
> system.

I agree with all this.  The unit tests are one style of test for
establishing the correctness of the code.  As you point out the unit
tests typically require a well-defined environment in which to run, and
it becomes a judgment-call as to whether a particular test's
environmental requirements are 'reasonable' or not.

For example, you can reasonably expect all developers to have an
environment to run unit tests that has enough RAM and a writable disk
etc. such that if those things do not exist the tests will simply fail.
 However, you may decide it is unreasonable to expect the environment to
include a populated LDAP server, or a carefully configured RMI server.
If you were to call that environment unreasonable then testing JNDI and
RMI would likely involve mock objects etc. to get good unit tests.

Of course, as you point out, once you are passing the unit tests you
also need the 'system' tests to ensure the code works in a real
environment.  Usage scenarios based on the bigger system are good, as is
running the bigger system's test suite on our runtime.

Regards,
Tim


> --
> Anton Avtamonov,
> Intel Middleware Products Division
> 

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.



Re: [testing] code for exotic configurations

Posted by Tim Ellison <t....@gmail.com>.
Anton Avtamonov wrote:
>> Note that I could create my own provider and test with it, but what I would
>> really want is to test how my EncryptedPrivateKeyInfo works with
>> AlgorithmParameters from real provider as well as how my other classes work
>> with real implementations of crypto Engines.
>>
>> Thanks,
>> Mikhail.
>>
> 
> Hi Mikhail,
> There are 'system' and 'unit' tests. Traditionally, unit tests are of
> developer-level. Each unit test is intended to test just a limited
> piece of functionality separately from other sub-systems (test for one
> fucntion, test for one class, etc). Such tests must create a desired
> environment over the testing fucntionality and run the scenario in the
> predefined conditions. Unit tests usually able to cover all scenarios
> (execution paths) for the tested parts of fucntionality.
> 
> What are you talking about looks like 'system' testing. Such tests
> usually run on the real environment and test the most often scenarious
> (the reduntant set, all scenarios usually cannot be covered). Such
> testing is not concentrated on the particular fucntionality, but
> covers the work of the whole system.
> A sample is: "run some demo application on some particular platform,
> with some particular providers installed and perform some operations".
> 
> I think currently we should focus on 'unit' test approach since it is
> more applicable during the development (so my advise is to revert your
> tests to install 'test' providers with the desired behavior as George
> proposed).
> However we should think about 'system' scenarios which can be run on
> the later stage and act as 'verification' of proper work of the entire
> system.

I agree with all this.  The unit tests are one style of test for
establishing the correctness of the code.  As you point out the unit
tests typically require a well-defined environment in which to run, and
it becomes a judgment-call as to whether a particular test's
environmental requirements are 'reasonable' or not.

For example, you can reasonably expect all developers to have an
environment to run unit tests that has enough RAM and a writable disk
etc. such that if those things do not exist the tests will simply fail.
 However, you may decide it is unreasonable to expect the environment to
include a populated LDAP server, or a carefully configured RMI server.
If you were to call that environment unreasonable then testing JNDI and
RMI would likely involve mock objects etc. to get good unit tests.

Of course, as you point out, once you are passing the unit tests you
also need the 'system' tests to ensure the code works in a real
environment.  Usage scenarios based on the bigger system are good, as is
running the bigger system's test suite on our runtime.

Regards,
Tim


> --
> Anton Avtamonov,
> Intel Middleware Products Division
> 

-- 

Tim Ellison (t.p.ellison@gmail.com)
IBM Java technology centre, UK.

Re: [testing] code for exotic configurations

Posted by Anton Avtamonov <an...@gmail.com>.
> Note that I could create my own provider and test with it, but what I would
> really want is to test how my EncryptedPrivateKeyInfo works with
> AlgorithmParameters from real provider as well as how my other classes work
> with real implementations of crypto Engines.
>
> Thanks,
> Mikhail.
>

Hi Mikhail,
There are 'system' and 'unit' tests. Traditionally, unit tests are of
developer-level. Each unit test is intended to test just a limited
piece of functionality separately from other sub-systems (test for one
fucntion, test for one class, etc). Such tests must create a desired
environment over the testing fucntionality and run the scenario in the
predefined conditions. Unit tests usually able to cover all scenarios
(execution paths) for the tested parts of fucntionality.

What are you talking about looks like 'system' testing. Such tests
usually run on the real environment and test the most often scenarious
(the reduntant set, all scenarios usually cannot be covered). Such
testing is not concentrated on the particular fucntionality, but
covers the work of the whole system.
A sample is: "run some demo application on some particular platform,
with some particular providers installed and perform some operations".

I think currently we should focus on 'unit' test approach since it is
more applicable during the development (so my advise is to revert your
tests to install 'test' providers with the desired behavior as George
proposed).
However we should think about 'system' scenarios which can be run on
the later stage and act as 'verification' of proper work of the entire
system.

--
Anton Avtamonov,
Intel Middleware Products Division

Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
Hi George

In my test I usually either do not specify provider or loop over all of them.
The example of the test is below:

public void testEncryptedPrivateKeyInfo () {
   try {
       AlgorithmParameters a = AlgorithmParameters.getInstance(...);

       EncryptedPrivateKeyInfo e = new EncryptedPrivateKeyInfo(a, myData);

       //do some check over e
   } catch (NoSuchAlgorithmException e) {
       logln("unable to verify EncryptedPrivateKeyInfo(...)  with
*real-life* provider");
   }
}

In most cases the test would work. But if a user does not have a provider with
AlgorithmParameters then the test would be skipped and constructor of
EncryptedPrivateKeyInfo would remain untested.

Note that I could create my own provider and test with it, but what I would
really want is to test how my EncryptedPrivateKeyInfo works with
AlgorithmParameters from real provider as well as how my other classes work
with real implementations of crypto Engines.

Thanks,
Mikhail.

On 1/26/06, George Harley1 <GH...@uk.ibm.com> wrote:
> Hi Mikhail,
>
> Thanks for your example.
>
> If I understand correctly (always a dangerous start to a sentence <g>),
> you would want a test case method to verify that a
> NoSuchAlgorithmException gets thrown from method
> AlgorithmParameters.getInstance() when the named provider does not have a
> concrete implementation of AlgorithmParameters available ?
> e.g.
>
> public void testGetInstanceThrowsNSAException() {
>    try {
>        AlgorithmParameters a = AlgorithmParameters.getInstance(...);
>        fail("Expected a NoSuchAlgorithmException");
>    } catch (Throwable t) {
>        assertTrue(t instanceof java.security.NoSuchAlgorithmException);
>    }
> }
>
>
> You wouldn't want a test like this to run normally (i.e. with a provider
> that *was* able to supply a concrete AlgorithmParameters) because the test
> would always fail. Instead, you would only want a test like this to be run
> where you have a provider incapable of returning the desired object.
>
> If you agree with the above interpretation of your "exotic configuration"
> then isn't it the case that you could simply augment your set of "good"
> providers with a provider (let's call it "BadProvider") that has been
> deliberately made incomplete and could be referred to by name in the above
> test method ? Then the getInstance call becomes
>
> AlgorithmParameters.getInstance("<algorithm name>", "BadProvider");
>
> This would mean that the test method *could* be included in the normal run
> of tests without need for skipping over. The "BadProvider" provider would
> be just another artifact used in the test configuration whose role is to
> intentionally create failure scenarios so that error handling works
> correctly.
>
> I'm not quite sure where the EncryptedPrivateKeyInfo constructor comes
> into your scenario, since if your provider is incapable of actually
> returning a valid AlgorithmParameters instance in the first place then how
> does the test code acquire one to test the EncryptedPrivateKeyInfo
> constructor ?
>
>
> Best regards,
> George
> ________________________________________
> George C. Harley
>
>
>
>
>
> Mikhail Loenko <ml...@gmail.com>
> 26/01/2006 13:13
> Please respond to
> harmony-dev@incubator.apache.org
>
>
> To
> harmony-dev@incubator.apache.org
> cc
>
> Subject
> Re: [testing] code for exotic configurations
>
>
>
>
>
>
> We have tests that verify that our security framework works with users
> providers installed in the system.
>
> Consider EncryptedPrivateKeyInfo class. To call one of its constructors
> you need AlgorithmParameters instance from a provider. If the providers
> installed in your system do not have implementation of AlgorithmParameters
> (or if you misconfigured your build) you'll unable to test that
> constructor
> of EncryptedPrivateKeyInfo.
>
> So exotic config is: User's providers do not have implementation of
> AlgorithmParameters.
>
> The test that fails when no AlgorithmParameters exists can scare a user,
> the test that silently skips has a chance to be skipping forever due to
> misconfiguration.
>
> That is an example.
>
> Thanks,
> Mikhail
>
> On 1/26/06, George Harley1 <GH...@uk.ibm.com> wrote:
> > Hi Mikhail,
> >
> > Just to help clarify things, could you give us all some examples of what
> > you define as an "exotic configuration" ?
> >
> > Thanks in advance,
> > George
> > ________________________________________
> > George C. Harley
> >
> >
> >
> >
> >
> > Mikhail Loenko <ml...@gmail.com>
> > 26/01/2006 12:11
> > Please respond to
> > harmony-dev@incubator.apache.org
> >
> >
> > To
> > harmony-dev@incubator.apache.org, geir@pobox.com
> > cc
> >
> > Subject
> > Re: [testing] code for exotic configurations
> >
> >
> >
> >
> >
> >
> > Do you mean that for a single test that verifies 10 lines of code
> > working on very specific configuration I have to create a parallel test
> > tree?
> >
> > What about tests that work in two different exotic configurations?
> Should
> > we duplicate them?
> >
> > Thanks,
> > Mikhail
> >
> > On 1/26/06, Geir Magnusson Jr <ge...@pobox.com> wrote:
> > > one solution is to simply group the "exotic" tests separately from the
> > > main tests, so they can be run optionally when you are in that exotic
> > > configuration.
> > >
> > > You can do this in several ways, including a naming convention, or
> > > another parallel code tree of the tests...
> > >
> > > I like the latter, as it makes it easier to "see"
> > >
> > > geir
> > >
> > >
> > > Mikhail Loenko wrote:
> > > > Well let's start a new thread as this is more general problem.
> > > >
> > > > So if we have some code designed for some exotic configurations
> > > > and we have tests that verify that exotic code.
> > > >
> > > > The test when run in usual configuration (not exotic one) should
> > > > report something that would not scare people. But if one
> > > > wants to test that specific exotic configuration that he should be
> > > > able to easily verify that he successfully made required conf and
> > > > the test worked well.
> > > >
> > > > The following options I see here:
> > > > 1) introduce a new test status (like skipped) to mark those tests
> that
> > > > did not actually run
> > > > 2) agree on exact wording that the skipped tests would print to
> allow
> > > > grep logs later
> > > > 3) introduce tests-indicators that would fail when current
> > > > configuration disallow
> > > > running certain tests
> > > >
> > > > Please let me know what you think
> > > >
> > > > Thanks,
> > > > Mikhail Loenko
> > > > Intel Middleware Products Division
> > > >
> > > >
> > >
> >
> >
> >
>
>
>

Re: [testing] code for exotic configurations

Posted by George Harley1 <GH...@uk.ibm.com>.
Hi Mikhail, 

Thanks for your example. 

If I understand correctly (always a dangerous start to a sentence <g>), 
you would want a test case method to verify that a 
NoSuchAlgorithmException gets thrown from method 
AlgorithmParameters.getInstance() when the named provider does not have a 
concrete implementation of AlgorithmParameters available ?
e.g.

public void testGetInstanceThrowsNSAException() {
    try {
        AlgorithmParameters a = AlgorithmParameters.getInstance(...); 
        fail("Expected a NoSuchAlgorithmException");
    } catch (Throwable t) {
        assertTrue(t instanceof java.security.NoSuchAlgorithmException);
    }
}


You wouldn't want a test like this to run normally (i.e. with a provider 
that *was* able to supply a concrete AlgorithmParameters) because the test 
would always fail. Instead, you would only want a test like this to be run 
where you have a provider incapable of returning the desired object. 

If you agree with the above interpretation of your "exotic configuration" 
then isn't it the case that you could simply augment your set of "good" 
providers with a provider (let's call it "BadProvider") that has been 
deliberately made incomplete and could be referred to by name in the above 
test method ? Then the getInstance call becomes

AlgorithmParameters.getInstance("<algorithm name>", "BadProvider");

This would mean that the test method *could* be included in the normal run 
of tests without need for skipping over. The "BadProvider" provider would 
be just another artifact used in the test configuration whose role is to 
intentionally create failure scenarios so that error handling works 
correctly. 

I'm not quite sure where the EncryptedPrivateKeyInfo constructor comes 
into your scenario, since if your provider is incapable of actually 
returning a valid AlgorithmParameters instance in the first place then how 
does the test code acquire one to test the EncryptedPrivateKeyInfo 
constructor ?


Best regards,
George
________________________________________
George C. Harley





Mikhail Loenko <ml...@gmail.com> 
26/01/2006 13:13
Please respond to
harmony-dev@incubator.apache.org


To
harmony-dev@incubator.apache.org
cc

Subject
Re: [testing] code for exotic configurations






We have tests that verify that our security framework works with users
providers installed in the system.

Consider EncryptedPrivateKeyInfo class. To call one of its constructors
you need AlgorithmParameters instance from a provider. If the providers
installed in your system do not have implementation of AlgorithmParameters
(or if you misconfigured your build) you'll unable to test that 
constructor
of EncryptedPrivateKeyInfo.

So exotic config is: User's providers do not have implementation of
AlgorithmParameters.

The test that fails when no AlgorithmParameters exists can scare a user,
the test that silently skips has a chance to be skipping forever due to
misconfiguration.

That is an example.

Thanks,
Mikhail

On 1/26/06, George Harley1 <GH...@uk.ibm.com> wrote:
> Hi Mikhail,
>
> Just to help clarify things, could you give us all some examples of what
> you define as an "exotic configuration" ?
>
> Thanks in advance,
> George
> ________________________________________
> George C. Harley
>
>
>
>
>
> Mikhail Loenko <ml...@gmail.com>
> 26/01/2006 12:11
> Please respond to
> harmony-dev@incubator.apache.org
>
>
> To
> harmony-dev@incubator.apache.org, geir@pobox.com
> cc
>
> Subject
> Re: [testing] code for exotic configurations
>
>
>
>
>
>
> Do you mean that for a single test that verifies 10 lines of code
> working on very specific configuration I have to create a parallel test
> tree?
>
> What about tests that work in two different exotic configurations? 
Should
> we duplicate them?
>
> Thanks,
> Mikhail
>
> On 1/26/06, Geir Magnusson Jr <ge...@pobox.com> wrote:
> > one solution is to simply group the "exotic" tests separately from the
> > main tests, so they can be run optionally when you are in that exotic
> > configuration.
> >
> > You can do this in several ways, including a naming convention, or
> > another parallel code tree of the tests...
> >
> > I like the latter, as it makes it easier to "see"
> >
> > geir
> >
> >
> > Mikhail Loenko wrote:
> > > Well let's start a new thread as this is more general problem.
> > >
> > > So if we have some code designed for some exotic configurations
> > > and we have tests that verify that exotic code.
> > >
> > > The test when run in usual configuration (not exotic one) should
> > > report something that would not scare people. But if one
> > > wants to test that specific exotic configuration that he should be
> > > able to easily verify that he successfully made required conf and
> > > the test worked well.
> > >
> > > The following options I see here:
> > > 1) introduce a new test status (like skipped) to mark those tests 
that
> > > did not actually run
> > > 2) agree on exact wording that the skipped tests would print to 
allow
> > > grep logs later
> > > 3) introduce tests-indicators that would fail when current
> > > configuration disallow
> > > running certain tests
> > >
> > > Please let me know what you think
> > >
> > > Thanks,
> > > Mikhail Loenko
> > > Intel Middleware Products Division
> > >
> > >
> >
>
>
>



Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
We have tests that verify that our security framework works with users
providers installed in the system.

Consider EncryptedPrivateKeyInfo class. To call one of its constructors
you need AlgorithmParameters instance from a provider. If the providers
installed in your system do not have implementation of AlgorithmParameters
(or if you misconfigured your build) you'll unable to test that constructor
of EncryptedPrivateKeyInfo.

So exotic config is: User's providers do not have implementation of
AlgorithmParameters.

The test that fails when no AlgorithmParameters exists can scare a user,
the test that silently skips has a chance to be skipping forever due to
misconfiguration.

That is an example.

Thanks,
Mikhail

On 1/26/06, George Harley1 <GH...@uk.ibm.com> wrote:
> Hi Mikhail,
>
> Just to help clarify things, could you give us all some examples of what
> you define as an "exotic configuration" ?
>
> Thanks in advance,
> George
> ________________________________________
> George C. Harley
>
>
>
>
>
> Mikhail Loenko <ml...@gmail.com>
> 26/01/2006 12:11
> Please respond to
> harmony-dev@incubator.apache.org
>
>
> To
> harmony-dev@incubator.apache.org, geir@pobox.com
> cc
>
> Subject
> Re: [testing] code for exotic configurations
>
>
>
>
>
>
> Do you mean that for a single test that verifies 10 lines of code
> working on very specific configuration I have to create a parallel test
> tree?
>
> What about tests that work in two different exotic configurations? Should
> we duplicate them?
>
> Thanks,
> Mikhail
>
> On 1/26/06, Geir Magnusson Jr <ge...@pobox.com> wrote:
> > one solution is to simply group the "exotic" tests separately from the
> > main tests, so they can be run optionally when you are in that exotic
> > configuration.
> >
> > You can do this in several ways, including a naming convention, or
> > another parallel code tree of the tests...
> >
> > I like the latter, as it makes it easier to "see"
> >
> > geir
> >
> >
> > Mikhail Loenko wrote:
> > > Well let's start a new thread as this is more general problem.
> > >
> > > So if we have some code designed for some exotic configurations
> > > and we have tests that verify that exotic code.
> > >
> > > The test when run in usual configuration (not exotic one) should
> > > report something that would not scare people. But if one
> > > wants to test that specific exotic configuration that he should be
> > > able to easily verify that he successfully made required conf and
> > > the test worked well.
> > >
> > > The following options I see here:
> > > 1) introduce a new test status (like skipped) to mark those tests that
> > > did not actually run
> > > 2) agree on exact wording that the skipped tests would print to allow
> > > grep logs later
> > > 3) introduce tests-indicators that would fail when current
> > > configuration disallow
> > > running certain tests
> > >
> > > Please let me know what you think
> > >
> > > Thanks,
> > > Mikhail Loenko
> > > Intel Middleware Products Division
> > >
> > >
> >
>
>
>

Re: [testing] code for exotic configurations

Posted by George Harley1 <GH...@uk.ibm.com>.
Hi Mikhail, 

Just to help clarify things, could you give us all some examples of what 
you define as an "exotic configuration" ?

Thanks in advance, 
George
________________________________________
George C. Harley





Mikhail Loenko <ml...@gmail.com> 
26/01/2006 12:11
Please respond to
harmony-dev@incubator.apache.org


To
harmony-dev@incubator.apache.org, geir@pobox.com
cc

Subject
Re: [testing] code for exotic configurations






Do you mean that for a single test that verifies 10 lines of code
working on very specific configuration I have to create a parallel test 
tree?

What about tests that work in two different exotic configurations? Should
we duplicate them?

Thanks,
Mikhail

On 1/26/06, Geir Magnusson Jr <ge...@pobox.com> wrote:
> one solution is to simply group the "exotic" tests separately from the
> main tests, so they can be run optionally when you are in that exotic
> configuration.
>
> You can do this in several ways, including a naming convention, or
> another parallel code tree of the tests...
>
> I like the latter, as it makes it easier to "see"
>
> geir
>
>
> Mikhail Loenko wrote:
> > Well let's start a new thread as this is more general problem.
> >
> > So if we have some code designed for some exotic configurations
> > and we have tests that verify that exotic code.
> >
> > The test when run in usual configuration (not exotic one) should
> > report something that would not scare people. But if one
> > wants to test that specific exotic configuration that he should be
> > able to easily verify that he successfully made required conf and
> > the test worked well.
> >
> > The following options I see here:
> > 1) introduce a new test status (like skipped) to mark those tests that
> > did not actually run
> > 2) agree on exact wording that the skipped tests would print to allow
> > grep logs later
> > 3) introduce tests-indicators that would fail when current
> > configuration disallow
> > running certain tests
> >
> > Please let me know what you think
> >
> > Thanks,
> > Mikhail Loenko
> > Intel Middleware Products Division
> >
> >
>



Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
Do you mean that for a single test that verifies 10 lines of code
working on very specific configuration I have to create a parallel test tree?

What about tests that work in two different exotic configurations? Should
we duplicate them?

Thanks,
Mikhail

On 1/26/06, Geir Magnusson Jr <ge...@pobox.com> wrote:
> one solution is to simply group the "exotic" tests separately from the
> main tests, so they can be run optionally when you are in that exotic
> configuration.
>
> You can do this in several ways, including a naming convention, or
> another parallel code tree of the tests...
>
> I like the latter, as it makes it easier to "see"
>
> geir
>
>
> Mikhail Loenko wrote:
> > Well let's start a new thread as this is more general problem.
> >
> > So if we have some code designed for some exotic configurations
> > and we have tests that verify that exotic code.
> >
> > The test when run in usual configuration (not exotic one) should
> > report something that would not scare people. But if one
> > wants to test that specific exotic configuration that he should be
> > able to easily verify that he successfully made required conf and
> > the test worked well.
> >
> > The following options I see here:
> > 1) introduce a new test status (like skipped) to mark those tests that
> > did not actually run
> > 2) agree on exact wording that the skipped tests would print to allow
> > grep logs later
> > 3) introduce tests-indicators that would fail when current
> > configuration disallow
> > running certain tests
> >
> > Please let me know what you think
> >
> > Thanks,
> > Mikhail Loenko
> > Intel Middleware Products Division
> >
> >
>

Re: [testing] code for exotic configurations

Posted by Geir Magnusson Jr <ge...@pobox.com>.
one solution is to simply group the "exotic" tests separately from the 
main tests, so they can be run optionally when you are in that exotic 
configuration.

You can do this in several ways, including a naming convention, or 
another parallel code tree of the tests...

I like the latter, as it makes it easier to "see"

geir


Mikhail Loenko wrote:
> Well let's start a new thread as this is more general problem.
> 
> So if we have some code designed for some exotic configurations
> and we have tests that verify that exotic code.
> 
> The test when run in usual configuration (not exotic one) should
> report something that would not scare people. But if one
> wants to test that specific exotic configuration that he should be
> able to easily verify that he successfully made required conf and
> the test worked well.
> 
> The following options I see here:
> 1) introduce a new test status (like skipped) to mark those tests that
> did not actually run
> 2) agree on exact wording that the skipped tests would print to allow
> grep logs later
> 3) introduce tests-indicators that would fail when current
> configuration disallow
> running certain tests
> 
> Please let me know what you think
> 
> Thanks,
> Mikhail Loenko
> Intel Middleware Products Division
> 
> 

Re: [testing] code for exotic configurations

Posted by Mikhail Loenko <ml...@gmail.com>.
Hello

We deep into discussion of what the test types are, which is useful
and interesting
but we a little bit went away from the original problem statement. I'd
like to try to
put the original problem in different words because it seems to be some
misunderstanding here.

So, we have some code, for example security classlib. It works in some
environment that includes other classlib modules, pluggable providers, OS, HW,
etc. And there are tests (let's call them just tests, no unit, no
system, etc) that
verify that my security classlib works well in reasonable environment
(other classlib
modules follow the spec, the installed providers are from some
'preferred' list, OS
is ..., etc).

The tests pass, everybody is happy so far.

One day I develop a new exotic provider that contains SecureRandom
implementation based on Moon phase, Solar activity and hash of the whole
internet snapshot. And I want to see how my security classlib works
with that new
provider.

I run the tests and see that some of them fail. At this point I'd like
to be able to
distinguish between:
- tests that fail because of absence e.g. AlgorithmParameters implementation in
the new provider (like an EncryptedPrivateKeyInfo test from my
previous posting in
this thread) and
- tests that fail because my code is not ready for that so good random numbers
returned by the new provider.


I would not like having 2 tests for EncryptedPrivateKeyInfo: one for normal
environment that fail when no AlgorithmParameters implementation available and
another one for exotic configurations  that allow not to have
AlgorithmParameters.


Another possible solution is to have the list of all possible
assumptions regarding
the configuration and put some test into exclusion list before run the
tests. In this
case I have to verify all possible assumptions to run any test


The third solution would be enable test reporting a warning that some
assumption
about the configuration or environment is not met. In normal
environment warnings
should be treated like errors. In exotic ones - in a different way.

One of the simplest solution would be - modify tests so that depending on some
system variable (like ignore.warnings) they will fail or pass.

Original approach used in security2 tests (log a warning) was not liked by the
community, so we have to find a good replacement.

Thanks,
Mikhail

Re: [testing] code for exotic configurations

Posted by Stepan Mishura <st...@gmail.com>.
I'd like to add two my kopeks :-)

Example, KerberosKey class depends on DES-based encryption to generate a DES
key from a user's password. A test may looks like the following:

// Test data from RFC 3961 (http://www.ietf.org/rfc/rfc3961.txt)
public void testGetEncoded() {
    byte[] expected = new byte[] { (byte) 0xcb, (byte) 0xc2, (byte) 0x2f,
                                   (byte) 0xae, (byte) 0x23, (byte) 0x52,
                                   (byte) 0x98, (byte) 0xe3 };

    KerberosPrincipal kp =
        new KerberosPrincipal("raeburn@ATHENA.MIT.EDU", 1);
    KerberosKey key = new KerberosKey(kp, "password". toCharArray(), "DES");

    assertTrue(Arrays.equals(expected, key.getEncoded()));
}

To run the test above the test's configuration has to include a provider
that provides required encryption type.

I'm not comfort with mocking provider implementation because this
complicates tests development: I need to write additional code for each
similar case (that is not a couple of lines) instead of using free provider
or harmony provider (if we'll have one) for testing purposes.

I wouldn't call this test 'a system test': it doesn't verify an underlying
system and a provider implementation. It just checks that KerberosKey class
has correct implementation. Also it doesn't look like as was written early
"quite complicated and can hardly perform the required environment settings
(tools installation, platform configuration, etc) from JUnit test."
Even more I won't use KerberosKey(KerberosPrincipal,char[], String)
constructor in any other testing scenarios. Otherwise I have to put it in a
'system test suite'.

Thanks,
Stepan Mishura
Intel Middleware Products Division




On 1/26/06, Mikhail Loenko <ml...@gmail.com> wrote:
>
> Well let's start a new thread as this is more general problem.
>
> So if we have some code designed for some exotic configurations
> and we have tests that verify that exotic code.
>
> The test when run in usual configuration (not exotic one) should
> report something that would not scare people. But if one
> wants to test that specific exotic configuration that he should be
> able to easily verify that he successfully made required conf and
> the test worked well.
>
> The following options I see here:
> 1) introduce a new test status (like skipped) to mark those tests that
> did not actually run
> 2) agree on exact wording that the skipped tests would print to allow
> grep logs later
> 3) introduce tests-indicators that would fail when current
> configuration disallow
> running certain tests
>
> Please let me know what you think
>
> Thanks,
> Mikhail Loenko
> Intel Middleware Products Division
>



--
Thanks,
Stepan Mishura
Intel Middleware Products Division