You are viewing a plain text version of this content. The canonical link for it is here.
Posted to derby-dev@db.apache.org by Julius Stroffek <Ju...@Sun.COM> on 2007/01/04 14:04:12 UTC
Testing implementation of private classes
Hi All,
I deal with a problem of testing a functionality of private classes. I
work on converting a derbynet/testProtocol.java test to junit
(DERBY-2031). I think that the tendency of writing derby tests is
a) Separate the testing source code from the derby source.
b) Produce a testing jar file derbyTesting.jar which could be run
against 'any' distribution/build of derby (more or less ;-).
c) The derby distribution itself should not contain any test code.
I have created a patch for DERBY-2031 which makes all the required
private classes and functions public. I think it does not matter in this
case. However, it is not a good practice to do this in general. I needed
some functionality of DDMReader and DDMWriter and some more classes. If
these classes will be changed to public it should not matter that
anybody could create its own instance of DDMReader/Writer because the
instances used by the DRDAConnectionThread are still hidden and can not
be obtained outside the package.
Bryan pointed me at the discussion at
http://www.nabble.com/Re%3A-JUnit-unit-tests---parallel-test-three--p5711307.html
where Kristian discussed this couple months ago. I talked to Kristian
and he told me that the problem was not solved at that time.
I have only two solutions to this:
1.) Change the visibility of required classes to public.
2.) Maintain the parallel package tree (suggested by Krisitian) with
implementation of tests of private classes. The tests will be in a same
package but different source directory. The build system will manage to
place the tests to derbyTesting.jar only. According the docs, the class
loader should choose the correct jar file to load a class (if the jar
sealing is turned off). I'll check this.
Since changing the visibility of code is not desired I see the option 2
as the preferred one. Are there any other possibilities? What are your
opinions?
Cheers
Julo
Re: Testing implementation of private classes
Posted by Julius Stroffek <Ju...@Sun.COM>.
Daniel John Debrunner wrote:
> What's the benefit of running unit tests against jar files? For
> functional tests I see the benefit, it is testing the user api using
> artifacts the user is expected to have on the classpath. However for
> unit tests I can't see what bugs would be found by testing against
> classes that are in a "fake" jar (assuming the unit tests are run
> against classes as well).
Actually, I can not see a really big difference on running unit tests
and function tests. The errors discovered by unit test may not appear
during a run of functional tests since the functional tests do not cover
each and every situation it may happen during an execution. However this
errors might show up to a user after some time or in a special case/deploy.
> So I see b) and c) as being the same really, manufacturing some fake
> jar (and for b it's more than derby.jar, may want to unit test network
> code for example) for no benefit. As an alternative I think the unit
> testing could easily be setup as a optional step in the build process.
> There could be an additional target in top level build.xml file that
> ran the unit tests against the classes and those that wanted could
> include it in their build process (e.g. a target all_with_unit_tests).
Yes, this might be a sufficient solution.
Julo
Re: Testing implementation of private classes
Posted by Dy...@Sun.COM.
Kristian Waagan <Kr...@Sun.COM> writes:
> Another proposal, which *I think* allow us to keep things as they are
> and generate a separate jar file for unit tests requiring
> package-private access:
> Use the non-standard option -Xbootclasspath.
>
> By specifying the jars on the boot classpath, the sealing protection
> will be overridden and the tests can be run. We can keep the existing
> tests as they are, and only use this option when running the
> "package-private unit tests". This approach also require two
> (currently three) steps to run all tests, but we won't have to change
> the existing jars. The unit tests can then also be run with
> distribution jars if anyone should wish to do that, and with both sane
> and insane builds.
>
> Does anyone see any showstoppers for this approach?
Sounds like a win-win to me...
--
dt
Re: Testing implementation of private classes
Posted by Julius Stroffek <Ju...@Sun.COM>.
Looks excellent. ;-)
Julo
Kristian Waagan wrote:
> Daniel John Debrunner wrote:
>> Julius.Stroffek wrote:
>>> Hi All,
>>>
>>> I'll try to divide the problem into two parts:
>>>
>>> 1.) Writing tests.
>>> I think that nobody argued that we should use something really
>>> different than the parallel source tree.
>>>
>>> 2.) Executing tests. There are more possibilities on this.
>>>
>>> a) executing the test against the classpath.
>>> b) creating the separate derby jar file for testing private classes.
>>> c) creating the separate jar with tests' implementation and
>>> creating the nearly same jar files as the derby product jars but
>>> which will not be sealed.
>>
>> What's the benefit of running unit tests against jar files? For
>> functional tests I see the benefit, it is testing the user api using
>> artifacts the user is expected to have on the classpath. However for
>> unit tests I can't see what bugs would be found by testing against
>> classes that are in a "fake" jar (assuming the unit tests are run
>> against classes as well).
>
> There is a practical benefit of using jars instead of classes when
> running the tests on remote hosts. Instead of copying thousands of
> small files, you copy a handful of jars.
> A little scripting on your own part can solve this easily of course,
> so I guess this boils down to almost nothing.
>
> Another proposal, which *I think* allow us to keep things as they are
> and generate a separate jar file for unit tests requiring
> package-private access:
> Use the non-standard option -Xbootclasspath.
>
> By specifying the jars on the boot classpath, the sealing protection
> will be overridden and the tests can be run. We can keep the existing
> tests as they are, and only use this option when running the
> "package-private unit tests". This approach also require two
> (currently three) steps to run all tests, but we won't have to change
> the existing jars. The unit tests can then also be run with
> distribution jars if anyone should wish to do that, and with both sane
> and insane builds.
>
> Does anyone see any showstoppers for this approach?
> I tested a small sample with Sun VMs. Does it work for VMs from other
> vendors?
>
>
>
Re: Testing implementation of private classes
Posted by Kristian Waagan <Kr...@Sun.COM>.
Daniel John Debrunner wrote:
> Kristian Waagan wrote:
>> Kristian Waagan wrote:
>>> Daniel John Debrunner wrote:
>>>> Julius.Stroffek wrote:
>> [ snip - suggestion about using non-standard option -Xbootclasspath ]
>>>
>>> Does anyone see any showstoppers for this approach?
>>> I tested a small sample with Sun VMs. Does it work for VMs from other
>>> vendors?
>>
>> FYI, I tested this with IBM's VMs (1.4 and 1.5) and also with JRockit
>> (1.4 and 1.5). They all seem to honor the
>> -Xbootclasspath/a:<comma-separated-list> option/syntax.
>
> Is it possible to determine if the test class is running from the
> bootclasspath? This would then allow the suite() method to determine if
> the class should run or not. Seems like the unit package-protected class
> tests should only run if:
> using classes || loaded from boot classpath
>
> And potentially some functional tests should not run in the boot
> classpath, e.g. those that would test what happens if a security
> permission is not granted, e.g. create a database in a folder with no
> write permission. I'm assuming here that classes in the boot class path
> have all permissions, is that correct?
Yes, they will be looked at the same way as the java.* classes are.
If you put derby.jar in the boot class path and derbyTesting.jar on the
class path, the security restrictions only apply to code calling
"restricted code" in derbyTesting (for instance FileWriter.write).
As soon as you move into derby.jar, you have all permissions. So if you
have the required OS privileges, you can create a database anywhere even
though the policy file states you are only allowed to do so in directory X.
Can we catch SecurityException (and check for sealing violation) when we
load the test classes and also try to load a Derby class at the same time?
Or get the class loader of a class, and check if it is null?
From the JavaDoc, it is not possible to conclude that the bootstrap
class loader is always null ("Some implementations may use null to
represent the bootstrap class loader."), and getting the class loader
might require permission to do so.
I think the functional tests should be run in a separate VM when run
from jars. When running from classes, we can run all tests (except those
testing jar-specific behavior).
Another option is to let tests that cannot run in the boot class path be
aware of that and return an empty suite, but this doesn't seem right to
me (increased complexity, insignificant gain).
It is a problem that the sealing violation can occur at different
places, depending on which jar/class we load first.
If a testing class is loaded first, an exception will be thrown when the
test loads a class from a Derby jar file.
If Derby is loaded first, the suite() method of the testing class will
never be reached.
Maybe the checking should be done in a class from derbyTesting if
possible, and if we agree to link the two together?
Currently, I'm more in favor of keeping the two sets of tests separate
by running them in different VMs. Might be some incorrect statements in
the mail, haven't really thought everything through, but please correct
me so we can work this issue out and see if the approach is feasible!
--
Kristian
>
> Thanks,
> Dan.
>
>
>
Re: Testing implementation of private classes
Posted by Daniel John Debrunner <dj...@apache.org>.
Kristian Waagan wrote:
> Kristian Waagan wrote:
>> Daniel John Debrunner wrote:
>>> Julius.Stroffek wrote:
> [ snip - suggestion about using non-standard option -Xbootclasspath ]
>>
>> Does anyone see any showstoppers for this approach?
>> I tested a small sample with Sun VMs. Does it work for VMs from other
>> vendors?
>
> FYI, I tested this with IBM's VMs (1.4 and 1.5) and also with JRockit
> (1.4 and 1.5). They all seem to honor the
> -Xbootclasspath/a:<comma-separated-list> option/syntax.
Is it possible to determine if the test class is running from the
bootclasspath? This would then allow the suite() method to determine if
the class should run or not. Seems like the unit package-protected class
tests should only run if:
using classes || loaded from boot classpath
And potentially some functional tests should not run in the boot
classpath, e.g. those that would test what happens if a security
permission is not granted, e.g. create a database in a folder with no
write permission. I'm assuming here that classes in the boot class path
have all permissions, is that correct?
Thanks,
Dan.
Re: Testing implementation of private classes
Posted by Kristian Waagan <Kr...@Sun.COM>.
Kristian Waagan wrote:
> Daniel John Debrunner wrote:
>> Julius.Stroffek wrote:
[ snip - suggestion about using non-standard option -Xbootclasspath ]
>
> Does anyone see any showstoppers for this approach?
> I tested a small sample with Sun VMs. Does it work for VMs from other
> vendors?
FYI, I tested this with IBM's VMs (1.4 and 1.5) and also with JRockit
(1.4 and 1.5). They all seem to honor the
-Xbootclasspath/a:<comma-separated-list> option/syntax.
--
Kristian
Re: Testing implementation of private classes
Posted by Kristian Waagan <Kr...@Sun.COM>.
Daniel John Debrunner wrote:
> Julius.Stroffek wrote:
>> Hi All,
>>
>> I'll try to divide the problem into two parts:
>>
>> 1.) Writing tests.
>> I think that nobody argued that we should use something really
>> different than the parallel source tree.
>>
>> 2.) Executing tests. There are more possibilities on this.
>>
>> a) executing the test against the classpath.
>> b) creating the separate derby jar file for testing private classes.
>> c) creating the separate jar with tests' implementation and
>> creating the nearly same jar files as the derby product jars but which
>> will not be sealed.
>
> What's the benefit of running unit tests against jar files? For
> functional tests I see the benefit, it is testing the user api using
> artifacts the user is expected to have on the classpath. However for
> unit tests I can't see what bugs would be found by testing against
> classes that are in a "fake" jar (assuming the unit tests are run
> against classes as well).
There is a practical benefit of using jars instead of classes when
running the tests on remote hosts. Instead of copying thousands of small
files, you copy a handful of jars.
A little scripting on your own part can solve this easily of course, so
I guess this boils down to almost nothing.
Another proposal, which *I think* allow us to keep things as they are
and generate a separate jar file for unit tests requiring
package-private access:
Use the non-standard option -Xbootclasspath.
By specifying the jars on the boot classpath, the sealing protection
will be overridden and the tests can be run. We can keep the existing
tests as they are, and only use this option when running the
"package-private unit tests". This approach also require two (currently
three) steps to run all tests, but we won't have to change the existing
jars. The unit tests can then also be run with distribution jars if
anyone should wish to do that, and with both sane and insane builds.
Does anyone see any showstoppers for this approach?
I tested a small sample with Sun VMs. Does it work for VMs from other
vendors?
--
Kristian
>
> So I see b) and c) as being the same really, manufacturing some fake jar
> (and for b it's more than derby.jar, may want to unit test network code
> for example) for no benefit. As an alternative I think the unit testing
> could easily be setup as a optional step in the build process. There
> could be an additional target in top level build.xml file that ran the
> unit tests against the classes and those that wanted could include it in
> their build process (e.g. a target all_with_unit_tests).
>
> Dam.
>
Re: Testing implementation of private classes
Posted by Daniel John Debrunner <dj...@apache.org>.
Julius.Stroffek wrote:
> Hi All,
>
> I'll try to divide the problem into two parts:
>
> 1.) Writing tests.
> I think that nobody argued that we should use something really different
> than the parallel source tree.
>
> 2.) Executing tests. There are more possibilities on this.
>
> a) executing the test against the classpath.
> b) creating the separate derby jar file for testing private classes.
> c) creating the separate jar with tests' implementation and creating
> the nearly same jar files as the derby product jars but which will not
> be sealed.
What's the benefit of running unit tests against jar files? For
functional tests I see the benefit, it is testing the user api using
artifacts the user is expected to have on the classpath. However for
unit tests I can't see what bugs would be found by testing against
classes that are in a "fake" jar (assuming the unit tests are run
against classes as well).
So I see b) and c) as being the same really, manufacturing some fake jar
(and for b it's more than derby.jar, may want to unit test network code
for example) for no benefit. As an alternative I think the unit testing
could easily be setup as a optional step in the build process. There
could be an additional target in top level build.xml file that ran the
unit tests against the classes and those that wanted could include it in
their build process (e.g. a target all_with_unit_tests).
Dam.
Re: Testing implementation of private classes
Posted by "Julius.Stroffek" <Ju...@Sun.COM>.
Hi All,
I'll try to divide the problem into two parts:
1.) Writing tests.
I think that nobody argued that we should use something really different
than the parallel source tree.
2.) Executing tests. There are more possibilities on this.
a) executing the test against the classpath.
b) creating the separate derby jar file for testing private classes.
c) creating the separate jar with tests' implementation and creating
the nearly same jar files as the derby product jars but which will not
be sealed.
Probably option c) is a bit too complicated to manage and does not bring
any special benefit against a) or b).
Should I create a separate jira for this or it could be done as a part
of DERBY-2031?
Any other suggestions?
Cheers
Julo
Kristian Waagan wrote:
> John Embretsen wrote:
>> Julius Stroffek wrote:
>>> Hi All,
>>>
>>> I deal with a problem of testing a functionality of private classes.
>>
>> <snip>
>>
>>> Bryan pointed me at the discussion at
>>> http://www.nabble.com/Re%3A-JUnit-unit-tests---parallel-test-three--p5711307.html
>>> where Kristian discussed this couple months ago. I talked to
>>> Kristian and he told me that the problem was not solved at that time.
>>>
>>> I have only two solutions to this:
>>>
>>> 1.) Change the visibility of required classes to public.
>>> 2.) Maintain the parallel package tree (suggested by Krisitian) with
>>> implementation of tests of private classes.
>> <snip>
>>
>> Just wondering, does anyone know if using reflection to bypass the
>> Java access protection (as Kristian mentioned in the above mentioned
>> thread) is completely out of the question?
>
> Hi John,
>
> The subject of the original mail might be a bit inaccurate, as my
> interpretation is that what we want to achieve is to write unit tests
> for package-private classes (no modifier in front of the class keyword).
> This is different from (inner) private classes.
>
> I think using reflection to test private *methods* should be avoided
> if possible. The test code gets a lot bigger and harder to understand.
> If you really need to test private methods, I guess using a static
> nested class would be just as good.
>
> I do not know the state of the tools providing the kind of
> functionality you mention, so it might be worth checking out anyway.
>
>
> Last, I would really appreciate a way to write unit tests for
> package-private classes. I think some good ideas have been put forth
> in this mail-thread, and hope the community will be able to reach
> consensus for an acceptable solution.
>
>
>
Re: Testing implementation of private classes
Posted by Kristian Waagan <Kr...@Sun.COM>.
John Embretsen wrote:
> Julius Stroffek wrote:
>> Hi All,
>>
>> I deal with a problem of testing a functionality of private classes.
>
> <snip>
>
>> Bryan pointed me at the discussion at
>> http://www.nabble.com/Re%3A-JUnit-unit-tests---parallel-test-three--p5711307.html
>> where Kristian discussed this couple months ago. I talked to Kristian
>> and he told me that the problem was not solved at that time.
>>
>> I have only two solutions to this:
>>
>> 1.) Change the visibility of required classes to public.
>> 2.) Maintain the parallel package tree (suggested by Krisitian) with
>> implementation of tests of private classes.
> <snip>
>
> Just wondering, does anyone know if using reflection to bypass the Java
> access protection (as Kristian mentioned in the above mentioned thread)
> is completely out of the question?
Hi John,
The subject of the original mail might be a bit inaccurate, as my
interpretation is that what we want to achieve is to write unit tests
for package-private classes (no modifier in front of the class keyword).
This is different from (inner) private classes.
I think using reflection to test private *methods* should be avoided if
possible. The test code gets a lot bigger and harder to understand. If
you really need to test private methods, I guess using a static nested
class would be just as good.
I do not know the state of the tools providing the kind of functionality
you mention, so it might be worth checking out anyway.
Last, I would really appreciate a way to write unit tests for
package-private classes. I think some good ideas have been put forth in
this mail-thread, and hope the community will be able to reach consensus
for an acceptable solution.
--
Kristian
>
> I'm asking because it is mentioned in the JUnit FAQ, and discussed in
> [1], although the article does not cover private classes, only fields
> and methods/constructors.
>
>
Re: Testing implementation of private classes
Posted by John Embretsen <Jo...@Sun.COM>.
Julius Stroffek wrote:
> Hi All,
>
> I deal with a problem of testing a functionality of private classes.
<snip>
> Bryan pointed me at the discussion at
> http://www.nabble.com/Re%3A-JUnit-unit-tests---parallel-test-three--p5711307.html
> where Kristian discussed this couple months ago. I talked to Kristian
> and he told me that the problem was not solved at that time.
>
> I have only two solutions to this:
>
> 1.) Change the visibility of required classes to public.
> 2.) Maintain the parallel package tree (suggested by Krisitian) with
> implementation of tests of private classes.
<snip>
Just wondering, does anyone know if using reflection to bypass the Java access
protection (as Kristian mentioned in the above mentioned thread) is completely
out of the question?
I'm asking because it is mentioned in the JUnit FAQ, and discussed in [1],
although the article does not cover private classes, only fields and
methods/constructors.
--
John
[1] "Subverting Java Access Protection for Unit Testing" -
http://www.onjava.com/pub/a/onjava/2003/11/12/reflection.html
Re: Testing implementation of private classes
Posted by Julius Stroffek <Ju...@Sun.COM>.
Thanks Dan, you are right. I did not notice the first one surrounded by
more text. ;-)
Julo
Daniel John Debrunner wrote:
> Julius Stroffek wrote:
>> Hi,
>>
>> I checked the 10.2.2.0 release and all the .jar files in /lib
>> directory are not sealed.
>
> derby.jar is sealed. I see Sealed entries in the manifest file.
>
> Dan.
>
>
Re: Testing implementation of private classes
Posted by Daniel John Debrunner <dj...@apache.org>.
Julius Stroffek wrote:
> Hi,
>
> I checked the 10.2.2.0 release and all the .jar files in /lib directory
> are not sealed.
derby.jar is sealed. I see Sealed entries in the manifest file.
Dan.
Re: Testing implementation of private classes
Posted by Julius Stroffek <Ju...@Sun.COM>.
Hi,
I checked the 10.2.2.0 release and all the .jar files in /lib directory
are not sealed. Do we plan to seal the jars in the future? If so, I
think that 'buildjars' target than could create the same jars twice -
one set for testing purposes only which will have the same content
except they will not be sealed.
Julo
Oystein Grovlen - Sun Norway wrote:
> Daniel John Debrunner wrote:
>> Julius Stroffek wrote:
>>> I have only two solutions to this:
>>>
>>> 1.) Change the visibility of required classes to public.
>>> 2.) Maintain the parallel package tree (suggested by Krisitian) with
>>> implementation of tests of private classes. The tests will be in a
>>> same package but different source directory. The build system will
>>> manage to place the tests to derbyTesting.jar only. According the
>>> docs, the class loader should choose the correct jar file to load a
>>> class (if the jar sealing is turned off). I'll check this.
>>>
>>> Since changing the visibility of code is not desired I see the
>>> option 2 as the preferred one. Are there any other possibilities?
>>> What are your opinions?
>>
>> Both solutions are affecting the visibility of code. Option 2) is
>> removing the option of sealing *all* the classes in that package,
>> thus defeating any package access protection.
>>
>> If the testing really is just for a specific class, then it does not
>> matter that the class is in one of the derby "product" jars. So a
>> couple of variants of 2) are:
>>
>> - test only when running tests from the classpath
>> - archive the class into another jar where it is not sealed and run
>> the tests against with that jar in the classpath and none of the
>> derby "product" jars.
>
> How about only sealing insane jars? Then we could run such tests
> against sane jars.
>
Re: Testing implementation of private classes
Posted by Oystein Grovlen - Sun Norway <Oy...@Sun.COM>.
Daniel John Debrunner wrote:
> Julius Stroffek wrote:
>> I have only two solutions to this:
>>
>> 1.) Change the visibility of required classes to public.
>> 2.) Maintain the parallel package tree (suggested by Krisitian) with
>> implementation of tests of private classes. The tests will be in a
>> same package but different source directory. The build system will
>> manage to place the tests to derbyTesting.jar only. According the
>> docs, the class loader should choose the correct jar file to load a
>> class (if the jar sealing is turned off). I'll check this.
>>
>> Since changing the visibility of code is not desired I see the option
>> 2 as the preferred one. Are there any other possibilities? What are
>> your opinions?
>
> Both solutions are affecting the visibility of code. Option 2) is
> removing the option of sealing *all* the classes in that package, thus
> defeating any package access protection.
>
> If the testing really is just for a specific class, then it does not
> matter that the class is in one of the derby "product" jars. So a couple
> of variants of 2) are:
>
> - test only when running tests from the classpath
> - archive the class into another jar where it is not sealed and run the
> tests against with that jar in the classpath and none of the derby
> "product" jars.
How about only sealing insane jars? Then we could run such tests
against sane jars.
--
Øystein
Re: Testing implementation of private classes
Posted by Daniel John Debrunner <dj...@apache.org>.
Julius Stroffek wrote:
> Hi All,
>
> I deal with a problem of testing a functionality of private classes. I
> work on converting a derbynet/testProtocol.java test to junit
> (DERBY-2031). I think that the tendency of writing derby tests is
>
> a) Separate the testing source code from the derby source.
> b) Produce a testing jar file derbyTesting.jar which could be run
> against 'any' distribution/build of derby (more or less ;-).
> c) The derby distribution itself should not contain any test code.
>
> I have created a patch for DERBY-2031 which makes all the required
> private classes and functions public. I think it does not matter in this
> case. However, it is not a good practice to do this in general. I needed
> some functionality of DDMReader and DDMWriter and some more classes. If
> these classes will be changed to public it should not matter that
> anybody could create its own instance of DDMReader/Writer because the
> instances used by the DRDAConnectionThread are still hidden and can not
> be obtained outside the package.
>
> Bryan pointed me at the discussion at
> http://www.nabble.com/Re%3A-JUnit-unit-tests---parallel-test-three--p5711307.html
> where Kristian discussed this couple months ago. I talked to Kristian
> and he told me that the problem was not solved at that time.
>
> I have only two solutions to this:
>
> 1.) Change the visibility of required classes to public.
> 2.) Maintain the parallel package tree (suggested by Krisitian) with
> implementation of tests of private classes. The tests will be in a same
> package but different source directory. The build system will manage to
> place the tests to derbyTesting.jar only. According the docs, the class
> loader should choose the correct jar file to load a class (if the jar
> sealing is turned off). I'll check this.
>
> Since changing the visibility of code is not desired I see the option 2
> as the preferred one. Are there any other possibilities? What are your
> opinions?
Both solutions are affecting the visibility of code. Option 2) is
removing the option of sealing *all* the classes in that package, thus
defeating any package access protection.
If the testing really is just for a specific class, then it does not
matter that the class is in one of the derby "product" jars. So a couple
of variants of 2) are:
- test only when running tests from the classpath
- archive the class into another jar where it is not sealed and run
the tests against with that jar in the classpath and none of the derby
"product" jars.
Dan.