You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@river.apache.org by "Peter Firmstone (JIRA)" <ji...@apache.org> on 2009/09/27 03:23:16 UTC
[jira] Created: (RIVER-319) Change River Build Dist structure to
support jtreg test automation
Change River Build Dist structure to support jtreg test automation
------------------------------------------------------------------
Key: RIVER-319
URL: https://issues.apache.org/jira/browse/RIVER-319
Project: River
Issue Type: Sub-task
Reporter: Peter Firmstone
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (RIVER-319) Change River Build Dist structure
to support jtreg test automation
Posted by Jonathan Costers <jo...@googlemail.com>.
Hi, I'm sorry, I haven't had the time to think about this too deeply so
can't really comment at this point.
But don't let that stop you from doing anything! Any
suggestion/comment/remark is welcome!
Best
Jonathan
2009/9/29 Peter Firmstone <ji...@zeus.net.au>
> We could always move the jar files to a temporary all permission test
> directory then move them back after testing is complete, the files that need
> to be moved can be specified in one place, so the multiple security policy
> files for each test need to be changed only once?
>
> If your happy with this solution, I'll add the comment to Jira.
>
> Cheers,
>
> Peter.
>
>
> Peter Firmstone wrote:
>
>> The difficult part is to grant all permission only to those files listed
>> below, without making the build process fragile (multiple changes required
>> for change). This was why I was against specifically listing jar's in the
>> policy files, however it is an alternative solution. It seemed easier (long
>> term) to just specify a directory for the platform jars.
>>
>> Cheers,
>>
>> Peter.
>>
>> Jonathan Costers wrote:
>>
>>> Thanks for the heads up.
>>> I've been re-reading Peter Jones' posts to refresh my memory, this all
>>> makes
>>> sense.
>>> I think (hope) we are able to pull this off without any restructuring
>>> though.
>>>
>>> Best
>>> Jonathan
>>>
>>> 2009/9/28 Peter Firmstone (JIRA) <ji...@apache.org>
>>>
>>>
>>>
>>>> [
>>>>
>>>> https://issues.apache.org/jira/browse/RIVER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760202#action_12760202]
>>>>
>>>>
>>>> Peter Firmstone commented on RIVER-319:
>>>> ---------------------------------------
>>>>
>>>> The jtreg tests have their own security policy files, they don't use the
>>>> jvm's, currently due to our distribution structure, we have to
>>>> physically
>>>> move (not copy or link, they must be moved) the following files to the
>>>> jvm's
>>>> extension directory: jre/lib/ext/ where each jtreg test currently grants
>>>> all
>>>> permission. This is the reason for your test failures.
>>>>
>>>> lib/jsk-platform.jar
>>>> lib/jsk-resources.jar
>>>> lib/jsk-lib.jar
>>>> lib/phoenix-init.jar
>>>> lib-ext/jsk-policy.jar
>>>>
>>>> Then in a directory relative to the one you choose for your working
>>>> directory, you must add a file named test.props
>>>>
>>>> the correct location for test.props:
>>>>
>>>>
>>>>> /**
>>>>> * Returns extra test properties. Looks for the file
>>>>>
>>>>>
>>>> "../../test.props"
>>>>
>>>>
>>>>> * and reads it in as a Properties file. Assuming the working
>>>>>
>>>>>
>>>> directory
>>>>
>>>>
>>>>> * is "<path>/JTwork/scratch", this will find "<path>/test.props".
>>>>> */
>>>>>
>>>>>
>>>> My test.props contains the following two lines:
>>>>
>>>> jsk.home=/opt/src/river/trunk
>>>> javatest.maxOutputSize=500000
>>>>
>>>> Alternatively all test.props file options can be passed in via jtreg to
>>>> the
>>>> jdk using "-Doption=something", jtreg is run in the working directory,
>>>> we
>>>> can create an arbitrary one, the example below assumes the suggested
>>>> build
>>>> release directory structure above, negating the need to physically move
>>>> the
>>>> platform jar files to the jre extensions directory. Note the debug
>>>> option,
>>>> it's very helpful. jsk.home could be set as an environment variable in
>>>> build.xml The command below also assumes the earlier suggestions to
>>>> modify
>>>> the jtreg test policy files.
>>>>
>>>> $jtreg -verbose:all,nopass -cpa: /opt/src/river/trunk/lib-platform -jdk
>>>> /usr/jdk/jdk1.5.0_15 -Djsk.home="/opt/src/river/trunk"
>>>> -Djava.security.debug=access,failure -Djavatest.maxOutputSize=500000
>>>> ./qa/jtreg/
>>>>
>>>> Java 1.5 is currently the preferred test environment for these tests,
>>>> while Java 1.6 is preferred for the regression tests. In order to get
>>>> all
>>>> tests singing on the same jdk we need to implement kerberos client
>>>> functionality, ktools etc, since these were removed from jdk1.6, this
>>>> can
>>>> wait until later though.
>>>>
>>>> Cheers,
>>>>
>>>> Peter.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Change River Build Dist structure to support jtreg test automation
>>>>> ------------------------------------------------------------------
>>>>>
>>>>> Key: RIVER-319
>>>>> URL: https://issues.apache.org/jira/browse/RIVER-319
>>>>> Project: River
>>>>> Issue Type: Sub-task
>>>>> Reporter: Peter Firmstone
>>>>>
>>>>>
>>>>>
>>>> --
>>>> This message is automatically generated by JIRA.
>>>> -
>>>> You can reply to this email to add a comment to the issue online.
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
Re: [jira] Commented: (RIVER-319) Change River Build Dist structure
to support jtreg test automation
Posted by Peter Firmstone <ji...@zeus.net.au>.
We could always move the jar files to a temporary all permission test
directory then move them back after testing is complete, the files that
need to be moved can be specified in one place, so the multiple security
policy files for each test need to be changed only once?
If your happy with this solution, I'll add the comment to Jira.
Cheers,
Peter.
Peter Firmstone wrote:
> The difficult part is to grant all permission only to those files
> listed below, without making the build process fragile (multiple
> changes required for change). This was why I was against specifically
> listing jar's in the policy files, however it is an alternative
> solution. It seemed easier (long term) to just specify a directory
> for the platform jars.
>
> Cheers,
>
> Peter.
>
> Jonathan Costers wrote:
>> Thanks for the heads up.
>> I've been re-reading Peter Jones' posts to refresh my memory, this
>> all makes
>> sense.
>> I think (hope) we are able to pull this off without any restructuring
>> though.
>>
>> Best
>> Jonathan
>>
>> 2009/9/28 Peter Firmstone (JIRA) <ji...@apache.org>
>>
>>
>>> [
>>> https://issues.apache.org/jira/browse/RIVER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760202#action_12760202]
>>>
>>>
>>> Peter Firmstone commented on RIVER-319:
>>> ---------------------------------------
>>>
>>> The jtreg tests have their own security policy files, they don't use
>>> the
>>> jvm's, currently due to our distribution structure, we have to
>>> physically
>>> move (not copy or link, they must be moved) the following files to
>>> the jvm's
>>> extension directory: jre/lib/ext/ where each jtreg test currently
>>> grants all
>>> permission. This is the reason for your test failures.
>>>
>>> lib/jsk-platform.jar
>>> lib/jsk-resources.jar
>>> lib/jsk-lib.jar
>>> lib/phoenix-init.jar
>>> lib-ext/jsk-policy.jar
>>>
>>> Then in a directory relative to the one you choose for your working
>>> directory, you must add a file named test.props
>>>
>>> the correct location for test.props:
>>>
>>>> /**
>>>> * Returns extra test properties. Looks for the file
>>>>
>>> "../../test.props"
>>>
>>>> * and reads it in as a Properties file. Assuming the working
>>>>
>>> directory
>>>
>>>> * is "<path>/JTwork/scratch", this will find "<path>/test.props".
>>>> */
>>>>
>>> My test.props contains the following two lines:
>>>
>>> jsk.home=/opt/src/river/trunk
>>> javatest.maxOutputSize=500000
>>>
>>> Alternatively all test.props file options can be passed in via jtreg
>>> to the
>>> jdk using "-Doption=something", jtreg is run in the working
>>> directory, we
>>> can create an arbitrary one, the example below assumes the suggested
>>> build
>>> release directory structure above, negating the need to physically
>>> move the
>>> platform jar files to the jre extensions directory. Note the debug
>>> option,
>>> it's very helpful. jsk.home could be set as an environment variable in
>>> build.xml The command below also assumes the earlier suggestions to
>>> modify
>>> the jtreg test policy files.
>>>
>>> $jtreg -verbose:all,nopass -cpa: /opt/src/river/trunk/lib-platform
>>> -jdk
>>> /usr/jdk/jdk1.5.0_15 -Djsk.home="/opt/src/river/trunk"
>>> -Djava.security.debug=access,failure -Djavatest.maxOutputSize=500000
>>> ./qa/jtreg/
>>>
>>> Java 1.5 is currently the preferred test environment for these tests,
>>> while Java 1.6 is preferred for the regression tests. In order to
>>> get all
>>> tests singing on the same jdk we need to implement kerberos client
>>> functionality, ktools etc, since these were removed from jdk1.6,
>>> this can
>>> wait until later though.
>>>
>>> Cheers,
>>>
>>> Peter.
>>>
>>>
>>>
>>>
>>>
>>>> Change River Build Dist structure to support jtreg test automation
>>>> ------------------------------------------------------------------
>>>>
>>>> Key: RIVER-319
>>>> URL: https://issues.apache.org/jira/browse/RIVER-319
>>>> Project: River
>>>> Issue Type: Sub-task
>>>> Reporter: Peter Firmstone
>>>>
>>>>
>>> --
>>> This message is automatically generated by JIRA.
>>> -
>>> You can reply to this email to add a comment to the issue online.
>>>
>>>
>>>
>>
>>
>
>
Re: [jira] Commented: (RIVER-319) Change River Build Dist structure
to support jtreg test automation
Posted by Peter Firmstone <ji...@zeus.net.au>.
The difficult part is to grant all permission only to those files listed
below, without making the build process fragile (multiple changes
required for change). This was why I was against specifically listing
jar's in the policy files, however it is an alternative solution. It
seemed easier (long term) to just specify a directory for the platform jars.
Cheers,
Peter.
Jonathan Costers wrote:
> Thanks for the heads up.
> I've been re-reading Peter Jones' posts to refresh my memory, this all makes
> sense.
> I think (hope) we are able to pull this off without any restructuring
> though.
>
> Best
> Jonathan
>
> 2009/9/28 Peter Firmstone (JIRA) <ji...@apache.org>
>
>
>> [
>> https://issues.apache.org/jira/browse/RIVER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760202#action_12760202]
>>
>> Peter Firmstone commented on RIVER-319:
>> ---------------------------------------
>>
>> The jtreg tests have their own security policy files, they don't use the
>> jvm's, currently due to our distribution structure, we have to physically
>> move (not copy or link, they must be moved) the following files to the jvm's
>> extension directory: jre/lib/ext/ where each jtreg test currently grants all
>> permission. This is the reason for your test failures.
>>
>> lib/jsk-platform.jar
>> lib/jsk-resources.jar
>> lib/jsk-lib.jar
>> lib/phoenix-init.jar
>> lib-ext/jsk-policy.jar
>>
>> Then in a directory relative to the one you choose for your working
>> directory, you must add a file named test.props
>>
>> the correct location for test.props:
>>
>>> /**
>>> * Returns extra test properties. Looks for the file
>>>
>> "../../test.props"
>>
>>> * and reads it in as a Properties file. Assuming the working
>>>
>> directory
>>
>>> * is "<path>/JTwork/scratch", this will find "<path>/test.props".
>>> */
>>>
>> My test.props contains the following two lines:
>>
>> jsk.home=/opt/src/river/trunk
>> javatest.maxOutputSize=500000
>>
>> Alternatively all test.props file options can be passed in via jtreg to the
>> jdk using "-Doption=something", jtreg is run in the working directory, we
>> can create an arbitrary one, the example below assumes the suggested build
>> release directory structure above, negating the need to physically move the
>> platform jar files to the jre extensions directory. Note the debug option,
>> it's very helpful. jsk.home could be set as an environment variable in
>> build.xml The command below also assumes the earlier suggestions to modify
>> the jtreg test policy files.
>>
>> $jtreg -verbose:all,nopass -cpa: /opt/src/river/trunk/lib-platform -jdk
>> /usr/jdk/jdk1.5.0_15 -Djsk.home="/opt/src/river/trunk"
>> -Djava.security.debug=access,failure -Djavatest.maxOutputSize=500000
>> ./qa/jtreg/
>>
>> Java 1.5 is currently the preferred test environment for these tests,
>> while Java 1.6 is preferred for the regression tests. In order to get all
>> tests singing on the same jdk we need to implement kerberos client
>> functionality, ktools etc, since these were removed from jdk1.6, this can
>> wait until later though.
>>
>> Cheers,
>>
>> Peter.
>>
>>
>>
>>
>>
>>> Change River Build Dist structure to support jtreg test automation
>>> ------------------------------------------------------------------
>>>
>>> Key: RIVER-319
>>> URL: https://issues.apache.org/jira/browse/RIVER-319
>>> Project: River
>>> Issue Type: Sub-task
>>> Reporter: Peter Firmstone
>>>
>>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>>
>
>
Re: [jira] Commented: (RIVER-319) Change River Build Dist structure
to support jtreg test automation
Posted by Jonathan Costers <jo...@googlemail.com>.
Thanks for the heads up.
I've been re-reading Peter Jones' posts to refresh my memory, this all makes
sense.
I think (hope) we are able to pull this off without any restructuring
though.
Best
Jonathan
2009/9/28 Peter Firmstone (JIRA) <ji...@apache.org>
>
> [
> https://issues.apache.org/jira/browse/RIVER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760202#action_12760202]
>
> Peter Firmstone commented on RIVER-319:
> ---------------------------------------
>
> The jtreg tests have their own security policy files, they don't use the
> jvm's, currently due to our distribution structure, we have to physically
> move (not copy or link, they must be moved) the following files to the jvm's
> extension directory: jre/lib/ext/ where each jtreg test currently grants all
> permission. This is the reason for your test failures.
>
> lib/jsk-platform.jar
> lib/jsk-resources.jar
> lib/jsk-lib.jar
> lib/phoenix-init.jar
> lib-ext/jsk-policy.jar
>
> Then in a directory relative to the one you choose for your working
> directory, you must add a file named test.props
>
> the correct location for test.props:
> >
> > /**
> > * Returns extra test properties. Looks for the file
> "../../test.props"
> > * and reads it in as a Properties file. Assuming the working
> directory
> > * is "<path>/JTwork/scratch", this will find "<path>/test.props".
> > */
>
> My test.props contains the following two lines:
>
> jsk.home=/opt/src/river/trunk
> javatest.maxOutputSize=500000
>
> Alternatively all test.props file options can be passed in via jtreg to the
> jdk using "-Doption=something", jtreg is run in the working directory, we
> can create an arbitrary one, the example below assumes the suggested build
> release directory structure above, negating the need to physically move the
> platform jar files to the jre extensions directory. Note the debug option,
> it's very helpful. jsk.home could be set as an environment variable in
> build.xml The command below also assumes the earlier suggestions to modify
> the jtreg test policy files.
>
> $jtreg -verbose:all,nopass -cpa: /opt/src/river/trunk/lib-platform -jdk
> /usr/jdk/jdk1.5.0_15 -Djsk.home="/opt/src/river/trunk"
> -Djava.security.debug=access,failure -Djavatest.maxOutputSize=500000
> ./qa/jtreg/
>
> Java 1.5 is currently the preferred test environment for these tests,
> while Java 1.6 is preferred for the regression tests. In order to get all
> tests singing on the same jdk we need to implement kerberos client
> functionality, ktools etc, since these were removed from jdk1.6, this can
> wait until later though.
>
> Cheers,
>
> Peter.
>
>
>
>
> > Change River Build Dist structure to support jtreg test automation
> > ------------------------------------------------------------------
> >
> > Key: RIVER-319
> > URL: https://issues.apache.org/jira/browse/RIVER-319
> > Project: River
> > Issue Type: Sub-task
> > Reporter: Peter Firmstone
> >
>
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
[jira] Commented: (RIVER-319) Change River Build Dist structure to
support jtreg test automation
Posted by "Peter Firmstone (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/RIVER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12759972#action_12759972 ]
Peter Firmstone commented on RIVER-319:
---------------------------------------
If I can restructure the Apache River build to package these jar's in another directory to avoid moving them into the jre/lib/ext directory without getting mixed up with the other River (jini) libraries, the whole jtreg test process can be run from an ant target, which would simplify running jtreg tests for new developers.
The (primary) difficulty is getting the security policies to grant AllPermission to those JARs (but not all code, in particular not to the test code) without that location being hard-coded. Security policy files support system property expansion, so perhaps the location could be specified with a system property supplied to jtreg (along with a class path as you had been doing before), although this property might also need to be explicitly propagated to child processes of the tests, like for the tests that start phoenix (rmid).
I'm not sure if there are any other assumptions made by those tests that those JAR files are in the extensions directory, or more to the point, that their classes are defined by the extensions class loader instead of the system class loader-- the only case I can think of is if the dynamic security policy provider is set via a system property. I doubt that there are much if anything issues like that, though-- easiest to discover by trying.
I was thinking something like the following directory structure for apache-river:
configentry - unchanged
doc - unchanged
lib - platform libraries removed (jsk-lib.jar, jsk-platform.jar, jsk-resources.jar)
lib-dl - unchanged
lib-ext - unchanged contains jsk-policy.jar
lib-platform - new directory containing all platform libraries except for jsk-dl.jar, this directory can be the basis for AllPermission
integration-tests-bin - quality assurance integration tests binary release, currently called qa
Then change this in the policy files:
grant codeBase "file:${java.home}/lib/ext/*" {
permission java.security.AllPermission;
};
To (only where required):
grant codeBase "file:${jsk.home}/lib-platform/*"{
permission java.security.AllPermission:
};
grant codeBase "file:${jsk.home}/lib-ext/*"{
permission java.security.AllPermission:
};
grant codeBase "file:${jsk.home}/lib/phoenix-init.jar"{
permission java.security.AllPermission:
};
Then pass jsk.home in via jtreg eg:
$jtreg -verbose:all,nopass -jdk /usr/jdk/jdk1.5.0_15 -Djsk.home="/opt/src/river/trunk" ./qa/jtreg/
Later when I get to ant script stage, jsk.home can be set using ant from an environment variable $JSK_HOME
> Change River Build Dist structure to support jtreg test automation
> ------------------------------------------------------------------
>
> Key: RIVER-319
> URL: https://issues.apache.org/jira/browse/RIVER-319
> Project: River
> Issue Type: Sub-task
> Reporter: Peter Firmstone
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (RIVER-319) Change River Build Dist structure to
support jtreg test automation
Posted by "Jonathan Costers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/RIVER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760126#action_12760126 ]
Jonathan Costers commented on RIVER-319:
----------------------------------------
I have just enabled running the jtreg tests from the Ant build process.
I am currently looking into your suggestions and some other options to solve some of the jtreg test run errors.
> Change River Build Dist structure to support jtreg test automation
> ------------------------------------------------------------------
>
> Key: RIVER-319
> URL: https://issues.apache.org/jira/browse/RIVER-319
> Project: River
> Issue Type: Sub-task
> Reporter: Peter Firmstone
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (RIVER-319) Change River Build Dist structure to
support jtreg test automation
Posted by "Peter Firmstone (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/RIVER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12760202#action_12760202 ]
Peter Firmstone commented on RIVER-319:
---------------------------------------
The jtreg tests have their own security policy files, they don't use the jvm's, currently due to our distribution structure, we have to physically move (not copy or link, they must be moved) the following files to the jvm's extension directory: jre/lib/ext/ where each jtreg test currently grants all permission. This is the reason for your test failures.
lib/jsk-platform.jar
lib/jsk-resources.jar
lib/jsk-lib.jar
lib/phoenix-init.jar
lib-ext/jsk-policy.jar
Then in a directory relative to the one you choose for your working directory, you must add a file named test.props
the correct location for test.props:
>
> /**
> * Returns extra test properties. Looks for the file "../../test.props"
> * and reads it in as a Properties file. Assuming the working directory
> * is "<path>/JTwork/scratch", this will find "<path>/test.props".
> */
My test.props contains the following two lines:
jsk.home=/opt/src/river/trunk
javatest.maxOutputSize=500000
Alternatively all test.props file options can be passed in via jtreg to the jdk using "-Doption=something", jtreg is run in the working directory, we can create an arbitrary one, the example below assumes the suggested build release directory structure above, negating the need to physically move the platform jar files to the jre extensions directory. Note the debug option, it's very helpful. jsk.home could be set as an environment variable in build.xml The command below also assumes the earlier suggestions to modify the jtreg test policy files.
$jtreg -verbose:all,nopass -cpa: /opt/src/river/trunk/lib-platform -jdk /usr/jdk/jdk1.5.0_15 -Djsk.home="/opt/src/river/trunk" -Djava.security.debug=access,failure -Djavatest.maxOutputSize=500000 ./qa/jtreg/
Java 1.5 is currently the preferred test environment for these tests, while Java 1.6 is preferred for the regression tests. In order to get all tests singing on the same jdk we need to implement kerberos client functionality, ktools etc, since these were removed from jdk1.6, this can wait until later though.
Cheers,
Peter.
> Change River Build Dist structure to support jtreg test automation
> ------------------------------------------------------------------
>
> Key: RIVER-319
> URL: https://issues.apache.org/jira/browse/RIVER-319
> Project: River
> Issue Type: Sub-task
> Reporter: Peter Firmstone
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (RIVER-319) Change River Build Dist structure to
support jtreg test automation
Posted by "Sim IJskes (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/RIVER-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sim IJskes updated RIVER-319:
-----------------------------
Component/s: build
> Change River Build Dist structure to support jtreg test automation
> ------------------------------------------------------------------
>
> Key: RIVER-319
> URL: https://issues.apache.org/jira/browse/RIVER-319
> Project: River
> Issue Type: Sub-task
> Components: build
> Reporter: Peter Firmstone
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.