You are viewing a plain text version of this content. The canonical link for it is here.
Posted to muse-dev@ws.apache.org by "Balan Subramanian (JIRA)" <ji...@apache.org> on 2007/07/10 18:00:04 UTC
[jira] Created: (MUSE-244) Test/compliance framework for Muse
Test/compliance framework for Muse
----------------------------------
Key: MUSE-244
URL: https://issues.apache.org/jira/browse/MUSE-244
Project: Muse
Issue Type: New Feature
Components: Other
Reporter: Balan Subramanian
Assignee: Dan Jemiolo
Fix For: 2.3.0
A test framework is required for Muse. This framework will:
1. help in testing - individually and in combination - the various WSDM capabilities implemented in Muse
2. help users test their own endpoints
3. help run stress tests
4. Manage deployments on various target platforms and their combinations (OSGi (Equinox, others), J2EE (WAS or Tomcat) each with either axis2 or mini) for different operating systems and JDK versions
5. Provide ability to automatically executing test suites aimed at 1) endpoint function 2) standard implementation testing 3) compliance testing using the SOAP messages
There have been a number of conversations about such a framework on the mailing list led primarily by Andrew Eberbach. Those will be collected and consolidated under this feature requirement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-dev-help@ws.apache.org
[jira] Issue Comment Edited: (MUSE-244) Test/compliance framework
for Muse
Posted by "Kam K. Yee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MUSE-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517020 ]
Kam K. Yee edited comment on MUSE-244 at 8/1/07 10:21 AM:
----------------------------------------------------------
======================================
PROPOSED IMPLEMENTATION - HIGH LEVEL
======================================
Following are some initial thoughts/proposal for the implementation of the various tests and framework. (This proposal incorporates and adds to the input from Andrew Eberbach's previous postings and email exchanges (Thanks! Andrew).
- The unit, functional, and stress tests will be JUnit based.
- ANT scripts will be created to setup and invoke these JUnit test suites.
- Conformance tests:
- The WS-I Organization (http://ws-i.org) has developed and made available the WS-I Interoperability Testing Tools to help developers determine whether their Web Services are conformant with WS-I profile guidelines. (More details on these tools and how they work can be found at http://ws-i.org/deliverables/workinggroup.aspx?wg=testingtools and http://www-128.ibm.com/developerworks/webservices/library/ws-wsitest/). The latest version is designed to test the Basic Profile 1.1 and SSBP 1.0. We will extended the testing tools to test Muse for conformance against the latest supported specification levels (WSRF 1.2, WSN 1.3, WSDM 1.1, and WS-MetadataExchange).
- End To End Test Automation:
- For end to end test automation, STAF/STAX seems to be a good automation framework to use and build upon. (More details on STAF/STAX can be found at http://staf.sourceforge.net/index.php).
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework designed around the idea of services (such as process invocation, resource management, logging, and monitoring). STAF removes the tedium of building an automation infrastructure, thus enabling testers to focus on building the automation solution. The STAF framework provides the foundation upon which to build higher level solutions, and provides a pluggable approach supported across a large variety of platforms and languages.
STAX is an execution engine which can help with thoroughly automating the distribution, execution, and results analysis of testcases. STAX builds on top of three existing technologies, STAF, XML, and Python, to place great automation power in the hands of testers. STAX also provides a GUI monitoring application which allows one to interact with and monitor the progress of jobs.
STAF/STAX will be used to automatically invoke and orchestrate the invocation of the ANT scripts upon receiving a notification (such as an email notifiying that a build has completed).
UNIT TESTING:
- Each top-level muse component (such as muse-core, muse-tools, etc.) will have a corresponding "-test" directory (muse-core-test, muse-tools-test, etc). This directory will have a structure mirroring the component it is testing (as is common practice with JUnit). Each non-trivial class will have a corresponding test suite and the test suites would be self-contained within the project. This is so that users don't have to check out several "-test" directories in order to run one (dependencies become very tedious, very quickly).
- A new target will be introduced to the build.xml to build a new archive (muse-test-unit-bin) containing only the unit test code and needed artifacts (e.g., WSDLs & XSDs).
- The identified steps for unit testing are as follows:
1. Download or locally build muse-bin runtime archive
2. Download or locally build muse-test-unit-bin archive
3. Unzip muse-bin
4. Unzip muse-test-unit-bin
5. Run muse-bin code generation tool to generate an endpoint with all capabilities for the target container
6. Deploy generated endpoint to the target container
7. Startup target container with generated endpoint
8. Invoke execution of JUnit unit test suites
- A new unit test ANT script will be created to invoke and execute the different JUnit test suites.
Later on when full automation is introduced, the automation framework will be invoking these ANT script target to automatically execute the test suites.
- This design also allows for developers to do a sanity check by running these tests locally on their machines prior to checking code into the repository.
FUNCTIONAL TESTING:
- This would loosely follow Dan's current approach of top-level directories that are related to specific capabilties (wsrf, wsn, etc). The structure here isn't as easy to generalize, but some directory/hierarchical organization that seems sane would suffice. Again, the goal here is self-containment as to avoid an overly complex test environment.
- A new target will be introduced to the build.xml to build a new archive (muse-test-fvt-bin) containing only the functional test code and needed artifacts (e.g., WSDLs & XSDs).
- The identified steps for functional testing are as follows:
1. Download or locally build muse-bin runtime archive
2. Download or locally build muse-test-fvt-bin archive
3. Unzip muse-bin
4. Unzip muse-test-fvt-bin
5. Run muse-bin code generation tool to generate endpoint(s) to be deployed to target container
6. Deploy generated endpoint(s) to the target container
7. Startup target container with generated endpoint(s)
8. Invoke execution of JUnit functional test suites
- A new functional test ANT script will be created to invoke and execute the different JUnit test suites.
Later on when full automation is introduced, the automation framework will be invoking these ANT script target to automatically execute the test suites.
- This design also allows for developers to do a sanity check by running these tests locally on their machines prior to checking code into the repository.
Finally, since modifying something simple is usually easier than generating something simple from scratch, we will create two "read only" top-level directories which would lay out a reasonable skeleton for users contributing their own junit and functional tests for top-level directories.
was:
======================================
PROPOSED IMPLEMENTATION - HIGH LEVEL
======================================
Following are some initial thoughts/proposal for the implementation of the various tests and framework.
(This proposal incorporates and adds to the input from Andrew Eberbach's previous postings and email exchanges (Thanks! Andrew).
- The unit, functional, and stress tests will be JUnit based.
- ANT scripts will be created to setup and invoke these JUnit test suites.
- Conformance tests:
- The WS-I Organization (http://ws-i.org) has developed and made available the WS-I Interoperability Testing Tools to help developers determine whether their Web Services
are conformant with WS-I profile guidelines. (More details on these tools and how they work can be found at http://ws-i.org/deliverables/workinggroup.aspx?wg=testingtools
and http://www-128.ibm.com/developerworks/webservices/library/ws-wsitest/). The latest version is designed to test the Basic Profile 1.1 and SSBP 1.0. We will extended
the testing tools to test Muse for conformance against the latest supported specification levels (WSRF 1.2, WSN 1.3, WSDM 1.1, and WS-MetadataExchange).
- End To End Test Automation:
- For end to end test automation, STAF/STAX seems to be a good automation framework to use and build upon. (More details on STAF/STAX can be found at
http://staf.sourceforge.net/index.php).
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework designed around the idea of services (such as process
invocation, resource management, logging, and monitoring). STAF removes the tedium of building an automation infrastructure, thus enabling testers to focus on building the
automation solution. The STAF framework provides the foundation upon which to build higher level solutions, and provides a pluggable approach supported across a large
variety of platforms and languages.
STAX is an execution engine which can help with thoroughly automating the distribution, execution, and results analysis of testcases. STAX builds on top of three
existing technologies, STAF, XML, and Python, to place great automation power in the hands of testers. STAX also provides a GUI monitoring application which allows one to
interact with and monitor the progress of jobs.
STAF/STAX will be used to automatically invoke and orchestrate the invocation of the ANT scripts upon receiving a notification (such as an email notifiying that a build
has completed).
UNIT TESTING:
- Each top-level muse component (such as muse-core, muse-tools, etc.) will have a corresponding "-test" directory (muse-core-test, muse-tools-test, etc). This
directory will have a structure mirroring the component it is testing (as is common practice with JUnit). Each non-trivial class will have a corresponding test suite and
the test suites would be self-contained within the project. This is so that users don't have to check out several "-test" directories in order to run one (dependencies
become very tedious, very quickly).
- A new target will be introduced to the build.xml to build a new archive (muse-test-unit-bin) containing only the unit test code and needed artifacts (e.g., WSDLs &
XSDs).
- The identified steps for unit testing are as follows:
1. Download or locally build muse-bin runtime archive
2. Download or locally build muse-test-unit-bin archive
3. Unzip muse-bin
4. Unzip muse-test-unit-bin
5. Run muse-bin code generation tool to generate an endpoint with all capabilities for the target container
6. Deploy generated endpoint to the target container
7. Startup target container with generated endpoint
8. Invoke execution of JUnit unit test suites
- A new unit test ANT script will be created to invoke and execute the different JUnit test suites.
Later on when full automation is introduced, the automation framework will be invoking these ANT script target to automatically execute the test suites.
- This design also allows for developers to do a sanity check by running these tests locally on their machines prior to checking code into the repository.
FUNCTIONAL TESTING:
- This would loosely follow Dan's current approach of top-level directories that are related to specific capabilties (wsrf, wsn, etc). The structure here isn't as easy to
generalize, but some directory/hierarchical organization that seems sane would suffice. Again, the goal here is self-containment as to avoid an overly complex test
environment.
- A new target will be introduced to the build.xml to build a new archive (muse-test-fvt-bin) containing only the functional test code and needed artifacts (e.g., WSDLs &
XSDs).
- The identified steps for functional testing are as follows:
1. Download or locally build muse-bin runtime archive
2. Download or locally build muse-test-fvt-bin archive
3. Unzip muse-bin
4. Unzip muse-test-fvt-bin
5. Run muse-bin code generation tool to generate endpoint(s) to be deployed to target container
6. Deploy generated endpoint(s) to the target container
7. Startup target container with generated endpoint(s)
8. Invoke execution of JUnit functional test suites
- A new functional test ANT script will be created to invoke and execute the different JUnit test suites.
Later on when full automation is introduced, the automation framework will be invoking these ANT script target to automatically execute the test suites.
- This design also allows for developers to do a sanity check by running these tests locally on their machines prior to checking code into the repository.
Finally, since modifying something simple is usually easier than generating something simple from scratch, we will create two "read only" top-level directories which would lay out a reasonable skeleton for users contributing their own junit and functional tests for top-level directories.
> Test/compliance framework for Muse
> ----------------------------------
>
> Key: MUSE-244
> URL: https://issues.apache.org/jira/browse/MUSE-244
> Project: Muse
> Issue Type: New Feature
> Components: Other
> Reporter: Balan Subramanian
> Assignee: Dan Jemiolo
> Fix For: 2.3.0
>
>
> A test framework is required for Muse. This framework will:
> 1. help in testing - individually and in combination - the various WSDM capabilities implemented in Muse
> 2. help users test their own endpoints
> 3. help run stress tests
> 4. Manage deployments on various target platforms and their combinations (OSGi (Equinox, others), J2EE (WAS or Tomcat) each with either axis2 or mini) for different operating systems and JDK versions
> 5. Provide ability to automatically executing test suites aimed at 1) endpoint function 2) standard implementation testing 3) compliance testing using the SOAP messages
> There have been a number of conversations about such a framework on the mailing list led primarily by Andrew Eberbach. Those will be collected and consolidated under this feature requirement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-dev-help@ws.apache.org
[jira] Commented: (MUSE-244) Test/compliance framework for Muse
Posted by "Kam K. Yee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MUSE-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517019 ]
Kam K. Yee commented on MUSE-244:
---------------------------------
This feature request encompases/requests multiple functionality which can be broken down into more individual, granular items:
- A flexible infrastructure and initial, extensible suite of unit tests
- A flexible infrastructure and initial, extensible suite of functional tests
- A flexible infrastructure and initial, extensible suite of stress tests
- A flexible infrastructure and initial, extensible suite of conformance tests, and
- A flexible infrastructure that will automatically set-up and execute the set of tests listed above on the supported platforms and levels of pre-reqs.
I'd like to propose splitting up this feature request into sub-features or individual features altogether.
Being new to this project/process, I'm not sure which approach will normally be followed.
Any comments on what the better approach is?
> Test/compliance framework for Muse
> ----------------------------------
>
> Key: MUSE-244
> URL: https://issues.apache.org/jira/browse/MUSE-244
> Project: Muse
> Issue Type: New Feature
> Components: Other
> Reporter: Balan Subramanian
> Assignee: Dan Jemiolo
> Fix For: 2.3.0
>
>
> A test framework is required for Muse. This framework will:
> 1. help in testing - individually and in combination - the various WSDM capabilities implemented in Muse
> 2. help users test their own endpoints
> 3. help run stress tests
> 4. Manage deployments on various target platforms and their combinations (OSGi (Equinox, others), J2EE (WAS or Tomcat) each with either axis2 or mini) for different operating systems and JDK versions
> 5. Provide ability to automatically executing test suites aimed at 1) endpoint function 2) standard implementation testing 3) compliance testing using the SOAP messages
> There have been a number of conversations about such a framework on the mailing list led primarily by Andrew Eberbach. Those will be collected and consolidated under this feature requirement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-dev-help@ws.apache.org
[jira] Commented: (MUSE-244) Test/compliance framework for Muse
Posted by "Kam K. Yee (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MUSE-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517020 ]
Kam K. Yee commented on MUSE-244:
---------------------------------
======================================
PROPOSED IMPLEMENTATION - HIGH LEVEL
======================================
Following are some initial thoughts/proposal for the implementation of the various tests and framework.
(This proposal incorporates and adds to the input from Andrew Eberbach's previous postings and email exchanges (Thanks! Andrew).
- The unit, functional, and stress tests will be JUnit based.
- ANT scripts will be created to setup and invoke these JUnit test suites.
- Conformance tests:
- The WS-I Organization (http://ws-i.org) has developed and made available the WS-I Interoperability Testing Tools to help developers determine whether their Web Services
are conformant with WS-I profile guidelines. (More details on these tools and how they work can be found at http://ws-i.org/deliverables/workinggroup.aspx?wg=testingtools
and http://www-128.ibm.com/developerworks/webservices/library/ws-wsitest/). The latest version is designed to test the Basic Profile 1.1 and SSBP 1.0. We will extended
the testing tools to test Muse for conformance against the latest supported specification levels (WSRF 1.2, WSN 1.3, WSDM 1.1, and WS-MetadataExchange).
- End To End Test Automation:
- For end to end test automation, STAF/STAX seems to be a good automation framework to use and build upon. (More details on STAF/STAX can be found at
http://staf.sourceforge.net/index.php).
The Software Testing Automation Framework (STAF) is an open source, multi-platform, multi-language framework designed around the idea of services (such as process
invocation, resource management, logging, and monitoring). STAF removes the tedium of building an automation infrastructure, thus enabling testers to focus on building the
automation solution. The STAF framework provides the foundation upon which to build higher level solutions, and provides a pluggable approach supported across a large
variety of platforms and languages.
STAX is an execution engine which can help with thoroughly automating the distribution, execution, and results analysis of testcases. STAX builds on top of three
existing technologies, STAF, XML, and Python, to place great automation power in the hands of testers. STAX also provides a GUI monitoring application which allows one to
interact with and monitor the progress of jobs.
STAF/STAX will be used to automatically invoke and orchestrate the invocation of the ANT scripts upon receiving a notification (such as an email notifiying that a build
has completed).
UNIT TESTING:
- Each top-level muse component (such as muse-core, muse-tools, etc.) will have a corresponding "-test" directory (muse-core-test, muse-tools-test, etc). This
directory will have a structure mirroring the component it is testing (as is common practice with JUnit). Each non-trivial class will have a corresponding test suite and
the test suites would be self-contained within the project. This is so that users don't have to check out several "-test" directories in order to run one (dependencies
become very tedious, very quickly).
- A new target will be introduced to the build.xml to build a new archive (muse-test-unit-bin) containing only the unit test code and needed artifacts (e.g., WSDLs &
XSDs).
- The identified steps for unit testing are as follows:
1. Download or locally build muse-bin runtime archive
2. Download or locally build muse-test-unit-bin archive
3. Unzip muse-bin
4. Unzip muse-test-unit-bin
5. Run muse-bin code generation tool to generate an endpoint with all capabilities for the target container
6. Deploy generated endpoint to the target container
7. Startup target container with generated endpoint
8. Invoke execution of JUnit unit test suites
- A new unit test ANT script will be created to invoke and execute the different JUnit test suites.
Later on when full automation is introduced, the automation framework will be invoking these ANT script target to automatically execute the test suites.
- This design also allows for developers to do a sanity check by running these tests locally on their machines prior to checking code into the repository.
FUNCTIONAL TESTING:
- This would loosely follow Dan's current approach of top-level directories that are related to specific capabilties (wsrf, wsn, etc). The structure here isn't as easy to
generalize, but some directory/hierarchical organization that seems sane would suffice. Again, the goal here is self-containment as to avoid an overly complex test
environment.
- A new target will be introduced to the build.xml to build a new archive (muse-test-fvt-bin) containing only the functional test code and needed artifacts (e.g., WSDLs &
XSDs).
- The identified steps for functional testing are as follows:
1. Download or locally build muse-bin runtime archive
2. Download or locally build muse-test-fvt-bin archive
3. Unzip muse-bin
4. Unzip muse-test-fvt-bin
5. Run muse-bin code generation tool to generate endpoint(s) to be deployed to target container
6. Deploy generated endpoint(s) to the target container
7. Startup target container with generated endpoint(s)
8. Invoke execution of JUnit functional test suites
- A new functional test ANT script will be created to invoke and execute the different JUnit test suites.
Later on when full automation is introduced, the automation framework will be invoking these ANT script target to automatically execute the test suites.
- This design also allows for developers to do a sanity check by running these tests locally on their machines prior to checking code into the repository.
Finally, since modifying something simple is usually easier than generating something simple from scratch, we will create two "read only" top-level directories which would lay out a reasonable skeleton for users contributing their own junit and functional tests for top-level directories.
> Test/compliance framework for Muse
> ----------------------------------
>
> Key: MUSE-244
> URL: https://issues.apache.org/jira/browse/MUSE-244
> Project: Muse
> Issue Type: New Feature
> Components: Other
> Reporter: Balan Subramanian
> Assignee: Dan Jemiolo
> Fix For: 2.3.0
>
>
> A test framework is required for Muse. This framework will:
> 1. help in testing - individually and in combination - the various WSDM capabilities implemented in Muse
> 2. help users test their own endpoints
> 3. help run stress tests
> 4. Manage deployments on various target platforms and their combinations (OSGi (Equinox, others), J2EE (WAS or Tomcat) each with either axis2 or mini) for different operating systems and JDK versions
> 5. Provide ability to automatically executing test suites aimed at 1) endpoint function 2) standard implementation testing 3) compliance testing using the SOAP messages
> There have been a number of conversations about such a framework on the mailing list led primarily by Andrew Eberbach. Those will be collected and consolidated under this feature requirement.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: muse-dev-unsubscribe@ws.apache.org
For additional commands, e-mail: muse-dev-help@ws.apache.org