running tests from OASIS

[kelvin goodson <kg...@thegoodsons.org.uk>]

Hi

after a long time away from Tuscany I'm in a position to start contributing
again.  I'm planning to get involved in testing SCA against the OASIS specs.
To that end I'm going to have a little play in my sandbox with svn:externals
and whatever other tricks I can discover to investigate whether we
can/should auto extract the OASIS tests on a Tuscany svn extract.

Kelvin.

Re: running tests from OASIS

[kelvin goodson <ke...@gmail.com>]

>
> ...snip
>
> Hi Kelvin
>
> Sounds good to me. I haven't updated and tried this yet but am looking
> at svn. Can you say something about how the Tuscany framework code,
> that makes the generic OASIS tests work, will be arranged?


just looking at that now ..

>
>
> Simon
>

Re: running tests from OASIS

[ant elder <an...@gmail.com>]

On Wed, May 6, 2009 at 11:45 AM, kelvin goodson <ke...@gmail.com> wrote:

> One thing I'm a little unsure of is whether there are any legal issues
> with automatically sucking the source out of the OASIS repo into your
> local build environment using the svn:externals property.  Does anyone
> have any insights on this?
>

I think we can assume its ok. AFAICT the intention is that they're
covered by the OASIS license and so qualify as read-only standards
documents which we can legally redistribute in both source and binary
so the svn external is ok. The license is still the one that assumes
its embedded in a spec document, it would be better if there was one
tailored for use with source files not included in the specs.

   ...ant

Re: running tests from OASIS

[kelvin goodson <ke...@gmail.com>]

The OASIS source repository now has a bridge interface to allow the
tests to invoke a selected runtime via runtime specific
implementations of the bridge interface. It also has a bridge
implementation for Tuscany there; the idea being that any implementor
of a runtime can have their bridge implementation hosted in the OASIS
source repo.  So all you need to do is set the test environment to
know which bridge implementation to use by setting a system property
or environment variable to the implementation class name.

Below are some instructions on how to run the current setup.  Once
this has settled down I'll add the instructions to the website.
I'm currently seeing 113 tests run, of which 40 fail.
I'm planning to start working through these to update the wiki
conformance page [1] to reflect our conformance status.

One thing I'm a little unsure of is whether there are any legal issues
with automatically sucking the source out of the OASIS repo into your
local build environment using the svn:externals property.  Does anyone
have any insights on this?

------------------------------------

Oasis Test Execution Documentation

To run the OASIS tests against Tuscany

- extract/update your 2.x source from svn, this will automatically
fetch the test code from the OASIS svn repository if not already
extracted
- change directory to otest/current
- ensure that your runtime environment has either an environment
variable or a Java System Property named
OASIS_TESTENV_RUNTIME_BRIDGE_CLASS set to client.TuscanyRuntimeBridge
- run mvn

Note that this will run the tests against the most recent version of
the OASIS tests,  and changes in the OASIS structure may cause
instability in this build process.  An alternative directory
otest/fixed_version is set to extract a fixed version of the OASIS svn
repository subtree into your extracted source,  the idea being that
the svn:externals property that causes this extraction to occur is
periodically updated to use a repository version id that is known to
work. Just follow the above steps in the otest/fixed_version directory
to run the tests in this way.

You can of course run mvn:eclipse:eclipse in either of these
directories and run the tests individually from the eclipse
environment. Note however that the test suite has the notion of source
as a resource.  To fix this in maven I have used the build helper
plugin to add a source directory.  To do this in eclipse you'll need
to add a source path of src/main resources/General to your eclipse
build properties for the project.



-----------------------------------------------

[1] http://cwiki.apache.org/confluence/display/TUSCANYWIKI/Conformance+Testing


===================================================

2009/4/24 Simon Laws <si...@googlemail.com>
>
> On Fri, Apr 24, 2009 at 10:51 AM, kelvin goodson <kg...@thegoodsons.org.uk> wrote:
> > I've just added the start of something that will make the otests available
> > to all. Under the new otest directory is a current and a fixed_version
> > directory. We can't rely on the foreign svn repository that holds these
> > tests not changing significantly (that wouldn't be a risk to the build since
> > I don't have any intention of hooking this into the build at the moment).
> >
> > The "current" directory has an svn:externals property set which does an
> > unqualified extract of the otests from the foreign svn repo.  The
> > "fixed_version" directory does the same svn:externals extract,  but
> > qualified with a revision number of the foreign repo. That way any one
> > wanting to have a play with these tests can go to an informally blessed
> > version. At the time of writing these work out to be identical.
> >
> > Without having dug very far at the moment I see 84 tests running, with 40
> > failures and 1 error.  I'll work to see how many of these are just down to
> > an imature tuscany infrastructure and how many are real failures/errors.
> > I'll then look at ensuring we have JIRAs to accompany the issues.
> >
> ...snip
>
> Hi Kelvin
>
> Sounds good to me. I haven't updated and tried this yet but am looking
> at svn. Can you say something about how the Tuscany framework code,
> that makes the generic OASIS tests work, will be arranged?
>
> Simon

Re: running tests from OASIS

[Simon Laws <si...@googlemail.com>]

On Fri, Apr 24, 2009 at 10:51 AM, kelvin goodson <kg...@thegoodsons.org.uk> wrote:
> I've just added the start of something that will make the otests available
> to all. Under the new otest directory is a current and a fixed_version
> directory. We can't rely on the foreign svn repository that holds these
> tests not changing significantly (that wouldn't be a risk to the build since
> I don't have any intention of hooking this into the build at the moment).
>
> The "current" directory has an svn:externals property set which does an
> unqualified extract of the otests from the foreign svn repo.  The
> "fixed_version" directory does the same svn:externals extract,  but
> qualified with a revision number of the foreign repo. That way any one
> wanting to have a play with these tests can go to an informally blessed
> version. At the time of writing these work out to be identical.
>
> Without having dug very far at the moment I see 84 tests running, with 40
> failures and 1 error.  I'll work to see how many of these are just down to
> an imature tuscany infrastructure and how many are real failures/errors.
> I'll then look at ensuring we have JIRAs to accompany the issues.
>
...snip

Hi Kelvin

Sounds good to me. I haven't updated and tried this yet but am looking
at svn. Can you say something about how the Tuscany framework code,
that makes the generic OASIS tests work, will be arranged?

Simon

Re: running tests from OASIS

[kelvin goodson <kg...@thegoodsons.org.uk>]

I've just added the start of something that will make the otests available
to all. Under the new otest directory is a current and a fixed_version
directory. We can't rely on the foreign svn repository that holds these
tests not changing significantly (that wouldn't be a risk to the build since
I don't have any intention of hooking this into the build at the moment).

The "current" directory has an svn:externals property set which does an
unqualified extract of the otests from the foreign svn repo.  The
"fixed_version" directory does the same svn:externals extract,  but
qualified with a revision number of the foreign repo. That way any one
wanting to have a play with these tests can go to an informally blessed
version. At the time of writing these work out to be identical.

Without having dug very far at the moment I see 84 tests running, with 40
failures and 1 error.  I'll work to see how many of these are just down to
an imature tuscany infrastructure and how many are real failures/errors.
I'll then look at ensuring we have JIRAs to accompany the issues.

Kelvin.

2009/4/22 kelvin goodson <kg...@thegoodsons.org.uk>

> Hi
>
> after a long time away from Tuscany I'm in a position to start contributing
> again.  I'm planning to get involved in testing SCA against the OASIS specs.
> To that end I'm going to have a little play in my sandbox with svn:externals
> and whatever other tricks I can discover to investigate whether we
> can/should auto extract the OASIS tests on a Tuscany svn extract.
>
> Kelvin.
>

Re: running tests from OASIS

[Simon Laws <si...@googlemail.com>]

Ah, I see, thanks for the link Kelvin.

Simon

Re: running tests from OASIS

[kelvin goodson <ke...@gmail.com>]

Tests can be viewed here
http://tools.oasis-open.org/version-control/svn/sca-assembly/TestCases
or extracted here
svn://tools.oasis-open.org/version-control/svn/sca-assembly/TestCases

Kelvin.

2009/4/22 Simon Laws <si...@googlemail.com>

> On Wed, Apr 22, 2009 at 11:05 AM, kelvin goodson <kg...@thegoodsons.org.uk>
> wrote:
> > Hi
> >
> > after a long time away from Tuscany I'm in a position to start
> contributing
> > again.  I'm planning to get involved in testing SCA against the OASIS
> specs.
> > To that end I'm going to have a little play in my sandbox with
> svn:externals
> > and whatever other tricks I can discover to investigate whether we
> > can/should auto extract the OASIS tests on a Tuscany svn extract.
> >
> > Kelvin.
> >
>
> Welcome back Kelvin!
>
> re. OASIS tests, that would be really useful. It will be a great help
> to us in getting a grip on what we need to fix in 2.x to be OASIS
> compatible. Could also be useful for 1.x for those tests that are the
> same between 1.x and 2.x (I assuming that there are quite a few).
>
> Remind me where in OASIS the tests are stored?
>
> Simon
>

Re: running tests from OASIS

[Simon Laws <si...@googlemail.com>]

On Wed, Apr 22, 2009 at 11:05 AM, kelvin goodson <kg...@thegoodsons.org.uk> wrote:
> Hi
>
> after a long time away from Tuscany I'm in a position to start contributing
> again.  I'm planning to get involved in testing SCA against the OASIS specs.
> To that end I'm going to have a little play in my sandbox with svn:externals
> and whatever other tricks I can discover to investigate whether we
> can/should auto extract the OASIS tests on a Tuscany svn extract.
>
> Kelvin.
>

Welcome back Kelvin!

re. OASIS tests, that would be really useful. It will be a great help
to us in getting a grip on what we need to fix in 2.x to be OASIS
compatible. Could also be useful for 1.x for those tests that are the
same between 1.x and 2.x (I assuming that there are quite a few).

Remind me where in OASIS the tests are stored?

Simon

Re: running tests from OASIS

[kelvin goodson <ke...@gmail.com>]

thanks ant .. it's good to be back :)


2009/4/22 ant elder <an...@gmail.com>

>
>
> On Wed, Apr 22, 2009 at 11:05 AM, kelvin goodson <kg...@thegoodsons.org.uk>wrote:
>
>> Hi
>>
>> after a long time away from Tuscany I'm in a position to start
>> contributing again.  I'm planning to get involved in testing SCA against the
>> OASIS specs. To that end I'm going to have a little play in my sandbox with
>> svn:externals and whatever other tricks I can discover to investigate
>> whether we can/should auto extract the OASIS tests on a Tuscany svn extract.
>>
>> Kelvin.
>>
>
> Woohoo, nice to see you back Kelven. If there's anything i can do to help
> you get back up to speed just ask.
>
>    ...ant
>
>

Re: running tests from OASIS

[ant elder <an...@gmail.com>]

On Wed, Apr 22, 2009 at 11:05 AM, kelvin goodson <kg...@thegoodsons.org.uk>wrote:

> Hi
>
> after a long time away from Tuscany I'm in a position to start contributing
> again.  I'm planning to get involved in testing SCA against the OASIS specs.
> To that end I'm going to have a little play in my sandbox with svn:externals
> and whatever other tricks I can discover to investigate whether we
> can/should auto extract the OASIS tests on a Tuscany svn extract.
>
> Kelvin.
>

Woohoo, nice to see you back Kelven. If there's anything i can do to help
you get back up to speed just ask.

   ...ant