Cross language interop testing

[Simon Laws <si...@googlemail.com>]

A long time ago I posted about doing some interoperability testing. I've got
back to this now and I've put some ideas about how we do this on the wiki,
here (http://wiki.apache.org/ws/Tuscany/Interop).

At a high level we have two products to consider for interoperability
testing, SDO and SCA. This testing should be performed across all
implementations, i.e. Java, C++ and PHP. As it stands the only available SCA
implementation that can talk to anything else is Java (C++ is nearly there
with Axis2 integration going on as we speak) so we can skip the SCA
interoperability conversation for the time being. That brings us to SDO.

SDO interoperability involves successfully moving SDO data (instance, model
and/or change history data) from one implementation of SDO to another. The
"scenarios" section gives a few ideas about why we might want to do this.
This is not intended to be a complete list so any more SDO interoperability
scenarios you can think of are gratefully accepted.

We have Java, C++ and PHP implementations of SDO to date but the features
available vary slightly across these implementations. From an
interoperability point of view Kelvin and myself talked offline and listed
the features that seemed important in the "Features" section. You will note
that we marked which implementations currently support which features. Is
this assessment correct? The order column simply indicates which order the
tests could be constructed in, for example, to get my head round this
process I have made a start on the test which involves reading an XML
document and writing it out again via the XMLHelper/DAS (item 1.1). This
seems like an obvious place to start while, for example, the Axiom
serialization code is note done yet in C++ so it would be difficult to start
with that.

In general the approach to this testing can be fairly simple I believe. For
each implementation we write a test program that performs the required test,
e.g. read and write an XML file. Then we compare:

A/ the input with the output to check that it works for an individual
implantation
B/ the output of all of the implementations to check interoperability. If
all of the inputs and outputs in A match there should be no issue here.

So if we lay these tests out in order I would expect to do the following in
Java, C++ and PHP where the feature is supported

1.1.  Read an XML file and write it out again

1.2.  Read an XML file and write out an XSD file

2.1  Read and write Records from a database

3.1  Read XML and write Axiom

3.2  Read Axiom and write XML

4   Serialize XML from/To SDO – I'm not sure that this is different
from 1.1at present as I seen documents mention it but haven't tried it

1.3 Read an XML file and make changes before writing it back out again.
Changes to be made through both dynamic and generated interfaces

4.2 Read from a database and make changes before writing is back out again.
Changes to be made through both dynamic and generated interfaces

As I mentioned previously I've had a go at 1.1. This first involved creating
XSD files to include all of the type mappings defined in the SDO
2.0.1specification. I have this now and am able to pass it in and out
of SDO in
Java and C++. PHP is still To Do. The results are included in the "schema
feature" table and I will raise JIRA as I investigate what I have found in
more detail.

Now I have the XML schema the other XML tests become easier as, to a great
extent, they are just variations on a theme.  I will have to generate a
relational schema to match of course.

So, is this going to work and is it sufficient? Any thoughts you have on any
of this are gratefully received on the mailing list. It may be useful for us
to have an IRC chat based on feedback to discuss the best way forward. I
will post with details depending on what feedback I get.

An unresolved issue is what to do with the test code and files that this
process will generate. The test code will happily live within each project
somewhere (where is a good place in each project structure?), for example,
it could be

Java: tuscany/java/sdo/impl/src/test/org/apache/Tuscany/sdo/interop
C++: tuscany/cpp/sdo/runtime/core/interop
PHP: tuscanyphp/sdo-1.0.1/tests/interop

The input files themselves are common across all projects as are the results
of cross implementation file comparisons. So where should they live. Maybe
the thing to do is just pick on project, C++ or Java, and put the common
files there.

We have a UK public holiday on Monday but I'll be back to this on Tuesday.

Re: Cross language interop testing

[Simon Laws <si...@googlemail.com>]

On 5/30/06, Geoffrey Winn <ge...@googlemail.com> wrote:
>
> Simon,
>
> On 26/05/06, Simon Laws <si...@googlemail.com> wrote:
> >
> > A long time ago I posted about doing some interoperability testing. I've
> > got
> > back to this now and I've put some ideas about how we do this on the
> wiki,
> > here (http://wiki.apache.org/ws/Tuscany/Interop).
>
>
> ...
>
> So if we lay these tests out in order I would expect to do the following
> in
> > Java, C++ and PHP where the feature is supported
> >
> > 1.1.  Read an XML file and write it out again
> >
> > 1.2.  Read an XML file and write out an XSD file
> >
> > 2.1  Read and write Records from a database
> >
> > 3.1  Read XML and write Axiom
> >
> > 3.2  Read Axiom and write XML
>
>
> I see the sense in 1.1 However, I'm not so sure about 1.2. Presumably that
> is saying that we read an XML instance and in the process of creating an
> SDO
> for it we have also defined a variety of types, properties and so on to
> describe it. Then we use that information to generate an XSD. This is
> pretty
> much the same as inspecting an XML instance document and then writing an
> XSD
> for it. It isn't obvious to me that different implementations would
> necessarily generate the same XSD in these cases. I'm willing to be
> convinced; I'm just not sure that it's fair to expect that different
> implementations will generate the same XSD when what they were doing was
> populating a model with whatever they needed to read an XML instance.
>
> Regards
>
> Geoff.



Yes, you have spotted a slip up here I think. This is an interesting test of
schema free XML loading but I'm told this doesn't work in PHP so probably
doesn't in C++. The spec implies that you can do this in Java although I
have to admit to not having tried it yet. As its difficult to test
interoperability with one implementation we could replace "1.2.  Read an XML
file and write out an XSD file" with "1.2.  Generate metadata statically in
code and write out an XSD file"  and test the case where the type model
comes from elsewhere prior to XSD generation. What we are testing here is
each implementations' type model to XSD mapping to see if they are
compatible.

Re: Cross language interop testing

[Geoffrey Winn <ge...@googlemail.com>]

Simon,

On 26/05/06, Simon Laws <si...@googlemail.com> wrote:
>
> A long time ago I posted about doing some interoperability testing. I've
> got
> back to this now and I've put some ideas about how we do this on the wiki,
> here (http://wiki.apache.org/ws/Tuscany/Interop).


...

So if we lay these tests out in order I would expect to do the following in
> Java, C++ and PHP where the feature is supported
>
> 1.1.  Read an XML file and write it out again
>
> 1.2.  Read an XML file and write out an XSD file
>
> 2.1  Read and write Records from a database
>
> 3.1  Read XML and write Axiom
>
> 3.2  Read Axiom and write XML


I see the sense in 1.1 However, I'm not so sure about 1.2. Presumably that
is saying that we read an XML instance and in the process of creating an SDO
for it we have also defined a variety of types, properties and so on to
describe it. Then we use that information to generate an XSD. This is pretty
much the same as inspecting an XML instance document and then writing an XSD
for it. It isn't obvious to me that different implementations would
necessarily generate the same XSD in these cases. I'm willing to be
convinced; I'm just not sure that it's fair to expect that different
implementations will generate the same XSD when what they were doing was
populating a model with whatever they needed to read an XML instance.

Regards

Geoff.