[jira] [Resolved] (SCXML-222) Add support for executing the SCXML IRP test cases

["Ate Douma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SCXML-222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ate Douma resolved SCXML-222.
-----------------------------
    Resolution: Fixed

I've added some basic usage documentation in the specific w3cTests.get, w3cTests.make and w3cTests.run profiles in the pom.xml.
For more details on using and maintaining the tests (especially the tests.xml file), for now see the javadoc and usage() method in the W3CTests.java class itself.

I'll add more complete online documentation for this feature after the ApacheCon EU Budapest next week. 

> Add support for executing the SCXML IRP test cases 
> ---------------------------------------------------
>
>                 Key: SCXML-222
>                 URL: https://issues.apache.org/jira/browse/SCXML-222
>             Project: Commons SCXML
>          Issue Type: New Feature
>    Affects Versions: 2.0
>            Reporter: Ate Douma
>            Assignee: Ate Douma
>             Fix For: 2.0
>
>
> A large part of the effort and changes done through SCXML-213 were tested and validated by execution (part of) the SCXML IRP test cases.
> The [SCXML IRP|http://www.w3.org/Voice/2013/scxml-irp/], or SCXML 1.0 Implementation Report Plan provides many 'assertion' tests based on the specification, and can be used to verify that the specification can be implemented. 
> This is NOT a formal specification compliance testsuite, as shouldn't be regarded as such either.
> But nonetheless its been *extremely* valuable to recognized and understand the finer and tricky details of the specification and how an implementation is expected to work.
> I've found many, even rather trivial, bugs in the current implementation through these tests, and therefore I'll now add standard support for executing these as 'tests' through Maven.
>   
> The support is provided through a dedicated and single 'W3CTests' class which is executed through separate Maven profiles (w3cTests.get, w3cTests.make and w3cTests.run).
> The IRP tests themselves are not checked in (although the license would allow it) but instead are dynamically downloaded by the W3CTests class (when executing 'get).
> The location where these test sources are download is under a separate folder src/w3c, which I'll mark with svn:ignore, so these will not accidentally get commited either.
> The current implementation of Commons SCXML however isn't (by far) specification compliant enough to succesfully execute all these (213) tests.
> Instead, I'm providing an extra 'tests.xml' configuration file through which we can configure which tests are enabled/disabled and other meta-data needed to perform and track the execution status.
> Currently 80 tests are passing succesfully, and some more for only either the xpath or ecmascript datamodel.
> Note: the IRP tests only assume an xpath, ecmascript or "null" (minimal) datamodel. 
> It probably will be worthwhile to add support to these tests (within Commons SCXML) for Jexl and Groovy as well.
> Most likely these will actually already be more compliant than the current ecmascript support is... 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)