Preferences on recent specification tests

[Dan Becker <da...@gmail.com>]

I see lots of recent Jira activity on various specification tests and 
conformance test items, e.g. TUSCANY-2923 [1]. These patches are useful 
  and help build a rock-steady product. Great!

However, some of the tests are for items that are not yet implemented in 
Tuscany, missing pieces, etc., and hence some of the tests fail.

What are peoples preferences with these issues? Should we commit the 
tests now, let them fail, and later get to solving the issue? Should we 
hold off on committing tests until we have a feature or fix ready? Other 
thoughts?

[1] http://issues.apache.org/jira/browse/TUSCANY-2923
-- 
Thanks, Dan Becker

Re: Preferences on recent specification tests

[Luciano Resende <lu...@gmail.com>]

On Thu, Mar 19, 2009 at 9:38 AM, Simon Laws <si...@googlemail.com> wrote:
> On Thu, Mar 19, 2009 at 4:08 PM, Raymond Feng <en...@gmail.com> wrote:
>> There is an agreed pattern we can use:
>>
>> For the failing test cases, open a JIRA, and annotate the test(s) with
>> JUNIT4 @Ignore("TUSCANY-xxxx").
>>

+1


-- 
Luciano Resende
Apache Tuscany, Apache PhotArk
http://people.apache.org/~lresende
http://lresende.blogspot.com/

Re: Preferences on recent specification tests

[Simon Laws <si...@googlemail.com>]

On Thu, Mar 19, 2009 at 4:08 PM, Raymond Feng <en...@gmail.com> wrote:
> There is an agreed pattern we can use:
>
> For the failing test cases, open a JIRA, and annotate the test(s) with
> JUNIT4 @Ignore("TUSCANY-xxxx").
>
snip...

Good point Raymond. I've seen that used in several of the tests so we
should keep to that and be consistent. We also have a couple of wiki
pages we can help to track what progress we are making.

[1] shows the tests and their availability in the two code bases.
[2] can help us identify which issues are spec difference related
rather than just errors in the runtime.

Simon

[1] http://cwiki.apache.org/confluence/display/TUSCANYWIKI/Conformance+Testing
[2] http://cwiki.apache.org/confluence/display/TUSCANYWIKI/OSOA+SCA+vs+OASIS+SCA

Re: Preferences on recent specification tests

[Raymond Feng <en...@gmail.com>]

There is an agreed pattern we can use:

For the failing test cases, open a JIRA, and annotate the test(s) with 
JUNIT4 @Ignore("TUSCANY-xxxx").

Thanks,
Raymond

--------------------------------------------------
From: "Dan Becker" <da...@gmail.com>
Sent: Thursday, March 19, 2009 8:49 AM
To: "tuscany-dev" <de...@tuscany.apache.org>
Subject: Preferences on recent specification tests

> I see lots of recent Jira activity on various specification tests and 
> conformance test items, e.g. TUSCANY-2923 [1]. These patches are useful 
> and help build a rock-steady product. Great!
>
> However, some of the tests are for items that are not yet implemented in 
> Tuscany, missing pieces, etc., and hence some of the tests fail.
>
> What are peoples preferences with these issues? Should we commit the tests 
> now, let them fail, and later get to solving the issue? Should we hold off 
> on committing tests until we have a feature or fix ready? Other thoughts?
>
> [1] http://issues.apache.org/jira/browse/TUSCANY-2923
> -- 
> Thanks, Dan Becker 

Re: Preferences on recent specification tests

[Simon Laws <si...@googlemail.com>]

On Thu, Mar 19, 2009 at 3:49 PM, Dan Becker <da...@gmail.com> wrote:
> I see lots of recent Jira activity on various specification tests and
> conformance test items, e.g. TUSCANY-2923 [1]. These patches are useful  and
> help build a rock-steady product. Great!
>
> However, some of the tests are for items that are not yet implemented in
> Tuscany, missing pieces, etc., and hence some of the tests fail.
>
> What are peoples preferences with these issues? Should we commit the tests
> now, let them fail, and later get to solving the issue? Should we hold off
> on committing tests until we have a feature or fix ready? Other thoughts?
>
> [1] http://issues.apache.org/jira/browse/TUSCANY-2923
> --
> Thanks, Dan Becker
>

Good question. I think they should be checked in an allowed to fail
with JIRAs raised for those failures. As the OASIS/OSOA specs are so
similar we are able to exploit he OASIS stest in our 1.x branch to a
great extent. This is the case with TUSCANY-2923.  In some cases there
will be tests that are not appropriate or that need changes for 1.x.
We may also find errors in 1.x that we hadn't spotted before. I still
think we should go ahead and check them in. Stest is not part of the
build yet so it doesn't actually fails the build. That should be the
case bu maybe for 2.x we want to turn the tests on selectively as we
implement more of the 2.x/OASIS function.

Simon