You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Sebb (JIRA)" <ji...@apache.org> on 2008/11/25 11:03:44 UTC
[jira] Created: (SCXML-91) Test case bugs
Test case bugs
--------------
Key: SCXML-91
URL: https://issues.apache.org/jira/browse/SCXML-91
Project: Commons SCXML
Issue Type: Bug
Affects Versions: 0.9
Reporter: Sebb
Test cases are difficult to debug if they fail.
This is because many test cases catch Exception, and don't report it fully.
Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
Several test cases report problems to System.out or System.err and carry on processing.
For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660460#action_12660460 ]
Sebb commented on SCXML-91:
---------------------------
SCXMLTestHelper still contains quite a few catch blocks that merely call fail().
These still need looking at to see if they can be replaced by throws clauses.
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654005#action_12654005 ]
Sebb commented on SCXML-91:
---------------------------
Fixed item 2) I/O errors:
Trunk:
URL: http://svn.apache.org/viewvc?rev=723933&view=rev
Log:
SCXML-91 - SCXMLTestHelper#testExecutorSerializability() should not ignore I/O errors.
J6 branch:
URL: http://svn.apache.org/viewvc?rev=723934&view=rev
Log:
SCXML-91 - SCXMLTestHelper#testExecutorSerializability() should not ignore I/O errors.
Changing the method to throw the errors would require lots of other changes.
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660740#action_12660740 ]
Sebb commented on SCXML-91:
---------------------------
SCXMLTestHelper has now been updated in J6 to throw Exceptions:
URL: http://svn.apache.org/viewvc?rev=731386&view=rev
Log:
Prepare for changes to SCXMLTestHelper
URL: http://svn.apache.org/viewvc?rev=731387&view=rev
Log:
Remove try/catch blocks for Exceptions that are not expected in testing
In trunk, the same changes have been made, except that DOM serialisation errors are allowed through
(which is closer to what I would like to see):
URL: http://svn.apache.org/viewvc?rev=731393&view=rev
Log:
Prepare for changes to SCXMLTestHelper
URL: http://svn.apache.org/viewvc?rev=731543&view=rev
Log:
Remove try/catch blocks for Exceptions that are not expected in testing
Improve non-serialisable check so it only allows DOM elements to fail
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Rahul Akolkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660442#action_12660442 ]
Rahul Akolkar commented on SCXML-91:
------------------------------------
Thanks, if you're happy with the changes for (1), I think we can resolve this issue now.
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Rahul Akolkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661381#action_12661381 ]
Rahul Akolkar commented on SCXML-91:
------------------------------------
Summarizing the dev list discussion (hopefully the crimson check will not require any more maintenance):
----
sebb wrote:
> but I think NSEs which are not caused by
> the faulty Sun 1.4 DOM should cause failures.
>
<snap/>
OK, so what you added to trunk does that. I guess we can retain the crimson check as-is and keep refining it as and when it breaks with other obscure JDKs and configurations you and I haven't tested on.
----
Looks like the major items on this issue are addressed?
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Rahul Akolkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651125#action_12651125 ]
Rahul Akolkar commented on SCXML-91:
------------------------------------
Regarding (2):
As I mentioned, the copious output suits my style (of running one test at a time when digging into details) so I'm in favor of retaining it.
I mentioned on the dev list in the RC thread why the DOM serialization shouldn't be a test failure.
Agree on the I/O errors (though we do print the trace IIRC).
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Rahul Akolkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654665#action_12654665 ]
Rahul Akolkar commented on SCXML-91:
------------------------------------
Please go ahead with any changes for (1) such as in the comment above.
As to (2), I'm afraid we're going around in circles and nothing new has been said since the thread on dev@.
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Rahul Akolkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661688#action_12661688 ]
Rahul Akolkar commented on SCXML-91:
------------------------------------
Sounds good.
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SCXML-91) Test case bugs
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sebb resolved SCXML-91.
-----------------------
Resolution: Fixed
I think this can now finally be resolved!
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Rahul Akolkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660887#action_12660887 ]
Rahul Akolkar commented on SCXML-91:
------------------------------------
Pasting some comments from the dev list in JIRA as well:
* Comment #1:
The idea (verifying if the NSE is actually due to the DOM implementation) is good.
I think this implementation (r731543) is fragile since:
(a) It relies on an exception message
(b) It only accounts for crimson, but in theory the package name could be anything (by virtue of the Java endorsed standards override mechanism)
Lets retain the NSE catch blocks as before. Other changes look good.
* Comment #2:
WRT the directory creation, if you'd rather have the build fail if creation fails, there would ideally be a nice message stating the problem (rather than a generic x failures and y errors kind of thing) and additonally, there'd be a build time switch that allows folks to say "we don't care about serialization" which would cause the build to succeed (this could also help the NSE catches you are looking at eliminating -- all NSEs could then cause tests to error out by default). So maybe the approach of providing such a switch helps both cases we're discussing.
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SCXML-91) Test case bugs
Posted by "Rahul Akolkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rahul Akolkar updated SCXML-91:
-------------------------------
Fix Version/s: 0.10
You've mentioned three separable items, and I will respond in order:
1) As mentioned on the dev list, we should fix the bit about catching exceptions only when expected for the particular test. I'm setting a fix version of v0.10 on this issue for that.
2) Regarding the System.err error reporting, especially for serialization failures, I contend that its doing its job. It is not a fatal error from the PoV of library code, its more like a configuration error if serialization is needed. The purpose is to draw attention of the person running the build by spewing to console, which it does well I think.
3) The testing output is desired (more in SCXML-92 later) and the stack traces are unavoidable. Some time ago, Commons introduced code in various places that both logs an exception and rethrows it. We now know that it is not the best practice, but ofcourse legacy persists until someone does something about it. Here is a pointer to the particular traces we see in the Commons SCXML build (in this case, they are from Commons Digester):
http://commons.apache.org/digester/xref/org/apache/commons/digester/Digester.html#1455
(above URL may become outdated over time since it refers to a line number in source code)
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661684#action_12661684 ]
Sebb commented on SCXML-91:
---------------------------
I've just realised that the stack dumps are part of the log message, so changing the log level to fatal (rather than warn) will eliminate these, and also eliminate the remaining non-essential output.
I think that's the neatest solution overall - agreed?
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Rahul Akolkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654295#action_12654295 ]
Rahul Akolkar commented on SCXML-91:
------------------------------------
OK, so thats that for (2) and (3). Did you have anything else in mind for (1)?
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Rahul Akolkar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661453#action_12661453 ]
Rahul Akolkar commented on SCXML-91:
------------------------------------
My way of saying the exceptions are expected is to have the string '404' in the URLs that aren't meant to be found :-) (if folks read the exception messages in the trace)
If you want to further improve any messages, please go ahead.
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12650962#action_12650962 ]
Sebb commented on SCXML-91:
---------------------------
1) OK
2) There is so much output generated that it is possible to overlook the serialisation errors.
Also, a lot of this output is expected; it's tedious scanning through all of it.
Perhaps a work-round would be to add a single test to check the serialisability of a known problematic class and report that as a failure.
SCXMLTestHelper#testExecutorSerializability() should not ignore I/O errors.
3) OK, I discovered this myself later.
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12661406#action_12661406 ]
Sebb commented on SCXML-91:
---------------------------
Yes, I think the test output is a lot easier to follow, and hopefully more bugs will be caught as fewer errors are ignored.
One minor tweak which might be worth considering would be to output messages to show which warning and error messages are expected, for example to print a message to say that the 2 stack traces are expected.
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654658#action_12654658 ]
Sebb commented on SCXML-91:
---------------------------
As to 1), what I had in mind was to change code of the form:
{code}
public void testInvoke01Sample() {
try {
SCXML scxml = SCXMLParser.parse(invoke01,
new SimpleErrorHandler());
exec = new SCXMLExecutor(new JexlEvaluator(), new SimpleDispatcher(),
new SimpleErrorReporter());
assertNotNull(exec);
exec.setRootContext(new JexlContext());
exec.setStateMachine(scxml);
exec.registerInvokerClass("scxml", SimpleSCXMLInvoker.class);
exec.go();
Set<TransitionTarget> currentStates = exec.getCurrentStatus().getStates();
assertEquals(1, currentStates.size());
assertEquals("invoker", currentStates.iterator().next().getId());
} catch (Exception e) {
e.printStackTrace();
fail(e.getMessage());
}
{code}
to
{code}
public void testInvoke01Sample() throws Exception {
SCXML scxml = SCXMLParser.parse(invoke01,
new SimpleErrorHandler());
exec = new SCXMLExecutor(new JexlEvaluator(), new SimpleDispatcher(),
new SimpleErrorReporter());
assertNotNull(exec);
exec.setRootContext(new JexlContext());
exec.setStateMachine(scxml);
exec.registerInvokerClass("scxml", SimpleSCXMLInvoker.class);
exec.go();
Set<TransitionTarget> currentStates = exec.getCurrentStatus().getStates();
assertEquals(1, currentStates.size());
assertEquals("invoker", currentStates.iterator().next().getId());
}
{code}
As to 2), I still think it would be sensible to report at least one test failure if the test environment does not allow all the tests to be completed successfully.
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12660430#action_12660430 ]
Sebb commented on SCXML-91:
---------------------------
Patches for 1) applied to trunk and J6
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SCXML-91) Test case bugs
Posted by "Sebb (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SCXML-91?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12651203#action_12651203 ]
Sebb commented on SCXML-91:
---------------------------
I would prefer to see minimal output when things are working well (Unix mindset perhaps, or maybe I'm lazy) and be able to switch on additional trace only if required.
This would be easy enough to do by changing the default to debug level.
> Test case bugs
> --------------
>
> Key: SCXML-91
> URL: https://issues.apache.org/jira/browse/SCXML-91
> Project: Commons SCXML
> Issue Type: Bug
> Affects Versions: 0.9
> Reporter: Sebb
> Fix For: 0.10
>
>
> Test cases are difficult to debug if they fail.
> This is because many test cases catch Exception, and don't report it fully.
> Test cases should only catch a (specific) Exception if the test is expected to generate one, and should otherwise throw the Exception.
> Several test cases report problems to System.out or System.err and carry on processing.
> For example, serialisation errors are largely ignored, and SCXMLTestHelper#testExecutorSerializability() ignores IO errors.
> Testing generates a lot of output, some of which appears to be errors (e.g. stack traces) yet the test passes.
> Ideally tests should suppress output stack traces which are expected during testing.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.