[jira] [Comment Edited] (JDO-765) TCK should fail if tests fail

["Craig L Russell (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/JDO-765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16239315#comment-16239315 ] 

Craig L Russell edited comment on JDO-765 at 11/4/17 11:58 PM:
---------------------------------------------------------------

I applied the patch and the behavior did change, but not exactly what I expected.

Good: When running with -Djdo.tck.onFailure=failEventually, the tck test ran through all of the tests, reported "Failed to execute goal org.apache.jdo:jdo-exectck:3.2-SNAPSHOT:runtck (default) on project jdo-tck: There were 2 TCK test failures" and then failed the mvn goal.

Bad: When running with -Djdo.tck.onFailure default, the tck test ran through all of the tests, reported "Failed to execute goal org.apache.jdo:jdo-exectck:3.2-SNAPSHOT:runtck (default) on project jdo-tck: There were 2 TCK test failures" and then failed the mvn goal.

Bad: When running with -Djdo.tck.onFailure=failFast, the tck test ran through all of the datastoreidentity tests before failing. I expected the test would fail right after the first failure, "Running tests for lifecycle.conf with datastoreidentity on 'derby' mapping= ... FAIL" but the tck test continued with the rest of the datastoreidentity tests. 




was (Author: clr):
I applied the patch and the behavior did change, but not exactly what I expected.

Good: When running with -Djdo.tck.onFailure=failEventually, the tck test ran through all of the tests, reported "Failed to execute goal org.apache.jdo:jdo-exectck:3.2-SNAPSHOT:runtck (default) on project jdo-tck: There were 2 TCK test failures" and then failed the mvn goal.

Good: When running with -Djdo.tck.onFailure default, the tck test ran through all of the tests for both datastoreidentity and applicationidentity, reported the two failures, and went on to the next mvn goal.

Bad: When running with -Djdo.tck.onFailure=failFast, the tck test ran through all of the datastoreidentity tests before failing. I expected the test would fail right after the first failure, "Running tests for lifecycle.conf with datastoreidentity on 'derby' mapping= ... FAIL" but the tck test continued with the rest of the datastoreidentity tests.



> TCK should fail if tests fail
> -----------------------------
>
>                 Key: JDO-765
>                 URL: https://issues.apache.org/jira/browse/JDO-765
>             Project: JDO
>          Issue Type: Bug
>            Reporter: Tilmann Zäschke
>            Priority: Trivial
>         Attachments: JDO-XXX-patch.txt
>
>
> Currently, TCK test failures are reported to the console and in the logs, but the Maven build still succeeds.
> The proposal is to make the behavior configurable, the default being that the Maven build fails if any TCK tests fail.
> The new configuration option would be called "-Djdo.tck.onFailure" and have the following options:
>  * "failFast" will immediately abort the test run. 
>  * "failEventually" (default) will execute the whole TCK before failing. 
>  * "logOnly" will report failures to console and logs only but return 'SUCCESS' to the Maven execution environment.
> Attached is a patch with an implementation of the proposed changes. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)