You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by Bogdan Calmac <bc...@gmail.com> on 2010/11/09 22:17:37 UTC
Running tests twice and the build success status
Our project is configured to run the unit tests twice, with oracle and mssql
databases. This is achieved by configuring two executions of the surefire
plugin and passing in a different system property. Here is the relevant
snippet from the POM:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<executions>
<execution>
<id>default-test</id>
<configuration>
<systemPropertyVariables><testDBTypes>mssql</testDBTypes></systemPropertyVariables>
<reportsDirectory>${project.build.directory}/surefire-reports-mssql</reportsDirectory>
</configuration>
</execution>
<execution>
<id>oracle</id>
<phase>test</phase>
<goals><goal>test</goal></goals>
<configuration>
<systemPropertyVariables><testDBTypes>oracle</testDBTypes></systemPropertyVariables>
<reportsDirectory>${project.build.directory}/surefire-reports-oracle</reportsDirectory>
</configuration>
</execution>
</executions>
</plugin>
This setup works fine but we've recently discovered that the build appears
as successful when the mssql tests pass and the oracle tests fail. The build
continues and the artifacts are deployed as if nothing happened. It looks
like the mssql execution sets some flag that tests passed and it doesn't
matter what happens with the oracle execution.
Any idea how to get the build to fail when either of the test executions
fail (which is the common-sense expectation)?
Could this be just a bug or are we abusing surefire with this kind of
configuration?
We are using maven 2.2.1, surefire 2.6 and testng 5.14.1
Also, here is the relevant console output for the build:
[INFO] [surefire:test {execution: default-test}]
[INFO] Surefire report directory:
c:\hudson\jobs\commons\workspace\target\surefire-reports-mssql
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestSuite
...
Tests run: 374, Failures: 0, Errors: 0, Skipped: 0
[HUDSON] Recording test results
[INFO] [surefire:test {execution: oracle}]
[INFO] Surefire report directory:
c:\hudson\jobs\commons\workspace\target\surefire-reports-oracle
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running TestSuite
Tests run: 374, Failures: 2, Errors: 0, Skipped: 0
[ERROR] There are test failures.
Please refer to
c:\hudson\jobs\commons\workspace\target\surefire-reports-oracle for the
individual test results.
[HUDSON] Recording test results
[INFO] [jar:jar {execution: default-jar}]
[INFO] [install:install {execution: default-install}]
[INFO] [deploy:deploy {execution: default-deploy}]
Uploading:
http://hudson/nexus/content/repositories/snapshots/com/knoa/knoa-commons/5.6.3-SNAPSHOT/knoa-commons-5.6.3-SNAPSHOT-sources.jar
221K uploaded (knoa-commons-5.6.3-SNAPSHOT-sources.jar)
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 10 minutes 16 seconds
[INFO] Finished at: Tue Nov 09 15:05:44 EST 2010
[INFO] Final Memory: 19M/47M
[INFO]
------------------------------------------------------------------------
--
View this message in context: http://maven.40175.n5.nabble.com/Running-tests-twice-and-the-build-success-status-tp3257693p3257693.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
How to prevent hudson from proceeding after test failures
Posted by Bogdan Calmac <bc...@gmail.com>.
This is in reply to an earlier post where I was complaining that hudson
changes default maven behavior and does not fail the build on test failures.
This means that the "deploy" goal would result in actually deploying the
artifacts even though the tests failed.
Here is the code from hudson.maven.reporters.SurefireArchiver.preExecute()
that changes mvn behavior to ignore test failures:
if (isSurefireTest(mojo)) {
if (!mojo.is("org.apache.maven.plugins",
"maven-failsafe-plugin", "integration-test")) {
// tell surefire:test to keep going even if there was a
failure,
// so that we can record this as yellow.
// note that because of the way Maven works, just updating
system property at this point is too late
XmlPlexusConfiguration c = (XmlPlexusConfiguration)
mojo.configuration.getChild("testFailureIgnore");
if(c!=null &&
c.getValue().equals("${maven.test.failure.ignore}") &&
System.getProperty("maven.test.failure.ignore")==null)
c.setValue("true");
}
}
So, it is enough to explicitly define
<testFailureIgnore>false</testFailureIgnore> in the execution of the
surefire plugin or define the system property
-Dmaven.test.failure.ignore=false to prevent the override. I tested it and
it works. Huray!
Bogdan Calmac wrote:
>
> OK, I've also tested with a simple maven projects and (as you mentioned)
> the artifacts are still deployed even after the tests fail.
>
> Now irrespective of the color of the balls and what hudson does on its
> side, I expect mvn to behave the same on hudson and cmd-line. If maven
> does not deploy artifacts after a test failure then hudson should respect
> that choice. I don't really need hudson to "improve" the way maven works.
> This is just a huge design bug of the hudson-maven integration. And the
> behavior is not even optional.
>
> Then you were saying that the hudson maven plugin is crappy. But I like
> the way it works in a project with many modules where it discovers the
> modules and can do incremental builds. I'm very satisfied with it so far,
> it's just this bug that spoils it.
>
>
>
--
View this message in context: http://maven.40175.n5.nabble.com/Running-tests-twice-and-the-build-success-status-tp3257693p3271735.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Running tests twice and the build success status
Posted by Bogdan Calmac <bc...@gmail.com>.
OK, I've also tested with a simple maven projects and (as you mentioned) the
artifacts are still deployed even after the tests fail.
Now irrespective of the color of the balls and what hudson does on its side,
I expect mvn to behave the same on hudson and cmd-line. If maven does not
deploy artifacts after a test failure then hudson should respect that
choice. I don't really need hudson to "improve" the way maven works. This is
just a huge design bug of the hudson-maven integration. And the behavior is
not even optional.
Then you were saying that the hudson maven plugin is crappy. But I like the
way it works in a project with many modules where it discovers the modules
and can do incremental builds. I'm very satisfied with it so far, it's just
this bug that spoils it.
--
View this message in context: http://maven.40175.n5.nabble.com/Running-tests-twice-and-the-build-success-status-tp3257693p3260403.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Running tests twice and the build success status
Posted by Stephen Connolly <st...@gmail.com>.
That would be because hudson turns off failing a build if tests fail.
In hudson, failing tests is a yellow ball.
Build broken is a red ball
Everything hunky-dory is a blue ball.
your issue comes from using the crappy maven 2 project type in hudson which
modifies your build behind your back.
Use freestyle and ensure that the test reports pattern matches all your
output directories and you'll get your yellow balls again (assuming you add
-Dmaven.test.failure.ignore=true so that your hudson build can give yellow
balls)
By the way, yellow balls are better because they tell you how many tests are
failing... Developer builds should fail fast. CI builds should report how
badly the tests are failing to allow you to track progress
- Stephen
---
Sent from my Android phone, so random spelling mistakes are a direct result
of using swype to type on the screen
On 10 Nov 2010 18:01, "Bogdan Calmac" <bc...@gmail.com> wrote:
OK, I've done some further investigation and discovered that this only
happens on our Hudson CI server. The cmdline maven build works as expected.
For reference, here is the JIRA for the Hudson project:
http://issues.hudson-ci.org/browse/HUDSON-8065
--
View this message in context:
http://maven.40175.n5.nabble.com/Running-tests-twice-and-the-build-success-status-tp3257693p3259119.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-...
Re: Running tests twice and the build success status
Posted by Bogdan Calmac <bc...@gmail.com>.
OK, I've done some further investigation and discovered that this only
happens on our Hudson CI server. The cmdline maven build works as expected.
For reference, here is the JIRA for the Hudson project:
http://issues.hudson-ci.org/browse/HUDSON-8065
--
View this message in context: http://maven.40175.n5.nabble.com/Running-tests-twice-and-the-build-success-status-tp3257693p3259119.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Running tests twice and the build success status
Posted by Stephen Connolly <st...@gmail.com>.
By decouple, I mean decouple failing from running... i.e. run tests, run
tests, check tests, check tests... Not run tests, check tests, run tests,
check tests
On 9 Nov 2010 21:43, "Bogdan Calmac" <bc...@gmail.com> wrote:
No, I don't necessarily want to decouple the tests from the build. It's also
OK if the build fails after running only the mssql tests. The only thing is
that I don't want the build to succeed when after of the test executions
failed.
--
View this message in context:
http://maven.40175.n5.nabble.com/Running-tests-twice-and-the-build-success-status-tp3257693p3257729.html
Sent from the Maven - Users mailing list archive at Nabble.com.
-----------------------------------...
Re: Running tests twice and the build success status
Posted by Bogdan Calmac <bc...@gmail.com>.
No, I don't necessarily want to decouple the tests from the build. It's also
OK if the build fails after running only the mssql tests. The only thing is
that I don't want the build to succeed when after of the test executions
failed.
--
View this message in context: http://maven.40175.n5.nabble.com/Running-tests-twice-and-the-build-success-status-tp3257693p3257729.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org
Re: Running tests twice and the build success status
Posted by Stephen Connolly <st...@gmail.com>.
You might be better served with the maven-failsafe-plugin as you want
to decouple running tests from failing the build (i.e. you want to run
all tests and then fail the build)
Also, if you are using a real database, they sound more like
integration tests... again a use case for m-f-p
But having said all that, I don't see why what you are trying to do
shouldn't work with m-s-p
-Stephen
On 9 November 2010 21:17, Bogdan Calmac <bc...@gmail.com> wrote:
>
> Our project is configured to run the unit tests twice, with oracle and mssql
> databases. This is achieved by configuring two executions of the surefire
> plugin and passing in a different system property. Here is the relevant
> snippet from the POM:
>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <executions>
> <execution>
> <id>default-test</id>
> <configuration>
>
> <systemPropertyVariables><testDBTypes>mssql</testDBTypes></systemPropertyVariables>
>
> <reportsDirectory>${project.build.directory}/surefire-reports-mssql</reportsDirectory>
> </configuration>
> </execution>
> <execution>
> <id>oracle</id>
> <phase>test</phase>
> <goals><goal>test</goal></goals>
> <configuration>
>
> <systemPropertyVariables><testDBTypes>oracle</testDBTypes></systemPropertyVariables>
>
> <reportsDirectory>${project.build.directory}/surefire-reports-oracle</reportsDirectory>
> </configuration>
> </execution>
> </executions>
> </plugin>
>
>
> This setup works fine but we've recently discovered that the build appears
> as successful when the mssql tests pass and the oracle tests fail. The build
> continues and the artifacts are deployed as if nothing happened. It looks
> like the mssql execution sets some flag that tests passed and it doesn't
> matter what happens with the oracle execution.
>
> Any idea how to get the build to fail when either of the test executions
> fail (which is the common-sense expectation)?
>
> Could this be just a bug or are we abusing surefire with this kind of
> configuration?
>
> We are using maven 2.2.1, surefire 2.6 and testng 5.14.1
>
> Also, here is the relevant console output for the build:
>
> [INFO] [surefire:test {execution: default-test}]
> [INFO] Surefire report directory:
> c:\hudson\jobs\commons\workspace\target\surefire-reports-mssql
>
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running TestSuite
> ...
> Tests run: 374, Failures: 0, Errors: 0, Skipped: 0
>
> [HUDSON] Recording test results
> [INFO] [surefire:test {execution: oracle}]
> [INFO] Surefire report directory:
> c:\hudson\jobs\commons\workspace\target\surefire-reports-oracle
>
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running TestSuite
> Tests run: 374, Failures: 2, Errors: 0, Skipped: 0
>
> [ERROR] There are test failures.
>
> Please refer to
> c:\hudson\jobs\commons\workspace\target\surefire-reports-oracle for the
> individual test results.
> [HUDSON] Recording test results
> [INFO] [jar:jar {execution: default-jar}]
> [INFO] [install:install {execution: default-install}]
> [INFO] [deploy:deploy {execution: default-deploy}]
> Uploading:
> http://hudson/nexus/content/repositories/snapshots/com/knoa/knoa-commons/5.6.3-SNAPSHOT/knoa-commons-5.6.3-SNAPSHOT-sources.jar
> 221K uploaded (knoa-commons-5.6.3-SNAPSHOT-sources.jar)
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESSFUL
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 10 minutes 16 seconds
> [INFO] Finished at: Tue Nov 09 15:05:44 EST 2010
> [INFO] Final Memory: 19M/47M
> [INFO]
> ------------------------------------------------------------------------
>
> --
> View this message in context: http://maven.40175.n5.nabble.com/Running-tests-twice-and-the-build-success-status-tp3257693p3257693.html
> Sent from the Maven - Users mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
For additional commands, e-mail: users-help@maven.apache.org