You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@netbeans.apache.org by "Arcturus Bootes (Jira)" <ji...@apache.org> on 2020/06/09 08:06:00 UTC
[jira] [Updated] (NETBEANS-4418) Unit tests silently failing
[ https://issues.apache.org/jira/browse/NETBEANS-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arcturus Bootes updated NETBEANS-4418:
--------------------------------------
Description:
I'm using NetBeans 11.3 with Azul Zulu 11 JDK with JavaFX.
I'm working on a NetBeans platform application and we recently noticed that our unit tests are silently failing. Unit tests fail but `ant test` completes with exit code 0. We are running this through Travis which marks the PR as if it passed.
Based on debugging the issue broadly is that the {{continue.after.failing.tests}} property (as defined in ${NETBEANS}/harness/README) does not work.
The reason it doesn't work is that the ant target used for ({{-do-testng}} in ${NETBEANS}/harness/common.xml) isn't processing it correctly
{code:java}
773: <fail if="tests.failed" unless="continue.after.failing.tests">Some tests failed; see details above.</fail> {code}
The {{unless}} keyword will pass if the parameter exists (even if it has no value or a false value), so it doesn't matter if you set it true or false, the fail state will never happen.
Additionally, ${NETBEANS}/harness/suite.xml ensures that the property always exists even if you don't set it.
{code:java}
584: <property name="continue.after.failing.tests" value="true"/> <!-- fallback --> {code}
You can re-produce this issue simply by creating a new module suite containing one module and one unit test which is set to always fail.
{code:java}
import static org.testng.Assert.fail;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class AntTestNGTest {
/**
* Test of main method, of class AntTest.
*/
@Test
public void testMain() throws InterruptedException {
System.out.println("main");
String[] args = null;
AntTest.main(args);
// TODO review the generated test code and remove the default call to fail.
fail("The test case is a prototype.");
}
}
{code}
Then run:
{code:java}
ant -Dnbplatform.active.dir="${NETBEANS_HOME}" -Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}" -Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness -Dcontinue.after.failing.tests=false test{code}
or
{code:java}
ant -Dnbplatform.active.dir="${NETBEANS_HOME}" -Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}" -Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness -Dcontinue.after.failing.tests=false -f AntTestModule test-unit{code}
or add {{continue.after.failing.tests=false}} to the {{project.properties }}file for your module suite and test through Netbeans.
Either way, the test will fail but the build will succeed.
If you then change the {{unless}} condition in ${NETBEANS}/harness/common.xml to point to a property which does not exist
{code:java}
773: <fail if="tests.failed" unless="continue.after.failing.tests.ghost">Some tests failed; see details above.</fail> {code}
The build will fail with the message "Some tests failed..." as expected.
A suggested fix would be to add some logic prior to this {{fail}} condition to check "continue.after.failing.tests" and create a new property if it is set to true - that way you could check for that property in the {{unless}} condition and it would either exist (fail does not happen, all tests are executed) or not-exist (fail state will happen if tests failed).
Also, while we're at it - I also confirmed the {{test.timeout}} property (also defined in ${NETBEANS}/harness/README) also does not work - probably for similar reasons.
was:
I'm using NetBeans 11.3 with Azul Zulu 11 JDK with JavaFX.
I'm working on a NetBeans platform application and we recently noticed that our unit tests are silently failing. Unit tests fail but `ant test` completes with exit code 0. We are running this through Travis which marks the PR as if it passed.
Based on debugging the issue broadly is that the {{continue.after.failing.tests}} property (as defined in ${NETBEANS}/harness/README) does not work.
The reason it doesn't work is that the ant target used for ({{-do-testng}} in ${NETBEANS}/harness/common.xml) isn't processing it correctly
{code:java}
773: <fail if="tests.failed" unless="continue.after.failing.tests">Some tests failed; see details above.</fail> {code}
The {{unless}} keyword will pass if the parameter exists (even if it has no value or a false value), so it doesn't matter if you set it true or false, the fail state will never happen.
Additionally, ${NETBEANS}/harness/suite.xml ensures that the property always exists even if you don't set it.
{code:java}
584: <property name="continue.after.failing.tests" value="true"/> <!-- fallback --> {code}
You can re-produce this issue simply by creating a new module suite containing one module and one unit test which is set to always fail.
{code:java}
import static org.testng.Assert.fail;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;
public class AntTestNGTest {
/**
* Test of main method, of class AntTest.
*/
@Test
public void testMain() throws InterruptedException {
System.out.println("main");
String[] args = null;
AntTest.main(args);
// TODO review the generated test code and remove the default call to fail.
fail("The test case is a prototype.");
}
}
{code}
Then run:
{code:java}
ant -Dnbplatform.active.dir="${NETBEANS_HOME}" -Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}" -Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness -Dcontinue.after.failing.tests=false test{code}
or
{code:java}
ant -Dnbplatform.active.dir="${NETBEANS_HOME}" -Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}" -Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness -Dcontinue.after.failing.tests=false -f AntTestModule test-unit{code}
or add {{continue.after.failing.tests=false}} to the {{project.properties }}file for your module suite and test through Netbeans.
Either way, the test will fail but the build will succeed.
If you then change the {{unless}} condition in ${NETBEANS}/harness/common.xml to point to a property which does not exist
{code:java}
773: <fail if="tests.failed" unless="continue.after.failing.tests.ghost">Some tests failed; see details above.</fail> {code}
The build will fail with the message "Some tests failed..." as expected.
A suggested fix would be to add some logic prior to this {{fail }}condition to check "continue.after.failing.tests" and create a new property if it is set to true - that way you could check for that property in the {{unless}} condition and it would either exist (fail does not happen, all tests are executed) or not-exist (fail state will happen if tests failed).
Also, while we're at it - I also confirmed the {{test.timeout}} property (also defined in ${NETBEANS}/harness/README) also does not work - probably for similar reasons.
> Unit tests silently failing
> ---------------------------
>
> Key: NETBEANS-4418
> URL: https://issues.apache.org/jira/browse/NETBEANS-4418
> Project: NetBeans
> Issue Type: Bug
> Components: apisupport - Harness, platform - Module System
> Affects Versions: 11.3
> Reporter: Arcturus Bootes
> Priority: Major
>
> I'm using NetBeans 11.3 with Azul Zulu 11 JDK with JavaFX.
>
> I'm working on a NetBeans platform application and we recently noticed that our unit tests are silently failing. Unit tests fail but `ant test` completes with exit code 0. We are running this through Travis which marks the PR as if it passed.
>
> Based on debugging the issue broadly is that the {{continue.after.failing.tests}} property (as defined in ${NETBEANS}/harness/README) does not work.
>
> The reason it doesn't work is that the ant target used for ({{-do-testng}} in ${NETBEANS}/harness/common.xml) isn't processing it correctly
>
> {code:java}
> 773: <fail if="tests.failed" unless="continue.after.failing.tests">Some tests failed; see details above.</fail> {code}
>
> The {{unless}} keyword will pass if the parameter exists (even if it has no value or a false value), so it doesn't matter if you set it true or false, the fail state will never happen.
>
> Additionally, ${NETBEANS}/harness/suite.xml ensures that the property always exists even if you don't set it.
>
> {code:java}
> 584: <property name="continue.after.failing.tests" value="true"/> <!-- fallback --> {code}
>
> You can re-produce this issue simply by creating a new module suite containing one module and one unit test which is set to always fail.
>
>
> {code:java}
> import static org.testng.Assert.fail;
> import org.testng.annotations.AfterClass;
> import org.testng.annotations.AfterMethod;
> import org.testng.annotations.BeforeClass;
> import org.testng.annotations.BeforeMethod;
> import org.testng.annotations.Test;
> public class AntTestNGTest {
>
> /**
> * Test of main method, of class AntTest.
> */
> @Test
> public void testMain() throws InterruptedException {
> System.out.println("main");
> String[] args = null;
> AntTest.main(args);
> // TODO review the generated test code and remove the default call to fail.
> fail("The test case is a prototype.");
> }
> }
> {code}
>
> Then run:
>
> {code:java}
> ant -Dnbplatform.active.dir="${NETBEANS_HOME}" -Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}" -Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness -Dcontinue.after.failing.tests=false test{code}
>
> or
> {code:java}
> ant -Dnbplatform.active.dir="${NETBEANS_HOME}" -Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}" -Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness -Dcontinue.after.failing.tests=false -f AntTestModule test-unit{code}
> or add {{continue.after.failing.tests=false}} to the {{project.properties }}file for your module suite and test through Netbeans.
>
> Either way, the test will fail but the build will succeed.
>
> If you then change the {{unless}} condition in ${NETBEANS}/harness/common.xml to point to a property which does not exist
>
> {code:java}
> 773: <fail if="tests.failed" unless="continue.after.failing.tests.ghost">Some tests failed; see details above.</fail> {code}
>
> The build will fail with the message "Some tests failed..." as expected.
> A suggested fix would be to add some logic prior to this {{fail}} condition to check "continue.after.failing.tests" and create a new property if it is set to true - that way you could check for that property in the {{unless}} condition and it would either exist (fail does not happen, all tests are executed) or not-exist (fail state will happen if tests failed).
>
> Also, while we're at it - I also confirmed the {{test.timeout}} property (also defined in ${NETBEANS}/harness/README) also does not work - probably for similar reasons.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@netbeans.apache.org
For additional commands, e-mail: commits-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists