[jira] [Created] (NETBEANS-4418) Unit tests silently failing

["Arcturus Bootes (Jira)" <ji...@apache.org>]

Arcturus Bootes created NETBEANS-4418:
-----------------------------------------

             Summary: Unit tests silently failing
                 Key: NETBEANS-4418
                 URL: https://issues.apache.org/jira/browse/NETBEANS-4418
             Project: NetBeans
          Issue Type: Bug
          Components: apisupport - Harness, platform - Module System
    Affects Versions: 11.3
            Reporter: Arcturus Bootes


I'm using NetBeans 11.3 with Azul Zulu 11 JDK with JavaFX.

 

I'm working on a NetBeans platform application and we recently noticed that our unit tests are silently failing. Unit tests fail but `ant test` completes with exit code 0. We are running this through Travis which marks the PR as if it passed.

 

Based on debugging the issue broadly is that the {{continue.after.failing.tests}} property (as defined in ${NETBEANS}/harness/README) does not work.
 
The reason it doesn't work is that the ant target used for ({{-do-testng}} in ${NETBEANS}/harness/common.xml) isn't processing it correctly
 
{code:java}
773: <fail if="tests.failed" unless="continue.after.failing.tests">Some tests failed; see details above.</fail> {code}
 
The {{unless}} keyword will pass if the parameter exists (even if it has no value or a false value), so it doesn't matter if you set it true or false, the fail state will never happen.
 
Additionally, ${NETBEANS}/harness/suite.xml ensures that the property always exists even if you don't set it.
 
{code:java}
584: <property name="continue.after.failing.tests" value="true"/> <!-- fallback --> {code}
 
You can re-produce this issue simply by creating a new module suite containing one module and one unit test which is set to always fail.
 
 
{code:java}
import static org.testng.Assert.fail;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

public class AntTestNGTest {
    
    /**
     * Test of main method, of class AntTest.
     */
    @Test
    public void testMain() throws InterruptedException {
        System.out.println("main");
        String[] args = null;
        AntTest.main(args);
        // TODO review the generated test code and remove the default call to fail.
        fail("The test case is a prototype.");
    }
}
 {code}
 
Then run:
 
{code:java}
ant -Dnbplatform.active.dir="${NETBEANS_HOME}" -Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}" -Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness -Dcontinue.after.failing.tests=false test{code}
​
or
{code:java}
ant -Dnbplatform.active.dir="${NETBEANS_HOME}" -Dnbplatform.default.netbeans.dest.dir="${NETBEANS_HOME}" -Dnbplatform.default.harness.dir="${NETBEANS_HOME}"/harness -Dcontinue.after.failing.tests=false -f AntTestModule test-unit{code}


or add {{continue.after.failing.tests=false}} to the {{project.properties }}file for your module suite and test through Netbeans.
​
Either way, the test will fail but the build will succeed.
​
If you then change the {{unless}} condition in ${NETBEANS}/harness/common.xml to point to a property which does not exist
 
{code:java}
773: <fail if="tests.failed" unless="continue.after.failing.tests.ghost">Some tests failed; see details above.</fail> {code}
 
The build will fail with the message "Some tests failed..." as expected.
 A suggested fix would be to add some logic prior to this {{fail }}condition to check "continue.after.failing.tests" and create a new property if it is set to true - that way you could check for that property in the {{unless}} condition and it would either exist (fail does not happen, all tests are executed) or not-exist (fail state will happen if tests failed).
​
Also, while we're at it - I also confirmed the {{test.timeout}} property (also defined in ${NETBEANS}/harness/README) also does not work - probably for similar reasons.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@netbeans.apache.org
For additional commands, e-mail: commits-help@netbeans.apache.org

For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists