[jira] [Commented] (MATH-1659) some test failures on legacy 4.0 beta1

["Alex Herbert (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MATH-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17708483#comment-17708483 ] 

Alex Herbert commented on MATH-1659:
------------------------------------

This is actually a typical build output. All the failures are in the optimizer tests. These are rerun as they are allowed to fail. The tests are marked as flakes in the final build report. Your attached trace actually passed all the tests within the allowed number of failures.

The optimizer tests were created using a set of hard to solve functions with many local minima. These would be difficult for any optimizer. The optimizer start points are randomly chosen, hence often the solution is not the optimal one.

Ideally the output should be suppressed so that errors are only shown when all repeats fail. The tests could be rewritten to use custom set-up and assertion operations within a repeat loop with a given number of retries. Currently the tests use the retries functionality of the surefire plugin (see [Maven Surefire plugin: re-run failing tests|https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html]).

One alternative is to use fixed random seeds to initialize the optimizers. But this can create an unrealistic test which can output a pass although pass is a minority fraction of all results (pass or fail). This minority pass fraction is what we have for a lot of the optimizer tests as they require many repeats to observe a pass. So here at least we are aware of the flakyness of tests, just the report is very verbose. I cannot see an option within surefire to quieten down the console output for the build.

> some test failures on legacy 4.0 beta1
> --------------------------------------
>
>                 Key: MATH-1659
>                 URL: https://issues.apache.org/jira/browse/MATH-1659
>             Project: Commons Math
>          Issue Type: Test
>          Components: legacy
>    Affects Versions: 4.0-beta1
>         Environment: Dev configuration where the problem occurs is
>  * Ubuntu 22.04,
>  * Maven 3.6.3
>  * Eclipse 4.23,
>  * java-18-openJdk-amd64 with JavaSE-1.8 compliance
>            Reporter: François Laferrière
>            Priority: Major
>         Attachments: maven.test.trace.txt
>
>
> When running Junit tests on ACM 4.0-snapshot I get a lot of failures. Mostly failures are associated to
>  * roundoff error beyond the tolerance of individual test.
>  * max number of function evalution exceded



--
This message was sent by Atlassian Jira
(v8.20.10#820010)