You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Gilles (JIRA)" <ji...@apache.org> on 2011/02/19 23:52:39 UTC
[jira] Created: (MATH-520) Some unit tests fail in
"CMAESOptimizerTest"
Some unit tests fail in "CMAESOptimizerTest"
--------------------------------------------
Key: MATH-520
URL: https://issues.apache.org/jira/browse/MATH-520
Project: Commons Math
Issue Type: Task
Reporter: Gilles
Fix For: 3.0
Out of 28 test runs, "CMAESOptimizerTest" failed 6 times. Those failures were sometimes in "testMaximize" and sometimes in "testDiagonalRosen".
Is there some way to reduce the failure ratio?
Anyways, it would probably be useful to introduce comments for those methods that are expected to fail sometimes.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MATH-520) Some unit tests fail in
"CMAESOptimizerTest"
Posted by "Luc Maisonobe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997112#comment-12997112 ]
Luc Maisonobe commented on MATH-520:
------------------------------------
Unit test should not fail "sometimes".
As long as they are used in automated tests, they should have deterministic behaviour. This means that if they use some random drawings, the seed of the generator *must* be set to a constant value in the unit tests.
Of course, for validation testing, this constraint should be removed to invastigate potential problems.
I think we have already encountered this kind of problems a few times, in the statistics package if I remember well.
> Some unit tests fail in "CMAESOptimizerTest"
> --------------------------------------------
>
> Key: MATH-520
> URL: https://issues.apache.org/jira/browse/MATH-520
> Project: Commons Math
> Issue Type: Task
> Reporter: Gilles
> Labels: test
> Fix For: 3.0
>
>
> Out of 28 test runs, "CMAESOptimizerTest" failed 6 times. Those failures were sometimes in "testMaximize" and sometimes in "testDiagonalRosen".
> Is there some way to reduce the failure ratio?
> Anyways, it would probably be useful to introduce comments for those methods that are expected to fail sometimes.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-520) Some unit tests fail in
"CMAESOptimizerTest"
Posted by "Gilles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068376#comment-13068376 ]
Gilles commented on MATH-520:
-----------------------------
IIRC, Dietmar said on the ML that it was OK to have the failing tests retried. Currently the tests that were sometimes failing are tried at most 3 times.
> Some unit tests fail in "CMAESOptimizerTest"
> --------------------------------------------
>
> Key: MATH-520
> URL: https://issues.apache.org/jira/browse/MATH-520
> Project: Commons Math
> Issue Type: Task
> Reporter: Gilles
> Labels: test
> Fix For: 3.0
>
>
> Out of 28 test runs, "CMAESOptimizerTest" failed 6 times. Those failures were sometimes in "testMaximize" and sometimes in "testDiagonalRosen".
> Is there some way to reduce the failure ratio?
> Anyways, it would probably be useful to introduce comments for those methods that are expected to fail sometimes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (MATH-520) Some unit tests fail in
"CMAESOptimizerTest"
Posted by "Phil Steitz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997139#comment-12997139 ]
Phil Steitz edited comment on MATH-520 at 2/20/11 2:03 PM:
-----------------------------------------------------------
I agree that the tests should (eventually) behave completely deterministically. My sloppiness in this led to annoying failures until we eventually fixed all of the non-deterministic tests that I had sprinkled through the statistics and random packages. At the same time, as the code was stabilizing, these tests did turn up some real bugs that might not have been discovered if the tests were not allowed to run non-deterministically.
I have no idea what is going on in this case, but I suspect there is a bug somewhere and the non-deterministic test may have helped us find it. So I would advocate resolving the issue without just resorting to finding a "winning" seed and letting the tests run for a while non-deterministically until we get closer to the 3.0 release.
was (Author: psteitz):
I agree that the tests should (eventually) behave completely deterministically. My sloppiness in this led to annoying failures until we eventually fixed all of the non=deterministic tests that I had sprinkled through the statistics and random packages. At the same time, as the code was stabilizing, these tests did turn up some real bugs that might not have been discovered if the tests were not allowed to run non-deterministically.
I have no idea what is going on in this case, but I suspect there is a bug somewhere and the non-deterministic test may have helped us find it. So I would advocate resolving the issue without just resorting to finding a "winning" seed and letting the tests run for a while non-deterministically until we get closer to the 3.0 release.
> Some unit tests fail in "CMAESOptimizerTest"
> --------------------------------------------
>
> Key: MATH-520
> URL: https://issues.apache.org/jira/browse/MATH-520
> Project: Commons Math
> Issue Type: Task
> Reporter: Gilles
> Labels: test
> Fix For: 3.0
>
>
> Out of 28 test runs, "CMAESOptimizerTest" failed 6 times. Those failures were sometimes in "testMaximize" and sometimes in "testDiagonalRosen".
> Is there some way to reduce the failure ratio?
> Anyways, it would probably be useful to introduce comments for those methods that are expected to fail sometimes.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-520) Some unit tests fail in
"CMAESOptimizerTest"
Posted by "Luc Maisonobe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13068360#comment-13068360 ]
Luc Maisonobe commented on MATH-520:
------------------------------------
Is this issue solved with the changes that were included on the test rerun ?
I don't recall when we last saw random failures from continuous integration that were due to CMAES.
> Some unit tests fail in "CMAESOptimizerTest"
> --------------------------------------------
>
> Key: MATH-520
> URL: https://issues.apache.org/jira/browse/MATH-520
> Project: Commons Math
> Issue Type: Task
> Reporter: Gilles
> Labels: test
> Fix For: 3.0
>
>
> Out of 28 test runs, "CMAESOptimizerTest" failed 6 times. Those failures were sometimes in "testMaximize" and sometimes in "testDiagonalRosen".
> Is there some way to reduce the failure ratio?
> Anyways, it would probably be useful to introduce comments for those methods that are expected to fail sometimes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MATH-520) Some unit tests fail in
"CMAESOptimizerTest"
Posted by "Luc Maisonobe (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Luc Maisonobe resolved MATH-520.
--------------------------------
Resolution: Fixed
Yes, I remember now.
So the issue is solved, using the retry feature.
> Some unit tests fail in "CMAESOptimizerTest"
> --------------------------------------------
>
> Key: MATH-520
> URL: https://issues.apache.org/jira/browse/MATH-520
> Project: Commons Math
> Issue Type: Task
> Reporter: Gilles
> Labels: test
> Fix For: 3.0
>
>
> Out of 28 test runs, "CMAESOptimizerTest" failed 6 times. Those failures were sometimes in "testMaximize" and sometimes in "testDiagonalRosen".
> Is there some way to reduce the failure ratio?
> Anyways, it would probably be useful to introduce comments for those methods that are expected to fail sometimes.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MATH-520) Some unit tests fail in
"CMAESOptimizerTest"
Posted by "Phil Steitz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997139#comment-12997139 ]
Phil Steitz commented on MATH-520:
----------------------------------
I agree that the tests should (eventually) behave completely deterministically. My sloppiness in this led to annoying failures until we eventually fixed all of the non=deterministic tests that I had sprinkled through the statistics and random packages. At the same time, as the code was stabilizing, these tests did turn up some real bugs that might not have been discovered if the tests were not allowed to run non-deterministically.
I have no idea what is going on in this case, but I suspect there is a bug somewhere and the non-deterministic test may have helped us find it. So I would advocate resolving the issue without just resorting to finding a "winning" seed and letting the tests run for a while non-deterministically until we get closer to the 3.0 release.
> Some unit tests fail in "CMAESOptimizerTest"
> --------------------------------------------
>
> Key: MATH-520
> URL: https://issues.apache.org/jira/browse/MATH-520
> Project: Commons Math
> Issue Type: Task
> Reporter: Gilles
> Labels: test
> Fix For: 3.0
>
>
> Out of 28 test runs, "CMAESOptimizerTest" failed 6 times. Those failures were sometimes in "testMaximize" and sometimes in "testDiagonalRosen".
> Is there some way to reduce the failure ratio?
> Anyways, it would probably be useful to introduce comments for those methods that are expected to fail sometimes.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira