You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Frank Hess (JIRA)" <ji...@apache.org> on 2012/09/22 01:48:07 UTC
[jira] [Created] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Frank Hess created MATH-868:
-------------------------------
Summary: Default sigma for CMAESOptimizer is wrong when using bounds
Key: MATH-868
URL: https://issues.apache.org/jira/browse/MATH-868
Project: Commons Math
Issue Type: Bug
Reporter: Frank Hess
The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Frank Hess (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461775#comment-13461775 ]
Frank Hess commented on MATH-868:
---------------------------------
I found an older ticket on this: MATH-702. Apparently, the intent is for inputSigma to be in the same units as the other parameters (not normalized to the interval zero to one). I can confirm when I play with the MATH-867 test it does worse with sigma that are much less than or greater than 1, I'm not sure what that means.
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Gilles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13476103#comment-13476103 ]
Gilles commented on MATH-868:
-----------------------------
Actually, MATH-874 provides a solution to make the API uniform.
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Frank Hess (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461777#comment-13461777 ]
Frank Hess commented on MATH-868:
---------------------------------
Also, in the Matlab code from http://www.lri.fr/~hansen I found a place where he defaults the sigma to 0.3 times the range:
insigma = 0.3*(ubounds-lbounds);
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Gilles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467227#comment-13467227 ]
Gilles commented on MATH-868:
-----------------------------
I think that it is the best we can do for now.
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Gilles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461294#comment-13461294 ]
Gilles commented on MATH-868:
-----------------------------
While running the code which you provided in MATH-867, I noticed that having an "inputSigma" either much larger or much smaller than 0.3 gives even worse results for that test (both when the optimum is near the lower bound and whne it's near the upper bound).
>From this, I'd suspect that the code is correct (always divided by the range), but that the documentation is wrong to suggest multiplying 0.3 by the range as a good value for "inputSigma".
I propose to modify the doc of "inputSigma", indicating that it should be about 0.3.
That looks strange, and the problem may well lie deeper (in how constraints are taken into account). Unless you have a test case, there isn't much more I can do, unfortunately.
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Frank Hess (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466829#comment-13466829 ]
Frank Hess commented on MATH-868:
---------------------------------
After the MATH-867 fix, it defaults inputSigma to 0.3 regardless of boundaries. This is effectively the same as the behavior I originally reported in this bug. I'm okay with that though, as the simpler code no longer makes it look like it is trying to do something fancier. I suggest the class be given a public DEFAULT_INPUTSIGMA constant equal to 0.3 (like the class's other DEFAULT_XXX constants) to make it explicit and documented what the default is.
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Gilles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480048#comment-13480048 ]
Gilles commented on MATH-868:
-----------------------------
Code and unit tests updated in revision 1400108.
Further changes postponed until preparation of 4.0 (see code marked with "XXX").
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
> Fix For: 4.0
>
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Gilles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478931#comment-13478931 ]
Gilles commented on MATH-868:
-----------------------------
Coming back to the issue the default value for sigma.
Currently, it is thus still 0.3 but IIUC, this might be a very bad default if the range of the parameters is large and the initial guess is far from the optimum (cf. Nikolaus's advice that it should be about one third of the initial search volume).
Nikolaus also mentioned that providing a default might simply not be a good idea at all.
Hence I'd suggest that if "inputSigma" is null, an exception be thrown (instead of making up a possibly meaningless default).
Unfortunately, that will break some applications (if they use the constructors without the "inputSigma" argument). Could the current behaviour (not specifying "inputSigma") be considered buggy enough to warrant a compatibility breaking?
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Gilles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466352#comment-13466352 ]
Gilles commented on MATH-868:
-----------------------------
The fix for MATH-867 probably fixes this issue too.
What do you think?
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Gilles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461128#comment-13461128 ]
Gilles commented on MATH-868:
-----------------------------
IIUC, the "inputSigma" should never be divided by the "range" but rather should be enforced to lie in the (0, 1) interval (and the documentation should suggest that "0.3" is a good default, and the value that is used when the user did not pass an "inputSigma" argument).
What do you think?
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Gilles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466912#comment-13466912 ]
Gilles commented on MATH-868:
-----------------------------
Quoting Nikolaus Hansen (from issue MATH-867):
bq. Points beyond, say, startpoint + 10*sigma are not likely to be sampled in the beginning.
and
bq. Codes I write typically don't accept a default value for inputSigma, [...]
I wonder whether it wouldn't be better to not have a default, i.e. deprecate the constructors that do not require the values to be specified.
The situation is even worse, I see now, in that "inputSigma" should not be passed at all at construction because it depends on the number of parameters and on their scale (values of the upper and lower bounds), whereas the interface to the CM optimizers was designed to allow multiple "optimize" calls to the _same_ optimizer with
* different functions (even with a different number of parameters),
* different start points,
* different boundaries.
All these arguments are passed with the call to "optimize". Ideally the "inputSigma" argument should be specified at that level too. But it can't because it's not (and cannot be) part of the general interface.
A solution to this API problem would be welcome for 4.0.
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Frank Hess (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13466999#comment-13466999 ]
Frank Hess commented on MATH-868:
---------------------------------
Well, the SimplexOptimizer handles this issue by having a "setSimplex" method which can be called before each call to optimize. CMAESOptimizer could have a "setInputSigma" method.
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MATH-868) Default sigma for CMAESOptimizer is
wrong when using bounds
Posted by "Gilles (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MATH-868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gilles updated MATH-868:
------------------------
Fix Version/s: 4.0
> Default sigma for CMAESOptimizer is wrong when using bounds
> -----------------------------------------------------------
>
> Key: MATH-868
> URL: https://issues.apache.org/jira/browse/MATH-868
> Project: Commons Math
> Issue Type: Bug
> Reporter: Frank Hess
> Fix For: 4.0
>
>
> The documentation suggests setting inputSigma to 1/3 the range you are fitting over. However, in CMAESOptimizer.initializeCMA() if boundaries are specified the sigmaArray is by default assigned a value of 0.3 divided by the range. If the user had specified the inputSigma to be 0.3 of the range (as suggested by the docs) then sigmaArray would have been assigned the value of 0.3. Thus, it looks like the 0.3 should not be divided by the range, only a user-specified inputSigma should get divided by the range.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira