You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Christoph Läubrich <la...@googlemail.com.INVALID> on 2021/06/08 06:14:10 UTC
Re: [math] example for constrain parameters for Least squares
Hi Gilles,
I have used the the INFINITY approach for a while now and it works quite
good. I just recently found a problem where I got very bad fits after a
handful of iterations using the LevenbergMarquardtOptimizer.
The problem arises whenever there is a relative small range of valid
values for *one* parameter.
Too keep up with the Gausian example assume that the mean is only valid
in a small window, but norm and sigma are completely free.
My guess is, if in the data there are outlier that indicates a strong
maximum outside this range the optimizer try to go in that 'direction'
because I reject this solution it 'gives up' as it seems evident that
there is no better solution. This then can result in a gausian that is
very thin and a really bad fit (cost e.g about 1E4).
If I help the optimizer (e.g. by adjusting the initial guess of sigma)
it finds a much better solution (cost about 1E-9).
So what I would need to tell the Optimizer (not sure if this is possible
at all!) that not the *whole* solution is bad, but only the choice of
*one* variable so it could use larger increments for the other variables.
Am 11.08.20 um 16:11 schrieb Gilles Sadowski:
> Hello.
>
> Le mar. 11 août 2020 à 12:08, Christoph Läubrich
> <la...@googlemail.com.invalid> a écrit :
>>
>> Thanks for your patience, maybe a better/simpler example would be [1], I
>> want to find the best fit using LSB
>
> "LSB" ?
>
>> under the constraint that the height
>> of the curve never is above 0.7 (even though without constrains the
>> optimizer would find a better solution around 8.5).
>
> It occurs to me that this would be more properly defined as a
> least-square problem by assigning an error bar (weight) to each
> data point.
>
>> So my first idea way to copy/extend GausianCurveFitter to accept some
>> kind of "maxHeight(...)", adjust the function to return INF for
>> h>maxHeight, I was just unsure that return INF is allowed (per
>> definition) as it is not mentioned anywhere in the userdocs. And if I
>> want to limit others (like max position) it would work the same way.
>
> Hopefully "infinity" will not cause an issue; you could try and check.
> [Then if it does, just use any suitably large value.]
>
>> In my final problem I need to limit the height, width and position of
>> the bell-curve to fall into certain bounds, but there is no direct
>> relation (e.g. width must be three times the height).
>
> Then, width is totally correlated (to height) and there are only
> 2 parameters to fit (height and mean); hence it's probably better
> (and more efficient) to use that fact instead of defining the
> correlation as a constraint i.e. define (untested):
> ---CUT---
> public class MyFunc implements ParametricUnivariateFunction {
> public double value(double x, double ... param) {
> final double diff = x - param[1]; // param[1] is the mean.
> final double norm = param[0]; // param[0] is the height.
> final double s = 3 * norm; // "constraint".
> final double i2s2 = 1 / (2 * s * s);
> return Gaussian.value(diff, norm, i2s2);
> }
>
> // And similarly for the "gradient":
> // https://gitbox.apache.org/repos/asf?p=commons-math.git;a=blob;f=src/main/java/org/apache/commons/math4/analysis/function/Gaussian.java;h=08dcac0d4d37179bc85a7071c84a6cb289c09f02;hb=HEAD#l143
> }
> ---CUT---
> to be passed to "SimpleCurveFitter".
>
> Regards,
> Gilles
>
>>
>>
>>
>> [1]
>> https://crlbucophysics101.files.wordpress.com/2015/02/gaussian.png?w=538&h=294
>>
>>
>>
>>
>> Am 11.08.20 um 11:18 schrieb Gilles Sadowski:
>>> Hi.
>>>
>>> 2020-08-11 8:51 UTC+02:00, Christoph Läubrich <la...@googlemail.com.invalid>:
>>>> Hi Gilles,
>>>>
>>>>
>>>> Just to make clear I don't suspect any error with GausianCurveFitter, I
>>>> just don't understand how the advice in the user-doc to restrict
>>>> parameter (for general problems) could be applied to a concrete problem
>>>> and thus chosen GausianCurvefitter as an example as it uses
>>>> LeastSquaresBuilder.
>>>>
>>>> I also noticed that Gaussian Fitter has a restriction on parameters
>>>> (norm can't be negative) that is handled in a third way (returning
>>>> Double.POSITIVE_INFINITY instead of Parameter Validator) not mentioned
>>>> in the userdoc at all, so I wonder if this is a general purpose solution
>>>> for restricting parameters (seems the simplest approach).
>>>
>>> I'd indeed suggest to first try the same trick as in "GaussianCurveFitter"
>>> (i.e. return a "high" value for arguments outside a known range).
>>> That way, you only have to define a suitable "ParametricUnivariateFunction"
>>> and pass it to "SimpleCurveFitter".
>>>
>>> One case for the "ParameterValidator" is when some of the model
>>> parameters might be correlated to others.
>>> But using it makes it necessary that you handle yourself all the
>>> arguments to be passed to the "LeastSquaresProblem".
>>>
>>>> To take the gausian example for my use case, consider an observed signal
>>>> similar to [1], given I know (from other source as the plain data) for
>>>> example that the result must be found in the range of 2...3 and I wanted
>>>> to restrict valid solutions to this area. The same might apply to the
>>>> norm: I know it must be between a given range and I want to restrict the
>>>> optimizer here even though there might be a solution outside of the
>>>> range that (compared of the R^2) fits better, e.g. a gausian fit well
>>>> inside the -1..1.
>>>>
>>>> I hope it is a little bit clearer.
>>>
>>> I'm not sure. The picture shows a function that is not a Gaussian.
>>> Do you mean that you want to fit only *part* of the data with a
>>> function that would not fit well *all* the data?
>>>
>>> Regards,
>>> Gilles
>>>
>>>>
>>>>
>>>> [1]
>>>> https://ascelibrary.org/cms/asset/6ca2b016-1a4f-4eed-80da-71219777cac1/1.jpg
>>>>
>>>> Am 11.08.20 um 00:42 schrieb Gilles Sadowski:
>>>>> Hello.
>>>>>
>>>>> Le lun. 10 août 2020 à 17:09, Christoph Läubrich
>>>>> <la...@googlemail.com.invalid> a écrit :
>>>>>>
>>>>>> The userguide [1] mentions that it is currently not directly possible to
>>>>>> contrain parameters directly but suggest one can use the
>>>>>> ParameterValidator, is there any example code for both mentioned
>>>>>> alternatives?
>>>>>>
>>>>>> For example GaussianCurveFitter uses LeastSquaresBuilder and I wan't to
>>>>>> archive that the mean is within a closed bound e.g from 5 to 6 where my
>>>>>> datapoints ranges from 0..90, how would this be archived?
>>>>>
>>>>> Could you set up a unit test as a practical example of what
>>>>> you need to achieve?
>>>>>
>>>>>> I'm especially interested because the FUNCTION inside
>>>>>> GaussianCurveFitter seems to reject invalid values (e.g. negative
>>>>>> valuenorm) by simply return Double.POSITIVE_INFINITY instead of using
>>>>>> either approach described in the user-docs.
>>>>>
>>>>> What I don't quite get is why you need to force the mean within a
>>>>> certain range; if the data match a Gaussian with a mean within that
>>>>> range, I would assume that the fitter will find the correct value...
>>>>> Sorry if I missed something. Hopefully the example will clarify.
>>>>>
>>>>> Best,
>>>>> Gilles
>>>>>
>>>>>>
>>>>>>
>>>>>> [1]
>>>>>> https://commons.apache.org/proper/commons-math/userguide/leastsquares.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
> For additional commands, e-mail: user-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [math] example for constrain parameters for Least squares
Posted by Christoph Läubrich <la...@googlemail.com.INVALID>.
Hi Gilles and Randy,
that sounds really interesting I'll try this out, both the sigmodial and
the scaling stuff!
I'm relative new to all this least squares things and I'm not a
mathematician so I need to learn a lot, so don't hesitate to point out
things that might seem obvious to the advanced user, that's really
welcome :-)
Am 08.06.21 um 15:23 schrieb Gilles Sadowski:
> Hello.
>
> Le mar. 8 juin 2021 à 08:14, Christoph Läubrich
> <la...@googlemail.com.invalid> a écrit :
>>
>> Hi Gilles,
>>
>> I have used the the INFINITY approach for a while now and it works quite
>> good. I just recently found a problem where I got very bad fits after a
>> handful of iterations using the LevenbergMarquardtOptimizer.
>>
>> The problem arises whenever there is a relative small range of valid
>> values for *one* parameter.
>>
>> Too keep up with the Gausian example assume that the mean is only valid
>> in a small window, but norm and sigma are completely free.
>>
>> My guess is, if in the data there are outlier that indicates a strong
>> maximum outside this range the optimizer try to go in that 'direction'
>> because I reject this solution it 'gives up' as it seems evident that
>> there is no better solution. This then can result in a gausian that is
>> very thin and a really bad fit (cost e.g about 1E4).
>>
>> If I help the optimizer (e.g. by adjusting the initial guess of sigma)
>> it finds a much better solution (cost about 1E-9).
>>
>> So what I would need to tell the Optimizer (not sure if this is possible
>> at all!) that not the *whole* solution is bad, but only the choice of
>> *one* variable so it could use larger increments for the other variables.
>
> If you want to restrict the range of, say, the mean:
>
> public class MyFunc implements ParametricUnivariateFunction {
> private final Sigmoid meanTransform;
>
> public MyFunc(double minMean, double maxMean) {
> meanTransform = new Sigmoid(minMean, maxMean);
> }
>
> public double value(double x, double ... param) {
> final double mu = meanTransform.value(param[1]); // param[1]
> is the mean.
> final double diff = x - mu;
> final double norm = param[0]; // param[0] is the height.
> final double s = param[2]; // param[2] is the standard deviation.
> final double i2s2 = 1 / (2 * s * s);
> return Gaussian.value(diff, norm, i2s2);
> }
> }
>
> // ...
> final MyFunc f = new MyFunc(min, max);
> final double[] best = fitter.fit(f); // Perform fit.
> final double bestMean = new Logit(min, max).value(best[1]);
>
> HTH,
> Gilles
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [math] example for constrain parameters for Least squares
Posted by Randy Motluck <r_...@yahoo.com.INVALID>.
Hi Gilles,
I may be wrong in understanding your issue, but in general, I believe you should scale your features.
https://towardsdatascience.com/gradient-descent-the-learning-rate-and-the-importance-of-feature-scaling-6c0b416596e1
https://societyofai.medium.com/simplest-way-for-feature-scaling-in-gradient-descent-ae0aaa383039
It becomes important when one feature is of greatly different scale than another. Say for example X is 0-100.0 while Y is 0-1000000000 or something.
Hope this finds you well and best of luck,
Randy Motluck
On Tuesday, June 8, 2021, 08:24:54 AM CDT, Gilles Sadowski <gi...@gmail.com> wrote:
Hello.
Le mar. 8 juin 2021 à 08:14, Christoph Läubrich
<la...@googlemail.com.invalid> a écrit :
>
> Hi Gilles,
>
> I have used the the INFINITY approach for a while now and it works quite
> good. I just recently found a problem where I got very bad fits after a
> handful of iterations using the LevenbergMarquardtOptimizer.
>
> The problem arises whenever there is a relative small range of valid
> values for *one* parameter.
>
> Too keep up with the Gausian example assume that the mean is only valid
> in a small window, but norm and sigma are completely free.
>
> My guess is, if in the data there are outlier that indicates a strong
> maximum outside this range the optimizer try to go in that 'direction'
> because I reject this solution it 'gives up' as it seems evident that
> there is no better solution. This then can result in a gausian that is
> very thin and a really bad fit (cost e.g about 1E4).
>
> If I help the optimizer (e.g. by adjusting the initial guess of sigma)
> it finds a much better solution (cost about 1E-9).
>
> So what I would need to tell the Optimizer (not sure if this is possible
> at all!) that not the *whole* solution is bad, but only the choice of
> *one* variable so it could use larger increments for the other variables.
If you want to restrict the range of, say, the mean:
public class MyFunc implements ParametricUnivariateFunction {
private final Sigmoid meanTransform;
public MyFunc(double minMean, double maxMean) {
meanTransform = new Sigmoid(minMean, maxMean);
}
public double value(double x, double ... param) {
final double mu = meanTransform.value(param[1]); // param[1]
is the mean.
final double diff = x - mu;
final double norm = param[0]; // param[0] is the height.
final double s = param[2]; // param[2] is the standard deviation.
final double i2s2 = 1 / (2 * s * s);
return Gaussian.value(diff, norm, i2s2);
}
}
// ...
final MyFunc f = new MyFunc(min, max);
final double[] best = fitter.fit(f); // Perform fit.
final double bestMean = new Logit(min, max).value(best[1]);
HTH,
Gilles
>>>> [...]
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org
Re: [math] example for constrain parameters for Least squares
Posted by Gilles Sadowski <gi...@gmail.com>.
Hello.
Le mar. 8 juin 2021 à 08:14, Christoph Läubrich
<la...@googlemail.com.invalid> a écrit :
>
> Hi Gilles,
>
> I have used the the INFINITY approach for a while now and it works quite
> good. I just recently found a problem where I got very bad fits after a
> handful of iterations using the LevenbergMarquardtOptimizer.
>
> The problem arises whenever there is a relative small range of valid
> values for *one* parameter.
>
> Too keep up with the Gausian example assume that the mean is only valid
> in a small window, but norm and sigma are completely free.
>
> My guess is, if in the data there are outlier that indicates a strong
> maximum outside this range the optimizer try to go in that 'direction'
> because I reject this solution it 'gives up' as it seems evident that
> there is no better solution. This then can result in a gausian that is
> very thin and a really bad fit (cost e.g about 1E4).
>
> If I help the optimizer (e.g. by adjusting the initial guess of sigma)
> it finds a much better solution (cost about 1E-9).
>
> So what I would need to tell the Optimizer (not sure if this is possible
> at all!) that not the *whole* solution is bad, but only the choice of
> *one* variable so it could use larger increments for the other variables.
If you want to restrict the range of, say, the mean:
public class MyFunc implements ParametricUnivariateFunction {
private final Sigmoid meanTransform;
public MyFunc(double minMean, double maxMean) {
meanTransform = new Sigmoid(minMean, maxMean);
}
public double value(double x, double ... param) {
final double mu = meanTransform.value(param[1]); // param[1]
is the mean.
final double diff = x - mu;
final double norm = param[0]; // param[0] is the height.
final double s = param[2]; // param[2] is the standard deviation.
final double i2s2 = 1 / (2 * s * s);
return Gaussian.value(diff, norm, i2s2);
}
}
// ...
final MyFunc f = new MyFunc(min, max);
final double[] best = fitter.fit(f); // Perform fit.
final double bestMean = new Logit(min, max).value(best[1]);
HTH,
Gilles
>>>> [...]
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@commons.apache.org
For additional commands, e-mail: user-help@commons.apache.org