You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Olexiy Movchan <om...@apriori.com> on 2014/09/04 11:37:33 UTC
[math] Side effect of LevenbergMarquardtOptimizer
Hello,
I created the math issue https://issues.apache.org/jira/browse/MATH-1144.
In version 2.0, LevenbergMarquardtOptimizer passed point to evaluator by reference. So our software could modify it on every step of algorithm.
In version 3.3, point is copied and then passed to evaluator, so it can't be updated by evaluator.
We use LevenbergMarquardtOptimizer for 3d surface fitting (cylinders, cones, tori) by sampled points. And surface parameters should be renormalized on every step of algorithm. Please see this article:
http://nvlpubs.nist.gov/nistpubs/jres/103/6/j36sha.pdf
Also please read the description of MATH-1144 jira issue.
Can you modify optimizer or evaluator interface to allow in/out parameters there?
Thanks,
Olexiy Movchan
Re: [math] Side effect of LevenbergMarquardtOptimizer
Posted by Evan Ward <ev...@nrl.navy.mil>.
On 09/08/2014 12:06 PM, Gilles wrote:
> On Mon, 8 Sep 2014 11:50:43 -0400, Evan Ward wrote:
>> I was suggesting (though not clearly :) keeping the
>> MultivariateJacobianFunction interface as is and adding a new interface
>> for any post processing of the evaluated point.
>
> Isn't it rather pre-processing (the point is changed before evaluation)?
before evaluation and after computing the point. We're talking about the
same thing.
>
>> Something like:
>>
>> interface StepFinalizer {
>>
>> RealVector apply(RealVector point);
>>
>> }
>>
>> Then we would add another getter/setter to LeastSquaresProblem for an
>> instance StepFinalizer.
>>
>> As I think more about it, I think this interface could also be used to
>> satisfy another feature request we've had in the past: recording the
>> trajectory that the optimizer takes. In this case the step finalizer
>> would have a side effect of adding the point to a list for later
>> retrieval. What do you think?
>
> In that latter case, it would not be clear (IMHO) which point was used by
> the optimizer and which should recorded in the list... I'd keep the
> issue
> separate and use an informative name for the new "validating" interface.
> [Even if it could indeed be diverted to allow tracking, but this was
> already the case with the "ConvergenceChecker".]
Thats a good point. Sounds good to me.
>
> Best,
> Gilles
>
>>>> [...]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [math] Side effect of LevenbergMarquardtOptimizer
Posted by Gilles <gi...@harfang.homelinux.org>.
On Mon, 8 Sep 2014 11:50:43 -0400, Evan Ward wrote:
> I was suggesting (though not clearly :) keeping the
> MultivariateJacobianFunction interface as is and adding a new
> interface
> for any post processing of the evaluated point.
Isn't it rather pre-processing (the point is changed before
evaluation)?
> Something like:
>
> interface StepFinalizer {
>
> RealVector apply(RealVector point);
>
> }
>
> Then we would add another getter/setter to LeastSquaresProblem for an
> instance StepFinalizer.
>
> As I think more about it, I think this interface could also be used
> to
> satisfy another feature request we've had in the past: recording the
> trajectory that the optimizer takes. In this case the step finalizer
> would have a side effect of adding the point to a list for later
> retrieval. What do you think?
In that latter case, it would not be clear (IMHO) which point was used
by
the optimizer and which should recorded in the list... I'd keep the
issue
separate and use an informative name for the new "validating"
interface.
[Even if it could indeed be diverted to allow tracking, but this was
already the case with the "ConvergenceChecker".]
Best,
Gilles
>>> [...]
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [math] Side effect of LevenbergMarquardtOptimizer
Posted by Evan Ward <ev...@nrl.navy.mil>.
I was suggesting (though not clearly :) keeping the
MultivariateJacobianFunction interface as is and adding a new interface
for any post processing of the evaluated point. Something like:
interface StepFinalizer {
RealVector apply(RealVector point);
}
Then we would add another getter/setter to LeastSquaresProblem for an
instance StepFinalizer.
As I think more about it, I think this interface could also be used to
satisfy another feature request we've had in the past: recording the
trajectory that the optimizer takes. In this case the step finalizer
would have a side effect of adding the point to a list for later
retrieval. What do you think?
Best Regards,
Evan
On 09/08/2014 11:13 AM, Gilles wrote:
> Hello.
>
>
>
> On Mon, 8 Sep 2014 08:56:36 -0400, Evan Ward wrote:
>
>> Gilles,
>>
>>
>>
>> I like this approach. My only thought is that a separate interface
>> for
>>
>> the validate method would be nicer for our Java 8 users. Then the
>>
>> implementation could be a lambda: (p) -> p.unitVector()
>>
>
>
> Do you mean define an interface like
>
>
>
> interface ValidatingMultivariateJacobianFunction
>
> extends MultivariateJacobianFunction {
>
>
>
> /**
>
> * @param point Point provided by the optimizer.
>
> * @return the point that will actually be evaluated.
>
> */
>
> RealVector validate(RealVector point);
>
> }
>
>
>
> and then use "instanceof" to check whether validation should occur
>
> or not?
>
>
>
> Gilles
>
>
>
>> Best Regards,
>>
>> Evan
>>
>>
>>
>> On 09/07/2014 08:11 PM, Gilles wrote:
>>
>>> On Thu, 4 Sep 2014 12:52:24 -0400, Evan Ward wrote:
>>>
>>>> Hi Olexiy,
>>>>
>>>>
>>>>
>>>> In my field we often encounter a similar problem when estimating
>>>>
>>>> attitude since a quaternion is only a valid rotation when it is
>>>>
>>>> normalized. We often escape this issue by estimating a "small"
>>>>
>>>> adjustment to an apriori guess. (For the details see [1].) For
>>>> this
>>>>
>>>> technique to work the cost function must be smooth and the apriori
>>>> guess
>>>>
>>>> must be "close enough" to the true value. Both of these assumptions
>>>> are
>>>>
>>>> also required to apply a non-linear least squares optimizer.
>>>> Perhaps you
>>>>
>>>> can apply a similar technique to your problem. (It seems that your
>>>> 'A'
>>>>
>>>> parameter is orientation in 3D space.)
>>>>
>>>>
>>>>
>>>> If there is a need for an extra steps, I would prefer to make those
>>>>
>>>> explicit rather than depending on side effects of cost function
>>>>
>>>> evaluation.
>>>>
>>>
>>>
>>> IIUC, the feature could be made explicit by adding a method to the
>>>
>>> "MultivariateJacobianFunction" interface to allow the user to change
>>>
>>> the point about to be evaluated:
>>>
>>>
>>>
>>> interface MultivariateJacobianFunction {
>>>
>>> Pair<RealVector, RealMatrix> value(RealVector point);
>>>
>>>
>>>
>>> /** @param point Point provided by the optimizer. */
>>>
>>> /** @return the point that will actually be evaluated. */
>>>
>>> RealVector validate(RealVector point);
>>>
>>> }
>>>
>>>
>>>
>>> Thus, in "LeastSquaresFactory":
>>>
>>>
>>>
>>> private static class LocalLeastSquaresProblem
>>>
>>> extends AbstractOptimizationProblem<Evaluation>
>>>
>>> implements LeastSquaresProblem {
>>>
>>>
>>>
>>> // ...
>>>
>>>
>>>
>>> private final MultivariateJacobianFunction model;
>>>
>>>
>>>
>>> // ...
>>>
>>>
>>>
>>> public Evaluation evaluate(final RealVector point) {
>>>
>>> final RealVector p = model.validate(point).copy(); // <---
>>> Change
>>>
>>> here (at line 403).
>>>
>>>
>>>
>>> if (lazyEvaluation) {
>>>
>>> return new LazyUnweightedEvaluation(model,
>>>
>>> target,
>>>
>>> p);
>>>
>>> } else {
>>>
>>> final Pair<RealVector, RealMatrix> value = model.value(p);
>>>
>>> return new UnweightedEvaluation(value.getFirst(),
>>>
>>> value.getSecond(),
>>>
>>> target,
>>>
>>> p);
>>>
>>> }
>>>
>>> }
>>>
>>>
>>>
>>> // ...
>>>
>>> }
>>>
>>>
>>>
>>> What do you think?
>>>
>>>
>>>
>>> Best,
>>>
>>> Gilles
>>>
>>>
>>>
>>>>
>>>>
>>>> Best Regards,
>>>>
>>>> Evan
>>>>
>>>>
>>>>
>>>> [1] Crassidis, John L., and John L. Junkins. /Optimal Estimation of
>>>>
>>>> Dynamic Systems/. Boca Raton, FL: CRC, 2012.
>>>>
>>>>
>>>>
>>>> On 09/04/2014 05:37 AM, Olexiy Movchan wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>>
>>>>>
>>>>> I created the math issue
>>>>>
>>>>> https://issues.apache.org/jira/browse/MATH-1144.
>>>>>
>>>>>
>>>>>
>>>>> In version 2.0, LevenbergMarquardtOptimizer passed point to
>>>>>
>>>>> evaluator by reference. So our software could modify it on every
>>>>>
>>>>> step of algorithm.
>>>>>
>>>>> In version 3.3, point is copied and then passed to evaluator, so
>>>>> it
>>>>>
>>>>> can't be updated by evaluator.
>>>>>
>>>>>
>>>>>
>>>>> We use LevenbergMarquardtOptimizer for 3d surface fitting
>>>>>
>>>>> (cylinders, cones, tori) by sampled points. And surface parameters
>>>>>
>>>>> should be renormalized on every step of algorithm. Please see this
>>>>>
>>>>> article:
>>>>>
>>>>>
>>>>> http://nvlpubs.nist.gov/nistpubs/jres/103/6/j36sha.pdf
>>>>>
>>>>>
>>>>>
>>>>> Also please read the description of MATH-1144 jira issue.
>>>>>
>>>>>
>>>>>
>>>>> Can you modify optimizer or evaluator interface to allow in/out
>>>>>
>>>>> parameters there?
>>>>>
>>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Olexiy Movchan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>>
>>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>>
>>> For additional commands, e-mail: dev-help@commons.apache.org
>>>
>>>
>>>
>>
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>>
>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>>
>> For additional commands, e-mail: dev-help@commons.apache.org
>>
>
>
>
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [math] Side effect of LevenbergMarquardtOptimizer
Posted by Gilles <gi...@harfang.homelinux.org>.
Hello.
On Mon, 8 Sep 2014 08:56:36 -0400, Evan Ward wrote:
> Gilles,
>
> I like this approach. My only thought is that a separate interface
> for
> the validate method would be nicer for our Java 8 users. Then the
> implementation could be a lambda: (p) -> p.unitVector()
Do you mean define an interface like
interface ValidatingMultivariateJacobianFunction
extends MultivariateJacobianFunction {
/**
* @param point Point provided by the optimizer.
* @return the point that will actually be evaluated.
*/
RealVector validate(RealVector point);
}
and then use "instanceof" to check whether validation should occur
or not?
Gilles
> Best Regards,
> Evan
>
> On 09/07/2014 08:11 PM, Gilles wrote:
>> On Thu, 4 Sep 2014 12:52:24 -0400, Evan Ward wrote:
>>> Hi Olexiy,
>>>
>>> In my field we often encounter a similar problem when estimating
>>> attitude since a quaternion is only a valid rotation when it is
>>> normalized. We often escape this issue by estimating a "small"
>>> adjustment to an apriori guess. (For the details see [1].) For
>>> this
>>> technique to work the cost function must be smooth and the apriori
>>> guess
>>> must be "close enough" to the true value. Both of these assumptions
>>> are
>>> also required to apply a non-linear least squares optimizer.
>>> Perhaps you
>>> can apply a similar technique to your problem. (It seems that your
>>> 'A'
>>> parameter is orientation in 3D space.)
>>>
>>> If there is a need for an extra steps, I would prefer to make those
>>> explicit rather than depending on side effects of cost function
>>> evaluation.
>>
>> IIUC, the feature could be made explicit by adding a method to the
>> "MultivariateJacobianFunction" interface to allow the user to change
>> the point about to be evaluated:
>>
>> interface MultivariateJacobianFunction {
>> Pair<RealVector, RealMatrix> value(RealVector point);
>>
>> /** @param point Point provided by the optimizer. */
>> /** @return the point that will actually be evaluated. */
>> RealVector validate(RealVector point);
>> }
>>
>> Thus, in "LeastSquaresFactory":
>>
>> private static class LocalLeastSquaresProblem
>> extends AbstractOptimizationProblem<Evaluation>
>> implements LeastSquaresProblem {
>>
>> // ...
>>
>> private final MultivariateJacobianFunction model;
>>
>> // ...
>>
>> public Evaluation evaluate(final RealVector point) {
>> final RealVector p = model.validate(point).copy(); // <---
>> Change
>> here (at line 403).
>>
>> if (lazyEvaluation) {
>> return new LazyUnweightedEvaluation(model,
>> target,
>> p);
>> } else {
>> final Pair<RealVector, RealMatrix> value = model.value(p);
>> return new UnweightedEvaluation(value.getFirst(),
>> value.getSecond(),
>> target,
>> p);
>> }
>> }
>>
>> // ...
>> }
>>
>> What do you think?
>>
>> Best,
>> Gilles
>>
>>>
>>> Best Regards,
>>> Evan
>>>
>>> [1] Crassidis, John L., and John L. Junkins. /Optimal Estimation of
>>> Dynamic Systems/. Boca Raton, FL: CRC, 2012.
>>>
>>> On 09/04/2014 05:37 AM, Olexiy Movchan wrote:
>>>> Hello,
>>>>
>>>> I created the math issue
>>>> https://issues.apache.org/jira/browse/MATH-1144.
>>>>
>>>> In version 2.0, LevenbergMarquardtOptimizer passed point to
>>>> evaluator by reference. So our software could modify it on every
>>>> step of algorithm.
>>>> In version 3.3, point is copied and then passed to evaluator, so
>>>> it
>>>> can't be updated by evaluator.
>>>>
>>>> We use LevenbergMarquardtOptimizer for 3d surface fitting
>>>> (cylinders, cones, tori) by sampled points. And surface parameters
>>>> should be renormalized on every step of algorithm. Please see this
>>>> article:
>>>>
>>>> http://nvlpubs.nist.gov/nistpubs/jres/103/6/j36sha.pdf
>>>>
>>>> Also please read the description of MATH-1144 jira issue.
>>>>
>>>> Can you modify optimizer or evaluator interface to allow in/out
>>>> parameters there?
>>>>
>>>> Thanks,
>>>> Olexiy Movchan
>>>>
>>>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> For additional commands, e-mail: dev-help@commons.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [math] Side effect of LevenbergMarquardtOptimizer
Posted by Evan Ward <ev...@nrl.navy.mil>.
Gilles,
I like this approach. My only thought is that a separate interface for
the validate method would be nicer for our Java 8 users. Then the
implementation could be a lambda: (p) -> p.unitVector()
Best Regards,
Evan
On 09/07/2014 08:11 PM, Gilles wrote:
> On Thu, 4 Sep 2014 12:52:24 -0400, Evan Ward wrote:
>> Hi Olexiy,
>>
>> In my field we often encounter a similar problem when estimating
>> attitude since a quaternion is only a valid rotation when it is
>> normalized. We often escape this issue by estimating a "small"
>> adjustment to an apriori guess. (For the details see [1].) For this
>> technique to work the cost function must be smooth and the apriori guess
>> must be "close enough" to the true value. Both of these assumptions are
>> also required to apply a non-linear least squares optimizer. Perhaps you
>> can apply a similar technique to your problem. (It seems that your 'A'
>> parameter is orientation in 3D space.)
>>
>> If there is a need for an extra steps, I would prefer to make those
>> explicit rather than depending on side effects of cost function
>> evaluation.
>
> IIUC, the feature could be made explicit by adding a method to the
> "MultivariateJacobianFunction" interface to allow the user to change
> the point about to be evaluated:
>
> interface MultivariateJacobianFunction {
> Pair<RealVector, RealMatrix> value(RealVector point);
>
> /** @param point Point provided by the optimizer. */
> /** @return the point that will actually be evaluated. */
> RealVector validate(RealVector point);
> }
>
> Thus, in "LeastSquaresFactory":
>
> private static class LocalLeastSquaresProblem
> extends AbstractOptimizationProblem<Evaluation>
> implements LeastSquaresProblem {
>
> // ...
>
> private final MultivariateJacobianFunction model;
>
> // ...
>
> public Evaluation evaluate(final RealVector point) {
> final RealVector p = model.validate(point).copy(); // <--- Change
> here (at line 403).
>
> if (lazyEvaluation) {
> return new LazyUnweightedEvaluation(model,
> target,
> p);
> } else {
> final Pair<RealVector, RealMatrix> value = model.value(p);
> return new UnweightedEvaluation(value.getFirst(),
> value.getSecond(),
> target,
> p);
> }
> }
>
> // ...
> }
>
> What do you think?
>
> Best,
> Gilles
>
>>
>> Best Regards,
>> Evan
>>
>> [1] Crassidis, John L., and John L. Junkins. /Optimal Estimation of
>> Dynamic Systems/. Boca Raton, FL: CRC, 2012.
>>
>> On 09/04/2014 05:37 AM, Olexiy Movchan wrote:
>>> Hello,
>>>
>>> I created the math issue
>>> https://issues.apache.org/jira/browse/MATH-1144.
>>>
>>> In version 2.0, LevenbergMarquardtOptimizer passed point to
>>> evaluator by reference. So our software could modify it on every
>>> step of algorithm.
>>> In version 3.3, point is copied and then passed to evaluator, so it
>>> can't be updated by evaluator.
>>>
>>> We use LevenbergMarquardtOptimizer for 3d surface fitting
>>> (cylinders, cones, tori) by sampled points. And surface parameters
>>> should be renormalized on every step of algorithm. Please see this
>>> article:
>>> http://nvlpubs.nist.gov/nistpubs/jres/103/6/j36sha.pdf
>>>
>>> Also please read the description of MATH-1144 jira issue.
>>>
>>> Can you modify optimizer or evaluator interface to allow in/out
>>> parameters there?
>>>
>>> Thanks,
>>> Olexiy Movchan
>>>
>>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
RE: [math] Side effect of LevenbergMarquardtOptimizer
Posted by Gilles <gi...@harfang.homelinux.org>.
On Mon, 8 Sep 2014 08:57:18 +0000, Olexiy Movchan wrote:
> Hi Gilles,
>
> Looks good to me. It is the explicit update now. I would like to
> propose different names for the new function:
> "applyConstraints()" or "renormalize()"
I had thought about the second one.
But, IMO, the (cosmetic) problem would be that those names could
seem to imply that the method is optional (i.e. users that don't
need the feature might implement a method that returns "null"),
whereas a validation is always applied (and the no-op behaviour
is thus to return the point).
Regards,
Gilles
> Thanks,
> Olexiy
>
> -----Original Message-----
> From: Gilles [mailto:gilles@harfang.homelinux.org]
> Sent: Monday, September 08, 2014 3:11 AM
> To: dev@commons.apache.org
> Subject: Re: [math] Side effect of LevenbergMarquardtOptimizer
>
> On Thu, 4 Sep 2014 12:52:24 -0400, Evan Ward wrote:
>> Hi Olexiy,
>>
>> In my field we often encounter a similar problem when estimating
>> attitude since a quaternion is only a valid rotation when it is
>> normalized. We often escape this issue by estimating a "small"
>> adjustment to an apriori guess. (For the details see [1].) For this
>> technique to work the cost function must be smooth and the apriori
>> guess must be "close enough" to the true value. Both of these
>> assumptions are also required to apply a non-linear least squares
>> optimizer. Perhaps you can apply a similar technique to your
>> problem.
>> (It seems that your 'A'
>> parameter is orientation in 3D space.)
>>
>> If there is a need for an extra steps, I would prefer to make those
>> explicit rather than depending on side effects of cost function
>> evaluation.
>
> IIUC, the feature could be made explicit by adding a method to the
> "MultivariateJacobianFunction" interface to allow the user to change
> the point about to be evaluated:
>
> interface MultivariateJacobianFunction {
> Pair<RealVector, RealMatrix> value(RealVector point);
>
> /** @param point Point provided by the optimizer. */
> /** @return the point that will actually be evaluated. */
> RealVector validate(RealVector point); }
>
> Thus, in "LeastSquaresFactory":
>
> private static class LocalLeastSquaresProblem
> extends AbstractOptimizationProblem<Evaluation>
> implements LeastSquaresProblem {
>
> // ...
>
> private final MultivariateJacobianFunction model;
>
> // ...
>
> public Evaluation evaluate(final RealVector point) {
> final RealVector p = model.validate(point).copy(); // <---
> Change here (at line 403).
>
> if (lazyEvaluation) {
> return new LazyUnweightedEvaluation(model,
> target,
> p);
> } else {
> final Pair<RealVector, RealMatrix> value = model.value(p);
> return new UnweightedEvaluation(value.getFirst(),
> value.getSecond(),
> target,
> p);
> }
> }
>
> // ...
> }
>
> What do you think?
>
> Best,
> Gilles
>
>>
>> Best Regards,
>> Evan
>>
>> [1] Crassidis, John L., and John L. Junkins. /Optimal Estimation of
>> Dynamic Systems/. Boca Raton, FL: CRC, 2012.
>>
>> On 09/04/2014 05:37 AM, Olexiy Movchan wrote:
>>> Hello,
>>>
>>> I created the math issue
>>> https://issues.apache.org/jira/browse/MATH-1144.
>>>
>>> In version 2.0, LevenbergMarquardtOptimizer passed point to
>>> evaluator
>>> by reference. So our software could modify it on every step of
>>> algorithm.
>>> In version 3.3, point is copied and then passed to evaluator, so it
>>> can't be updated by evaluator.
>>>
>>> We use LevenbergMarquardtOptimizer for 3d surface fitting
>>> (cylinders,
>>> cones, tori) by sampled points. And surface parameters should be
>>> renormalized on every step of algorithm. Please see this
>>> article:
>>>
>>> http://nvlpubs.nist.gov/nistpubs/jres/103/6/j36sha.pdf
>>>
>>> Also please read the description of MATH-1144 jira issue.
>>>
>>> Can you modify optimizer or evaluator interface to allow in/out
>>> parameters there?
>>>
>>> Thanks,
>>> Olexiy Movchan
>>>
>>>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
RE: [math] Side effect of LevenbergMarquardtOptimizer
Posted by Olexiy Movchan <om...@apriori.com>.
Hi Gilles,
Looks good to me. It is the explicit update now. I would like to propose different names for the new function:
"applyConstraints()" or "renormalize()"
Thanks,
Olexiy
-----Original Message-----
From: Gilles [mailto:gilles@harfang.homelinux.org]
Sent: Monday, September 08, 2014 3:11 AM
To: dev@commons.apache.org
Subject: Re: [math] Side effect of LevenbergMarquardtOptimizer
On Thu, 4 Sep 2014 12:52:24 -0400, Evan Ward wrote:
> Hi Olexiy,
>
> In my field we often encounter a similar problem when estimating
> attitude since a quaternion is only a valid rotation when it is
> normalized. We often escape this issue by estimating a "small"
> adjustment to an apriori guess. (For the details see [1].) For this
> technique to work the cost function must be smooth and the apriori
> guess must be "close enough" to the true value. Both of these
> assumptions are also required to apply a non-linear least squares
> optimizer. Perhaps you can apply a similar technique to your problem.
> (It seems that your 'A'
> parameter is orientation in 3D space.)
>
> If there is a need for an extra steps, I would prefer to make those
> explicit rather than depending on side effects of cost function
> evaluation.
IIUC, the feature could be made explicit by adding a method to the "MultivariateJacobianFunction" interface to allow the user to change the point about to be evaluated:
interface MultivariateJacobianFunction {
Pair<RealVector, RealMatrix> value(RealVector point);
/** @param point Point provided by the optimizer. */
/** @return the point that will actually be evaluated. */
RealVector validate(RealVector point); }
Thus, in "LeastSquaresFactory":
private static class LocalLeastSquaresProblem
extends AbstractOptimizationProblem<Evaluation>
implements LeastSquaresProblem {
// ...
private final MultivariateJacobianFunction model;
// ...
public Evaluation evaluate(final RealVector point) {
final RealVector p = model.validate(point).copy(); // <--- Change here (at line 403).
if (lazyEvaluation) {
return new LazyUnweightedEvaluation(model,
target,
p);
} else {
final Pair<RealVector, RealMatrix> value = model.value(p);
return new UnweightedEvaluation(value.getFirst(),
value.getSecond(),
target,
p);
}
}
// ...
}
What do you think?
Best,
Gilles
>
> Best Regards,
> Evan
>
> [1] Crassidis, John L., and John L. Junkins. /Optimal Estimation of
> Dynamic Systems/. Boca Raton, FL: CRC, 2012.
>
> On 09/04/2014 05:37 AM, Olexiy Movchan wrote:
>> Hello,
>>
>> I created the math issue
>> https://issues.apache.org/jira/browse/MATH-1144.
>>
>> In version 2.0, LevenbergMarquardtOptimizer passed point to evaluator
>> by reference. So our software could modify it on every step of
>> algorithm.
>> In version 3.3, point is copied and then passed to evaluator, so it
>> can't be updated by evaluator.
>>
>> We use LevenbergMarquardtOptimizer for 3d surface fitting (cylinders,
>> cones, tori) by sampled points. And surface parameters should be
>> renormalized on every step of algorithm. Please see this
>> article:
>>
>> http://nvlpubs.nist.gov/nistpubs/jres/103/6/j36sha.pdf
>>
>> Also please read the description of MATH-1144 jira issue.
>>
>> Can you modify optimizer or evaluator interface to allow in/out
>> parameters there?
>>
>> Thanks,
>> Olexiy Movchan
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [math] Side effect of LevenbergMarquardtOptimizer
Posted by Gilles <gi...@harfang.homelinux.org>.
On Thu, 4 Sep 2014 12:52:24 -0400, Evan Ward wrote:
> Hi Olexiy,
>
> In my field we often encounter a similar problem when estimating
> attitude since a quaternion is only a valid rotation when it is
> normalized. We often escape this issue by estimating a "small"
> adjustment to an apriori guess. (For the details see [1].) For this
> technique to work the cost function must be smooth and the apriori
> guess
> must be "close enough" to the true value. Both of these assumptions
> are
> also required to apply a non-linear least squares optimizer. Perhaps
> you
> can apply a similar technique to your problem. (It seems that your
> 'A'
> parameter is orientation in 3D space.)
>
> If there is a need for an extra steps, I would prefer to make those
> explicit rather than depending on side effects of cost function
> evaluation.
IIUC, the feature could be made explicit by adding a method to the
"MultivariateJacobianFunction" interface to allow the user to change
the point about to be evaluated:
interface MultivariateJacobianFunction {
Pair<RealVector, RealMatrix> value(RealVector point);
/** @param point Point provided by the optimizer. */
/** @return the point that will actually be evaluated. */
RealVector validate(RealVector point);
}
Thus, in "LeastSquaresFactory":
private static class LocalLeastSquaresProblem
extends AbstractOptimizationProblem<Evaluation>
implements LeastSquaresProblem {
// ...
private final MultivariateJacobianFunction model;
// ...
public Evaluation evaluate(final RealVector point) {
final RealVector p = model.validate(point).copy(); // <--- Change
here (at line 403).
if (lazyEvaluation) {
return new LazyUnweightedEvaluation(model,
target,
p);
} else {
final Pair<RealVector, RealMatrix> value = model.value(p);
return new UnweightedEvaluation(value.getFirst(),
value.getSecond(),
target,
p);
}
}
// ...
}
What do you think?
Best,
Gilles
>
> Best Regards,
> Evan
>
> [1] Crassidis, John L., and John L. Junkins. /Optimal Estimation of
> Dynamic Systems/. Boca Raton, FL: CRC, 2012.
>
> On 09/04/2014 05:37 AM, Olexiy Movchan wrote:
>> Hello,
>>
>> I created the math issue
>> https://issues.apache.org/jira/browse/MATH-1144.
>>
>> In version 2.0, LevenbergMarquardtOptimizer passed point to
>> evaluator by reference. So our software could modify it on every step
>> of algorithm.
>> In version 3.3, point is copied and then passed to evaluator, so it
>> can't be updated by evaluator.
>>
>> We use LevenbergMarquardtOptimizer for 3d surface fitting
>> (cylinders, cones, tori) by sampled points. And surface parameters
>> should be renormalized on every step of algorithm. Please see this
>> article:
>>
>> http://nvlpubs.nist.gov/nistpubs/jres/103/6/j36sha.pdf
>>
>> Also please read the description of MATH-1144 jira issue.
>>
>> Can you modify optimizer or evaluator interface to allow in/out
>> parameters there?
>>
>> Thanks,
>> Olexiy Movchan
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [math] Side effect of LevenbergMarquardtOptimizer
Posted by Evan Ward <ev...@nrl.navy.mil>.
On 09/08/2014 04:50 AM, Olexiy Movchan wrote:
> Hi Evan,
>
> Possibly a small jitter of initial guess would solve this issue. But it is hard to tell if this method guaranties convergence in all problematic cases.
>
> Normalization approach already works and allows to converge in those cases.
For non-linear least squares you already need an accurate initial guess,
though I could see the convergence region being different between the
two methods. Have you tried it?
Best Regards,
Evan
>
> Thanks,
> Olexiy
>
> -----Original Message-----
> From: Evan Ward [mailto:evan.ward@nrl.navy.mil]
> Sent: Thursday, September 04, 2014 7:52 PM
> To: dev@commons.apache.org
> Subject: Re: [math] Side effect of LevenbergMarquardtOptimizer
>
> Hi Olexiy,
>
> In my field we often encounter a similar problem when estimating attitude since a quaternion is only a valid rotation when it is normalized. We often escape this issue by estimating a "small"
> adjustment to an apriori guess. (For the details see [1].) For this technique to work the cost function must be smooth and the apriori guess must be "close enough" to the true value. Both of these assumptions are also required to apply a non-linear least squares optimizer. Perhaps you can apply a similar technique to your problem. (It seems that your 'A'
> parameter is orientation in 3D space.)
>
> If there is a need for an extra steps, I would prefer to make those explicit rather than depending on side effects of cost function evaluation.
>
> Best Regards,
> Evan
>
> [1] Crassidis, John L., and John L. Junkins. /Optimal Estimation of Dynamic Systems/. Boca Raton, FL: CRC, 2012.
>
> On 09/04/2014 05:37 AM, Olexiy Movchan wrote:
>> Hello,
>>
>> I created the math issue https://issues.apache.org/jira/browse/MATH-1144.
>>
>> In version 2.0, LevenbergMarquardtOptimizer passed point to evaluator by reference. So our software could modify it on every step of algorithm.
>> In version 3.3, point is copied and then passed to evaluator, so it can't be updated by evaluator.
>>
>> We use LevenbergMarquardtOptimizer for 3d surface fitting (cylinders, cones, tori) by sampled points. And surface parameters should be renormalized on every step of algorithm. Please see this article:
>> http://nvlpubs.nist.gov/nistpubs/jres/103/6/j36sha.pdf
>>
>> Also please read the description of MATH-1144 jira issue.
>>
>> Can you modify optimizer or evaluator interface to allow in/out parameters there?
>>
>> Thanks,
>> Olexiy Movchan
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
RE: [math] Side effect of LevenbergMarquardtOptimizer
Posted by Olexiy Movchan <om...@apriori.com>.
Hi Evan,
Possibly a small jitter of initial guess would solve this issue. But it is hard to tell if this method guaranties convergence in all problematic cases.
Normalization approach already works and allows to converge in those cases.
Thanks,
Olexiy
-----Original Message-----
From: Evan Ward [mailto:evan.ward@nrl.navy.mil]
Sent: Thursday, September 04, 2014 7:52 PM
To: dev@commons.apache.org
Subject: Re: [math] Side effect of LevenbergMarquardtOptimizer
Hi Olexiy,
In my field we often encounter a similar problem when estimating attitude since a quaternion is only a valid rotation when it is normalized. We often escape this issue by estimating a "small"
adjustment to an apriori guess. (For the details see [1].) For this technique to work the cost function must be smooth and the apriori guess must be "close enough" to the true value. Both of these assumptions are also required to apply a non-linear least squares optimizer. Perhaps you can apply a similar technique to your problem. (It seems that your 'A'
parameter is orientation in 3D space.)
If there is a need for an extra steps, I would prefer to make those explicit rather than depending on side effects of cost function evaluation.
Best Regards,
Evan
[1] Crassidis, John L., and John L. Junkins. /Optimal Estimation of Dynamic Systems/. Boca Raton, FL: CRC, 2012.
On 09/04/2014 05:37 AM, Olexiy Movchan wrote:
> Hello,
>
> I created the math issue https://issues.apache.org/jira/browse/MATH-1144.
>
> In version 2.0, LevenbergMarquardtOptimizer passed point to evaluator by reference. So our software could modify it on every step of algorithm.
> In version 3.3, point is copied and then passed to evaluator, so it can't be updated by evaluator.
>
> We use LevenbergMarquardtOptimizer for 3d surface fitting (cylinders, cones, tori) by sampled points. And surface parameters should be renormalized on every step of algorithm. Please see this article:
> http://nvlpubs.nist.gov/nistpubs/jres/103/6/j36sha.pdf
>
> Also please read the description of MATH-1144 jira issue.
>
> Can you modify optimizer or evaluator interface to allow in/out parameters there?
>
> Thanks,
> Olexiy Movchan
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [math] Side effect of LevenbergMarquardtOptimizer
Posted by Evan Ward <ev...@nrl.navy.mil>.
Hi Olexiy,
In my field we often encounter a similar problem when estimating
attitude since a quaternion is only a valid rotation when it is
normalized. We often escape this issue by estimating a "small"
adjustment to an apriori guess. (For the details see [1].) For this
technique to work the cost function must be smooth and the apriori guess
must be "close enough" to the true value. Both of these assumptions are
also required to apply a non-linear least squares optimizer. Perhaps you
can apply a similar technique to your problem. (It seems that your 'A'
parameter is orientation in 3D space.)
If there is a need for an extra steps, I would prefer to make those
explicit rather than depending on side effects of cost function evaluation.
Best Regards,
Evan
[1] Crassidis, John L., and John L. Junkins. /Optimal Estimation of
Dynamic Systems/. Boca Raton, FL: CRC, 2012.
On 09/04/2014 05:37 AM, Olexiy Movchan wrote:
> Hello,
>
> I created the math issue https://issues.apache.org/jira/browse/MATH-1144.
>
> In version 2.0, LevenbergMarquardtOptimizer passed point to evaluator by reference. So our software could modify it on every step of algorithm.
> In version 3.3, point is copied and then passed to evaluator, so it can't be updated by evaluator.
>
> We use LevenbergMarquardtOptimizer for 3d surface fitting (cylinders, cones, tori) by sampled points. And surface parameters should be renormalized on every step of algorithm. Please see this article:
> http://nvlpubs.nist.gov/nistpubs/jres/103/6/j36sha.pdf
>
> Also please read the description of MATH-1144 jira issue.
>
> Can you modify optimizer or evaluator interface to allow in/out parameters there?
>
> Thanks,
> Olexiy Movchan
>
>