You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Mikkel Meyer Andersen <mi...@mikl.dk> on 2011/02/21 13:52:49 UTC
[MATH] Matrix hierarchy reorganization
Dear community,
I've started on implementing Kalman filter and have once again run
into a structural challenge in regards to matrix hierarchy (as with
https://issues.apache.org/jira/browse/MATH-435 and
https://issues.apache.org/jira/browse/MATH-437 ).
We have two base types of matrices: RealMatrix and FieldMatrix, both
extending AnyMatrix. Both RealMatrix and FieldMatrix have a lot of the
same functionality, e.g. add, subtract, scalarAdd, scalarMultiply,
multiply, preMultiply etc.
I think it would be beneficial to try to move these standard matrix
methods to AnyMatrix, or at least introduce a new intermediate
AnyAlgebraicMatrix (or similar). It would be natural in several
aspects to just require an AnyAlgebraicMatrix, e.g. with Kalman filter
or repeated squaring, instead of forcing the user to use either
RealMatrix or FieldMatrix (or implement almost similar code for both).
What do you think about such a reorganization of the matrix hierarchy?
Cheers, Mikkel.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [MATH] Matrix hierarchy reorganization
Posted by Ted Dunning <te...@gmail.com>.
I really would love to have mahout-math have more in common with
commons math but there are likely some nasty conflicts in goals. What
mahout has needed up to now is fast evolution which has not been
possible with commons math because of the compatibility constraints
and because it is difficult to sub-class CM matrices effectively.
While our fast change rate has moderated, our need for efficient
sub-classing is still quite exigent.
The other major needs include the ability to serialize matrices
efficiently in the hadoop style and to carry around labels on rows and
sometimes columns. The serialization nearly requires a dependency on
hadoop which is probably bad for CM. We also use the functional style
of colt quite a lot which is pretty different from jama style.
Given these constraints I really don't know how much commonality is
feasible, but it would be great to pool our efforts if we can. Luc,
Gilles, Mikkel, have you looked at the Mahout matrices? If you do I
am sure that you will see that we have left large gaps in what we
offer. We are, in fact, decreasing what we support over time since
that is often the most effective way to move forward in our quest to
improve our test coverage.
Let's do talk about where we can go from here.
On Monday, February 21, 2011, Gilles Sadowski
<gi...@harfang.homelinux.org> wrote:
>> [...]
>>
>> The things we will for sure have some difficult discussions are:
>> - dependencies
>> (I guess Phil and myself don't like dependencies, whereas Ted and Gilles
>> are happy with them)
>
> My position is probably bestth summarized as follows:
> 1. I'm against dependencies for everything concerning the "core business" of
> Commons Math: A light-weight library of mathematical algorithms.
> 2. I'm for dependencies when avoiding them means re-inventing the wheel
> (e.g. localization, logging).
> 3. Consistently with point 1, I'm against locking CM in self-inflicted
> dependencies, a.o. publicizing an API that does not belong to the "core
> business".
>
>> What I would like to see:
>> - more exchanges with the Mahout community
>
> Are you ready to depend on their library?
>
>> [...]
>
> Best regards,
> Gilles
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [MATH] Matrix hierarchy reorganization
Posted by Gilles Sadowski <gi...@harfang.homelinux.org>.
> [...]
>
> The things we will for sure have some difficult discussions are:
> - dependencies
> (I guess Phil and myself don't like dependencies, whereas Ted and Gilles
> are happy with them)
My position is probably best summarized as follows:
1. I'm against dependencies for everything concerning the "core business" of
Commons Math: A light-weight library of mathematical algorithms.
2. I'm for dependencies when avoiding them means re-inventing the wheel
(e.g. localization, logging).
3. Consistently with point 1, I'm against locking CM in self-inflicted
dependencies, a.o. publicizing an API that does not belong to the "core
business".
> What I would like to see:
> - more exchanges with the Mahout community
Are you ready to depend on their library?
> [...]
Best regards,
Gilles
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [MATH] Matrix hierarchy reorganization
Posted by Mikkel Meyer Andersen <mi...@mikl.dk>.
2011/2/21 <lu...@free.fr>:
> Hi Mikkel,
>
> ----- "Mikkel Meyer Andersen" <mi...@mikl.dk> a écrit :
>
>> Dear community,
>>
>> I've started on implementing Kalman filter and have once again run
>> into a structural challenge in regards to matrix hierarchy (as with
>> https://issues.apache.org/jira/browse/MATH-435 and
>> https://issues.apache.org/jira/browse/MATH-437 ).
>>
>> We have two base types of matrices: RealMatrix and FieldMatrix, both
>> extending AnyMatrix. Both RealMatrix and FieldMatrix have a lot of
>> the
>> same functionality, e.g. add, subtract, scalarAdd, scalarMultiply,
>> multiply, preMultiply etc.
>>
>> I think it would be beneficial to try to move these standard matrix
>> methods to AnyMatrix, or at least introduce a new intermediate
>> AnyAlgebraicMatrix (or similar). It would be natural in several
>> aspects to just require an AnyAlgebraicMatrix, e.g. with Kalman
>> filter
>> or repeated squaring, instead of forcing the user to use either
>> RealMatrix or FieldMatrix (or implement almost similar code for
>> both).
>>
>> What do you think about such a reorganization of the matrix
>> hierarchy?
>
> I think we really need to reorganize our linear algebra, but am not sure about
> this specific change, as I fear it would deter performances (which already
> needs improvements).
>
> At ApacheCon Atlanta, we had a short discussion with Isabel Drost form the Mahout
> project. We would really much have liked to have Ted Tunning around too
> but he was not there. We said to Isabel that once we have 2.2 out and
> will be reflecting about 3.0, we would like to have some requirements from
> the Mahout project. We were not able up to now to satisfy their needs because
> of backward compatibility. However, now is the time we can change anything
> we want.
>
> The current problems I see with our linear algebra package are the following ones:
> - clumsy hierarchy
> - inefficient for large matrices (say above 4000 rows and columns)
> - insufficient support for sparse matrices (most algorithms simply don't use sparsity)
> - SVD is still not stable, after 3 complete rewrites
> - missing features (eigen decomposition for non-symetric matrices)
>
> The cool features we have are:
> - support for field matrices (this is really useful with Complex elements,
> with Fraction elements when we need exact computation of polynomial coefficients for
> example, and with Dfp elements for high accuracy)
> - reasonable API for decomposition, mainly Jama compatible but slightly better IMHO
>
> The things we can change at 3.0 are:
> - API
> - implementation
>
> The things we will for sure have some difficult discussions are:
> - dependencies
> (I guess Phil and myself don't like dependencies, whereas Ted and Gilles
> are happy with them)
>
> What I would like to see:
> - more exchanges with the Mahout community
>
> I'm not sure if we should have this discussion here, or on the Mahout list or
> on both. Ted, what to you think about this, are their a sufficient number of
> Mahout people here or should we cross-post everything ?
>
> Thanks for bringing up this really important topic Mikkel!
> Luc
>
>>
>> Cheers, Mikkel.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
>> For additional commands, e-mail: dev-help@commons.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
Hi Luc,
It wasn't my meaning to open such huge a discussion at the moment :-).
Sorry for that. I think we should postpone this discussion until 2.2
is out. Meanwhile, I'll be thinking about possible ways to go with the
hierarchy.
Cheers, Mikkel.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [MATH] Matrix hierarchy reorganization
Posted by lu...@free.fr.
Hi Mikkel,
----- "Mikkel Meyer Andersen" <mi...@mikl.dk> a écrit :
> Dear community,
>
> I've started on implementing Kalman filter and have once again run
> into a structural challenge in regards to matrix hierarchy (as with
> https://issues.apache.org/jira/browse/MATH-435 and
> https://issues.apache.org/jira/browse/MATH-437 ).
>
> We have two base types of matrices: RealMatrix and FieldMatrix, both
> extending AnyMatrix. Both RealMatrix and FieldMatrix have a lot of
> the
> same functionality, e.g. add, subtract, scalarAdd, scalarMultiply,
> multiply, preMultiply etc.
>
> I think it would be beneficial to try to move these standard matrix
> methods to AnyMatrix, or at least introduce a new intermediate
> AnyAlgebraicMatrix (or similar). It would be natural in several
> aspects to just require an AnyAlgebraicMatrix, e.g. with Kalman
> filter
> or repeated squaring, instead of forcing the user to use either
> RealMatrix or FieldMatrix (or implement almost similar code for
> both).
>
> What do you think about such a reorganization of the matrix
> hierarchy?
I think we really need to reorganize our linear algebra, but am not sure about
this specific change, as I fear it would deter performances (which already
needs improvements).
At ApacheCon Atlanta, we had a short discussion with Isabel Drost form the Mahout
project. We would really much have liked to have Ted Tunning around too
but he was not there. We said to Isabel that once we have 2.2 out and
will be reflecting about 3.0, we would like to have some requirements from
the Mahout project. We were not able up to now to satisfy their needs because
of backward compatibility. However, now is the time we can change anything
we want.
The current problems I see with our linear algebra package are the following ones:
- clumsy hierarchy
- inefficient for large matrices (say above 4000 rows and columns)
- insufficient support for sparse matrices (most algorithms simply don't use sparsity)
- SVD is still not stable, after 3 complete rewrites
- missing features (eigen decomposition for non-symetric matrices)
The cool features we have are:
- support for field matrices (this is really useful with Complex elements,
with Fraction elements when we need exact computation of polynomial coefficients for
example, and with Dfp elements for high accuracy)
- reasonable API for decomposition, mainly Jama compatible but slightly better IMHO
The things we can change at 3.0 are:
- API
- implementation
The things we will for sure have some difficult discussions are:
- dependencies
(I guess Phil and myself don't like dependencies, whereas Ted and Gilles
are happy with them)
What I would like to see:
- more exchanges with the Mahout community
I'm not sure if we should have this discussion here, or on the Mahout list or
on both. Ted, what to you think about this, are their a sufficient number of
Mahout people here or should we cross-post everything ?
Thanks for bringing up this really important topic Mikkel!
Luc
>
> Cheers, Mikkel.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org