You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by go canal <go...@yahoo.com.INVALID> on 2015/10/03 05:47:22 UTC
matrix inversion in plan ?
HiI saw some distributed matrix functions included in Samsara now. Wondering if we have a plan to support matrix inversion ?BTW, am I correct that it is distributed memory based, not out-of-core ? thanks, canal
Re: matrix inversion in plan ?
Posted by go canal <go...@yahoo.com.INVALID>.
many thanks, here is the details:
given square matrix Z, . get Z inverse Zi. get degree of freedome of Zi : U. finally, calculate final which is: A + B * U * C
thanks, canal
On Monday, October 5, 2015 2:00 PM, Ted Dunning <te...@gmail.com> wrote:
This sort of thing would definitely compute the inverse.
And it is definitely to be avoided.
How about you give some specifics so I can say what should be done?
On Sun, Oct 4, 2015 at 7:31 PM, go canal <go...@yahoo.com.invalid> wrote:
Thank you all, the solver is something like this, am I correct:
Matrix m = ....
Matrix inverse = new QRDecomposition(m).solve(new DiagonalMatrix(1, m.rowSize()));
The problem I have is that the matrix is too big, I need distributed, or out-of-core solution.
thanks, canal
On Monday, October 5, 2015 6:25 AM, Peter Jaumann <pe...@gmail.com> wrote:
This should be done with a matrix solver indeed!!!
On Oct 4, 2015 11:53 AM, "Ted Dunning" <te...@gmail.com> wrote:
>
>
> It is almost certain that starting with an inversion is a serious error.
>
> Are you sure you don't want a matrix solver instead?
>
> Sent from my iPhone
>
> > On Oct 3, 2015, at 20:09, go canal <go...@yahoo.com.INVALID> wrote:
> >
> > oh, it is so unfortunate that the first step of my project requires the
inversion of a very large matrix. will have to revert back to scalapack or
MR based solutions I guess.
> > thanks, canal
> >
> >
> > On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
ted.dunning@gmail.com> wrote:
> >
> >
> > I doubt seriously that Samsara will support matrix inversion per se. The
> > problem is
> >
> > a) it densifies sparse matrices
> >
> > b) it is much more costly than solving a linear system
> >
> > Samsara is roughly memory based, but different back-ends will try to
spill
> > to disk if necessary. It is likely that the resulting degradation in
> > performance would be dramatic and thus unacceptable to most users.
> >
> >
> >
> >> On Fri, Oct 2, 2015 at 8:47 PM, go canal <go...@yahoo.com.invalid>
wrote:
> >>
> >> HiI saw some distributed matrix functions included in Samsara now.
> >> Wondering if we have a plan to support matrix inversion ?BTW, am I
correct
> >> that it is distributed memory based, not out-of-core ? thanks, canal
> >
> >
Re: matrix inversion in plan ?
Posted by Ted Dunning <te...@gmail.com>.
This sort of thing would definitely compute the inverse.
And it is definitely to be avoided.
How about you give some specifics so I can say what should be done?
On Sun, Oct 4, 2015 at 7:31 PM, go canal <go...@yahoo.com.invalid> wrote:
> Thank you all, the solver is something like this, am I correct:
> Matrix m = ....
> Matrix inverse = new QRDecomposition(m).solve(new DiagonalMatrix(1,
> m.rowSize()));
>
> The problem I have is that the matrix is too big, I need distributed, or
> out-of-core solution.
>
> thanks, canal
>
>
> On Monday, October 5, 2015 6:25 AM, Peter Jaumann <
> peter.jaumann1@gmail.com> wrote:
>
>
> This should be done with a matrix solver indeed!!!
>
>
>
> On Oct 4, 2015 11:53 AM, "Ted Dunning" <te...@gmail.com> wrote:
> >
> >
> > It is almost certain that starting with an inversion is a serious error.
> >
> > Are you sure you don't want a matrix solver instead?
> >
> > Sent from my iPhone
> >
> > > On Oct 3, 2015, at 20:09, go canal <go...@yahoo.com.INVALID> wrote:
> > >
> > > oh, it is so unfortunate that the first step of my project requires the
> inversion of a very large matrix. will have to revert back to scalapack or
> MR based solutions I guess.
> > > thanks, canal
> > >
> > >
> > > On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
> ted.dunning@gmail.com> wrote:
> > >
> > >
> > > I doubt seriously that Samsara will support matrix inversion per se.
> The
> > > problem is
> > >
> > > a) it densifies sparse matrices
> > >
> > > b) it is much more costly than solving a linear system
> > >
> > > Samsara is roughly memory based, but different back-ends will try to
> spill
> > > to disk if necessary. It is likely that the resulting degradation in
> > > performance would be dramatic and thus unacceptable to most users.
> > >
> > >
> > >
> > >> On Fri, Oct 2, 2015 at 8:47 PM, go canal <go...@yahoo.com.invalid>
> wrote:
> > >>
> > >> HiI saw some distributed matrix functions included in Samsara now.
> > >> Wondering if we have a plan to support matrix inversion ?BTW, am I
> correct
> > >> that it is distributed memory based, not out-of-core ? thanks, canal
> > >
> > >
>
>
>
>
Re: matrix inversion in plan ?
Posted by Andrew Musselman <an...@gmail.com>.
Totally an approximation; depends on why people are asking for the inverse
and whether it'd do.
On Thu, Oct 8, 2015 at 4:20 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> or pseudoinverse really, i guess
>
> On Thu, Oct 8, 2015 at 3:58 PM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
> > Mahout translation (approximation, since ssvd is reduced-rank, not the
> > true thing):
> >
> > val (drmU, drmV, s) = dssvd(drmA, k = 100)
> > val drmInvA = drmV %*% diagv(1 /=: s) %*% drmU.t
> >
> > Still, technically, it is a right inverse as in reality m is rarely the
> > same as n. Also, k must be k<= drmA.nrow min drmA.ncol
> >
> >
> > On Thu, Oct 8, 2015 at 2:52 PM, Andrew Musselman <
> > andrew.musselman@gmail.com> wrote:
> >
> >> Yeah, nice trick Ted; here's a how-to for the list:
> >> http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
> >>
> >> On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning <te...@gmail.com>
> >> wrote:
> >>
> >> > Yes. You can get the inverse from an SVD or emulate its effect.
> >> >
> >> > Can you share the actual mathematical specification for your problem?
> >> >
> >> > If you can't, then there is little we can do to help.
> >> >
> >> >
> >> >
> >> > On Wed, Oct 7, 2015 at 11:35 PM, go canal <go...@yahoo.com.invalid>
> >> > wrote:
> >> >
> >> > > Unfortunately I do not know much details of these. The steps of
> these
> >> > > calculation is passed to me from a research team. I am helping them
> >> with
> >> > > coding part only. I myself is not good at math :-(
> >> > > btw, I think Mahout supports out-of-core SVD, am I correct ? If
> so, I
> >> > can
> >> > > get inverse of matrix from SVD right ? thanks, canal
> >> > >
> >> > >
> >> > > On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> >> > > ted.dunning@gmail.com> wrote:
> >> > >
> >> > >
> >> > > That isn't enough detail.
> >> > >
> >> > > How do you mean to compute degrees of freedom? WHy do you need the
> >> > inverse
> >> > > to do this?
> >> > >
> >> > > Where did you get this algorithm?
> >> > >
> >> > > Is this even appropriate at large scale?
> >> > >
> >> > > Is this a stable computation?
> >> > >
> >> > >
> >> > >
> >> > > On Sun, Oct 4, 2015 at 11:18 PM, go canal <gocanal@yahoo.com.invalid
> >
> >> > > wrote:
> >> > >
> >> > > > I will be more than interested to extend to complex double, when
> the
> >> > > > solver is ready for double data type. thanks, canal
> >> > > >
> >> > > >
> >> > > > On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> >> > > > ted.dunning@gmail.com> wrote:
> >> > > >
> >> > > >
> >> > > > On Sun, Oct 4, 2015 at 10:32 PM, go canal
> >> <go...@yahoo.com.invalid>
> >> > > > wrote:
> >> > > >
> >> > > > > in fact i need to support both double and complex double for
> >> either
> >> > > > > distributed memory based or out-of-core.
> >> > > >
> >> > > >
> >> > > > Ahh...
> >> > > >
> >> > > > Well Mahout doesn't support complex anything. So this isn't going
> to
> >> > help
> >> > > > you.
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> >
> >>
> >
> >
>
Re: matrix inversion in plan ?
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
or pseudoinverse really, i guess
On Thu, Oct 8, 2015 at 3:58 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Mahout translation (approximation, since ssvd is reduced-rank, not the
> true thing):
>
> val (drmU, drmV, s) = dssvd(drmA, k = 100)
> val drmInvA = drmV %*% diagv(1 /=: s) %*% drmU.t
>
> Still, technically, it is a right inverse as in reality m is rarely the
> same as n. Also, k must be k<= drmA.nrow min drmA.ncol
>
>
> On Thu, Oct 8, 2015 at 2:52 PM, Andrew Musselman <
> andrew.musselman@gmail.com> wrote:
>
>> Yeah, nice trick Ted; here's a how-to for the list:
>> http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
>>
>> On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning <te...@gmail.com>
>> wrote:
>>
>> > Yes. You can get the inverse from an SVD or emulate its effect.
>> >
>> > Can you share the actual mathematical specification for your problem?
>> >
>> > If you can't, then there is little we can do to help.
>> >
>> >
>> >
>> > On Wed, Oct 7, 2015 at 11:35 PM, go canal <go...@yahoo.com.invalid>
>> > wrote:
>> >
>> > > Unfortunately I do not know much details of these. The steps of these
>> > > calculation is passed to me from a research team. I am helping them
>> with
>> > > coding part only. I myself is not good at math :-(
>> > > btw, I think Mahout supports out-of-core SVD, am I correct ? If so, I
>> > can
>> > > get inverse of matrix from SVD right ? thanks, canal
>> > >
>> > >
>> > > On Monday, October 5, 2015 2:25 PM, Ted Dunning <
>> > > ted.dunning@gmail.com> wrote:
>> > >
>> > >
>> > > That isn't enough detail.
>> > >
>> > > How do you mean to compute degrees of freedom? WHy do you need the
>> > inverse
>> > > to do this?
>> > >
>> > > Where did you get this algorithm?
>> > >
>> > > Is this even appropriate at large scale?
>> > >
>> > > Is this a stable computation?
>> > >
>> > >
>> > >
>> > > On Sun, Oct 4, 2015 at 11:18 PM, go canal <go...@yahoo.com.invalid>
>> > > wrote:
>> > >
>> > > > I will be more than interested to extend to complex double, when the
>> > > > solver is ready for double data type. thanks, canal
>> > > >
>> > > >
>> > > > On Monday, October 5, 2015 2:02 PM, Ted Dunning <
>> > > > ted.dunning@gmail.com> wrote:
>> > > >
>> > > >
>> > > > On Sun, Oct 4, 2015 at 10:32 PM, go canal
>> <go...@yahoo.com.invalid>
>> > > > wrote:
>> > > >
>> > > > > in fact i need to support both double and complex double for
>> either
>> > > > > distributed memory based or out-of-core.
>> > > >
>> > > >
>> > > > Ahh...
>> > > >
>> > > > Well Mahout doesn't support complex anything. So this isn't going to
>> > help
>> > > > you.
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> >
>>
>
>
Re: matrix inversion in plan ?
Posted by Andrew Musselman <an...@gmail.com>.
Go, if you'd like to put your research team in touch with the list we may
be able to help work through a good approach; let us know.
On Thu, Oct 8, 2015 at 3:58 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Mahout translation (approximation, since ssvd is reduced-rank, not the true
> thing):
>
> val (drmU, drmV, s) = dssvd(drmA, k = 100)
> val drmInvA = drmV %*% diagv(1 /=: s) %*% drmU.t
>
> Still, technically, it is a right inverse as in reality m is rarely the
> same as n. Also, k must be k<= drmA.nrow min drmA.ncol
>
>
> On Thu, Oct 8, 2015 at 2:52 PM, Andrew Musselman <
> andrew.musselman@gmail.com
> > wrote:
>
> > Yeah, nice trick Ted; here's a how-to for the list:
> > http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
> >
> > On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning <te...@gmail.com>
> wrote:
> >
> > > Yes. You can get the inverse from an SVD or emulate its effect.
> > >
> > > Can you share the actual mathematical specification for your problem?
> > >
> > > If you can't, then there is little we can do to help.
> > >
> > >
> > >
> > > On Wed, Oct 7, 2015 at 11:35 PM, go canal <go...@yahoo.com.invalid>
> > > wrote:
> > >
> > > > Unfortunately I do not know much details of these. The steps of these
> > > > calculation is passed to me from a research team. I am helping them
> > with
> > > > coding part only. I myself is not good at math :-(
> > > > btw, I think Mahout supports out-of-core SVD, am I correct ? If so,
> I
> > > can
> > > > get inverse of matrix from SVD right ? thanks, canal
> > > >
> > > >
> > > > On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> > > > ted.dunning@gmail.com> wrote:
> > > >
> > > >
> > > > That isn't enough detail.
> > > >
> > > > How do you mean to compute degrees of freedom? WHy do you need the
> > > inverse
> > > > to do this?
> > > >
> > > > Where did you get this algorithm?
> > > >
> > > > Is this even appropriate at large scale?
> > > >
> > > > Is this a stable computation?
> > > >
> > > >
> > > >
> > > > On Sun, Oct 4, 2015 at 11:18 PM, go canal <gocanal@yahoo.com.invalid
> >
> > > > wrote:
> > > >
> > > > > I will be more than interested to extend to complex double, when
> the
> > > > > solver is ready for double data type. thanks, canal
> > > > >
> > > > >
> > > > > On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> > > > > ted.dunning@gmail.com> wrote:
> > > > >
> > > > >
> > > > > On Sun, Oct 4, 2015 at 10:32 PM, go canal
> <gocanal@yahoo.com.invalid
> > >
> > > > > wrote:
> > > > >
> > > > > > in fact i need to support both double and complex double for
> either
> > > > > > distributed memory based or out-of-core.
> > > > >
> > > > >
> > > > > Ahh...
> > > > >
> > > > > Well Mahout doesn't support complex anything. So this isn't going
> to
> > > help
> > > > > you.
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
>
Re: matrix inversion in plan ?
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Mahout translation (approximation, since ssvd is reduced-rank, not the true
thing):
val (drmU, drmV, s) = dssvd(drmA, k = 100)
val drmInvA = drmV %*% diagv(1 /=: s) %*% drmU.t
Still, technically, it is a right inverse as in reality m is rarely the
same as n. Also, k must be k<= drmA.nrow min drmA.ncol
On Thu, Oct 8, 2015 at 2:52 PM, Andrew Musselman <andrew.musselman@gmail.com
> wrote:
> Yeah, nice trick Ted; here's a how-to for the list:
> http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
>
> On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning <te...@gmail.com> wrote:
>
> > Yes. You can get the inverse from an SVD or emulate its effect.
> >
> > Can you share the actual mathematical specification for your problem?
> >
> > If you can't, then there is little we can do to help.
> >
> >
> >
> > On Wed, Oct 7, 2015 at 11:35 PM, go canal <go...@yahoo.com.invalid>
> > wrote:
> >
> > > Unfortunately I do not know much details of these. The steps of these
> > > calculation is passed to me from a research team. I am helping them
> with
> > > coding part only. I myself is not good at math :-(
> > > btw, I think Mahout supports out-of-core SVD, am I correct ? If so, I
> > can
> > > get inverse of matrix from SVD right ? thanks, canal
> > >
> > >
> > > On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> > > ted.dunning@gmail.com> wrote:
> > >
> > >
> > > That isn't enough detail.
> > >
> > > How do you mean to compute degrees of freedom? WHy do you need the
> > inverse
> > > to do this?
> > >
> > > Where did you get this algorithm?
> > >
> > > Is this even appropriate at large scale?
> > >
> > > Is this a stable computation?
> > >
> > >
> > >
> > > On Sun, Oct 4, 2015 at 11:18 PM, go canal <go...@yahoo.com.invalid>
> > > wrote:
> > >
> > > > I will be more than interested to extend to complex double, when the
> > > > solver is ready for double data type. thanks, canal
> > > >
> > > >
> > > > On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> > > > ted.dunning@gmail.com> wrote:
> > > >
> > > >
> > > > On Sun, Oct 4, 2015 at 10:32 PM, go canal <gocanal@yahoo.com.invalid
> >
> > > > wrote:
> > > >
> > > > > in fact i need to support both double and complex double for either
> > > > > distributed memory based or out-of-core.
> > > >
> > > >
> > > > Ahh...
> > > >
> > > > Well Mahout doesn't support complex anything. So this isn't going to
> > help
> > > > you.
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
>
Re: matrix inversion in plan ?
Posted by Andrew Musselman <an...@gmail.com>.
Yeah, nice trick Ted; here's a how-to for the list:
http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning <te...@gmail.com> wrote:
> Yes. You can get the inverse from an SVD or emulate its effect.
>
> Can you share the actual mathematical specification for your problem?
>
> If you can't, then there is little we can do to help.
>
>
>
> On Wed, Oct 7, 2015 at 11:35 PM, go canal <go...@yahoo.com.invalid>
> wrote:
>
> > Unfortunately I do not know much details of these. The steps of these
> > calculation is passed to me from a research team. I am helping them with
> > coding part only. I myself is not good at math :-(
> > btw, I think Mahout supports out-of-core SVD, am I correct ? If so, I
> can
> > get inverse of matrix from SVD right ? thanks, canal
> >
> >
> > On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> > ted.dunning@gmail.com> wrote:
> >
> >
> > That isn't enough detail.
> >
> > How do you mean to compute degrees of freedom? WHy do you need the
> inverse
> > to do this?
> >
> > Where did you get this algorithm?
> >
> > Is this even appropriate at large scale?
> >
> > Is this a stable computation?
> >
> >
> >
> > On Sun, Oct 4, 2015 at 11:18 PM, go canal <go...@yahoo.com.invalid>
> > wrote:
> >
> > > I will be more than interested to extend to complex double, when the
> > > solver is ready for double data type. thanks, canal
> > >
> > >
> > > On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> > > ted.dunning@gmail.com> wrote:
> > >
> > >
> > > On Sun, Oct 4, 2015 at 10:32 PM, go canal <go...@yahoo.com.invalid>
> > > wrote:
> > >
> > > > in fact i need to support both double and complex double for either
> > > > distributed memory based or out-of-core.
> > >
> > >
> > > Ahh...
> > >
> > > Well Mahout doesn't support complex anything. So this isn't going to
> help
> > > you.
> > >
> > >
> > >
> >
> >
> >
>
Re: matrix inversion in plan ?
Posted by Ted Dunning <te...@gmail.com>.
Yes. You can get the inverse from an SVD or emulate its effect.
Can you share the actual mathematical specification for your problem?
If you can't, then there is little we can do to help.
On Wed, Oct 7, 2015 at 11:35 PM, go canal <go...@yahoo.com.invalid> wrote:
> Unfortunately I do not know much details of these. The steps of these
> calculation is passed to me from a research team. I am helping them with
> coding part only. I myself is not good at math :-(
> btw, I think Mahout supports out-of-core SVD, am I correct ? If so, I can
> get inverse of matrix from SVD right ? thanks, canal
>
>
> On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> ted.dunning@gmail.com> wrote:
>
>
> That isn't enough detail.
>
> How do you mean to compute degrees of freedom? WHy do you need the inverse
> to do this?
>
> Where did you get this algorithm?
>
> Is this even appropriate at large scale?
>
> Is this a stable computation?
>
>
>
> On Sun, Oct 4, 2015 at 11:18 PM, go canal <go...@yahoo.com.invalid>
> wrote:
>
> > I will be more than interested to extend to complex double, when the
> > solver is ready for double data type. thanks, canal
> >
> >
> > On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> > ted.dunning@gmail.com> wrote:
> >
> >
> > On Sun, Oct 4, 2015 at 10:32 PM, go canal <go...@yahoo.com.invalid>
> > wrote:
> >
> > > in fact i need to support both double and complex double for either
> > > distributed memory based or out-of-core.
> >
> >
> > Ahh...
> >
> > Well Mahout doesn't support complex anything. So this isn't going to help
> > you.
> >
> >
> >
>
>
>
Re: matrix inversion in plan ?
Posted by go canal <go...@yahoo.com.INVALID>.
Unfortunately I do not know much details of these. The steps of these calculation is passed to me from a research team. I am helping them with coding part only. I myself is not good at math :-(
btw, I think Mahout supports out-of-core SVD, am I correct ? If so, I can get inverse of matrix from SVD right ? thanks, canal
On Monday, October 5, 2015 2:25 PM, Ted Dunning <te...@gmail.com> wrote:
That isn't enough detail.
How do you mean to compute degrees of freedom? WHy do you need the inverse
to do this?
Where did you get this algorithm?
Is this even appropriate at large scale?
Is this a stable computation?
On Sun, Oct 4, 2015 at 11:18 PM, go canal <go...@yahoo.com.invalid> wrote:
> I will be more than interested to extend to complex double, when the
> solver is ready for double data type. thanks, canal
>
>
> On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> ted.dunning@gmail.com> wrote:
>
>
> On Sun, Oct 4, 2015 at 10:32 PM, go canal <go...@yahoo.com.invalid>
> wrote:
>
> > in fact i need to support both double and complex double for either
> > distributed memory based or out-of-core.
>
>
> Ahh...
>
> Well Mahout doesn't support complex anything. So this isn't going to help
> you.
>
>
>
Re: matrix inversion in plan ?
Posted by Ted Dunning <te...@gmail.com>.
That isn't enough detail.
How do you mean to compute degrees of freedom? WHy do you need the inverse
to do this?
Where did you get this algorithm?
Is this even appropriate at large scale?
Is this a stable computation?
On Sun, Oct 4, 2015 at 11:18 PM, go canal <go...@yahoo.com.invalid> wrote:
> I will be more than interested to extend to complex double, when the
> solver is ready for double data type. thanks, canal
>
>
> On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> ted.dunning@gmail.com> wrote:
>
>
> On Sun, Oct 4, 2015 at 10:32 PM, go canal <go...@yahoo.com.invalid>
> wrote:
>
> > in fact i need to support both double and complex double for either
> > distributed memory based or out-of-core.
>
>
> Ahh...
>
> Well Mahout doesn't support complex anything. So this isn't going to help
> you.
>
>
>
Re: matrix inversion in plan ?
Posted by go canal <go...@yahoo.com.INVALID>.
I will be more than interested to extend to complex double, when the solver is ready for double data type. thanks, canal
On Monday, October 5, 2015 2:02 PM, Ted Dunning <te...@gmail.com> wrote:
On Sun, Oct 4, 2015 at 10:32 PM, go canal <go...@yahoo.com.invalid> wrote:
> in fact i need to support both double and complex double for either
> distributed memory based or out-of-core.
Ahh...
Well Mahout doesn't support complex anything. So this isn't going to help
you.
Re: matrix inversion in plan ?
Posted by Ted Dunning <te...@gmail.com>.
On Sun, Oct 4, 2015 at 10:32 PM, go canal <go...@yahoo.com.invalid> wrote:
> in fact i need to support both double and complex double for either
> distributed memory based or out-of-core.
Ahh...
Well Mahout doesn't support complex anything. So this isn't going to help
you.
Re: matrix inversion in plan ?
Posted by go canal <go...@yahoo.com.INVALID>.
both sparse and dense.
there are a few steps for the whole calculation:. inverse of M. get degree of freedom. multiplication. addition
in fact i need to support both double and complex double for either distributed memory based or out-of-core.
I found one MR based solution for large matrix inversion https://github.com/JingenXiang/MatrixInversion and I have modified the code to support complex double. Execution seems ok but i do not understand the final output format. It seems that the columns are swapped. thanks, canal
On Monday, October 5, 2015 12:26 PM, Allen McIntosh <am...@appcomsci.com> wrote:
1) Is m sparse?
2) Once you have computed "inverse", what are you going to do with it?
On 10/04/2015 10:31 PM, go canal wrote:
> Thank you all, the solver is something like this, am I correct:
> Matrix m = ....
> Matrix inverse = new QRDecomposition(m).solve(new DiagonalMatrix(1, m.rowSize()));
>
> The problem I have is that the matrix is too big, I need distributed, or out-of-core solution.
>
> thanks, canal
>
>
> On Monday, October 5, 2015 6:25 AM, Peter Jaumann <pe...@gmail.com> wrote:
>
>
> This should be done with a matrix solver indeed!!!
>
>
>
> On Oct 4, 2015 11:53 AM, "Ted Dunning" <te...@gmail.com> wrote:
>>
>>
>> It is almost certain that starting with an inversion is a serious error.
>>
>> Are you sure you don't want a matrix solver instead?
>>
>> Sent from my iPhone
>>
>>> On Oct 3, 2015, at 20:09, go canal <go...@yahoo.com.INVALID> wrote:
>>>
>>> oh, it is so unfortunate that the first step of my project requires the
> inversion of a very large matrix. will have to revert back to scalapack or
> MR based solutions I guess.
>>> thanks, canal
>>>
>>>
>>> On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
> ted.dunning@gmail.com> wrote:
>>>
>>>
>>> I doubt seriously that Samsara will support matrix inversion per se. The
>>> problem is
>>>
>>> a) it densifies sparse matrices
>>>
>>> b) it is much more costly than solving a linear system
>>>
>>> Samsara is roughly memory based, but different back-ends will try to
> spill
>>> to disk if necessary. It is likely that the resulting degradation in
>>> performance would be dramatic and thus unacceptable to most users.
>>>
>>>
>>>
>>>> On Fri, Oct 2, 2015 at 8:47 PM, go canal <go...@yahoo.com.invalid>
> wrote:
>>>>
>>>> HiI saw some distributed matrix functions included in Samsara now.
>>>> Wondering if we have a plan to support matrix inversion ?BTW, am I
> correct
>>>> that it is distributed memory based, not out-of-core ? thanks, canal
>>>
>>>
>
>
>
>
Re: matrix inversion in plan ?
Posted by Allen McIntosh <am...@appcomsci.com>.
1) Is m sparse?
2) Once you have computed "inverse", what are you going to do with it?
On 10/04/2015 10:31 PM, go canal wrote:
> Thank you all, the solver is something like this, am I correct:
> Matrix m = ....
> Matrix inverse = new QRDecomposition(m).solve(new DiagonalMatrix(1, m.rowSize()));
>
> The problem I have is that the matrix is too big, I need distributed, or out-of-core solution.
>
> thanks, canal
>
>
> On Monday, October 5, 2015 6:25 AM, Peter Jaumann <pe...@gmail.com> wrote:
>
>
> This should be done with a matrix solver indeed!!!
>
>
>
> On Oct 4, 2015 11:53 AM, "Ted Dunning" <te...@gmail.com> wrote:
>>
>>
>> It is almost certain that starting with an inversion is a serious error.
>>
>> Are you sure you don't want a matrix solver instead?
>>
>> Sent from my iPhone
>>
>>> On Oct 3, 2015, at 20:09, go canal <go...@yahoo.com.INVALID> wrote:
>>>
>>> oh, it is so unfortunate that the first step of my project requires the
> inversion of a very large matrix. will have to revert back to scalapack or
> MR based solutions I guess.
>>> thanks, canal
>>>
>>>
>>> On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
> ted.dunning@gmail.com> wrote:
>>>
>>>
>>> I doubt seriously that Samsara will support matrix inversion per se. The
>>> problem is
>>>
>>> a) it densifies sparse matrices
>>>
>>> b) it is much more costly than solving a linear system
>>>
>>> Samsara is roughly memory based, but different back-ends will try to
> spill
>>> to disk if necessary. It is likely that the resulting degradation in
>>> performance would be dramatic and thus unacceptable to most users.
>>>
>>>
>>>
>>>> On Fri, Oct 2, 2015 at 8:47 PM, go canal <go...@yahoo.com.invalid>
> wrote:
>>>>
>>>> HiI saw some distributed matrix functions included in Samsara now.
>>>> Wondering if we have a plan to support matrix inversion ?BTW, am I
> correct
>>>> that it is distributed memory based, not out-of-core ? thanks, canal
>>>
>>>
>
>
>
>
Re: matrix inversion in plan ?
Posted by go canal <go...@yahoo.com.INVALID>.
Thank you all, the solver is something like this, am I correct:
Matrix m = ....
Matrix inverse = new QRDecomposition(m).solve(new DiagonalMatrix(1, m.rowSize()));
The problem I have is that the matrix is too big, I need distributed, or out-of-core solution.
thanks, canal
On Monday, October 5, 2015 6:25 AM, Peter Jaumann <pe...@gmail.com> wrote:
This should be done with a matrix solver indeed!!!
On Oct 4, 2015 11:53 AM, "Ted Dunning" <te...@gmail.com> wrote:
>
>
> It is almost certain that starting with an inversion is a serious error.
>
> Are you sure you don't want a matrix solver instead?
>
> Sent from my iPhone
>
> > On Oct 3, 2015, at 20:09, go canal <go...@yahoo.com.INVALID> wrote:
> >
> > oh, it is so unfortunate that the first step of my project requires the
inversion of a very large matrix. will have to revert back to scalapack or
MR based solutions I guess.
> > thanks, canal
> >
> >
> > On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
ted.dunning@gmail.com> wrote:
> >
> >
> > I doubt seriously that Samsara will support matrix inversion per se. The
> > problem is
> >
> > a) it densifies sparse matrices
> >
> > b) it is much more costly than solving a linear system
> >
> > Samsara is roughly memory based, but different back-ends will try to
spill
> > to disk if necessary. It is likely that the resulting degradation in
> > performance would be dramatic and thus unacceptable to most users.
> >
> >
> >
> >> On Fri, Oct 2, 2015 at 8:47 PM, go canal <go...@yahoo.com.invalid>
wrote:
> >>
> >> HiI saw some distributed matrix functions included in Samsara now.
> >> Wondering if we have a plan to support matrix inversion ?BTW, am I
correct
> >> that it is distributed memory based, not out-of-core ? thanks, canal
> >
> >
Re: matrix inversion in plan ?
Posted by Peter Jaumann <pe...@gmail.com>.
This should be done with a matrix solver indeed!!!
On Oct 4, 2015 11:53 AM, "Ted Dunning" <te...@gmail.com> wrote:
>
>
> It is almost certain that starting with an inversion is a serious error.
>
> Are you sure you don't want a matrix solver instead?
>
> Sent from my iPhone
>
> > On Oct 3, 2015, at 20:09, go canal <go...@yahoo.com.INVALID> wrote:
> >
> > oh, it is so unfortunate that the first step of my project requires the
inversion of a very large matrix. will have to revert back to scalapack or
MR based solutions I guess.
> > thanks, canal
> >
> >
> > On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
ted.dunning@gmail.com> wrote:
> >
> >
> > I doubt seriously that Samsara will support matrix inversion per se. The
> > problem is
> >
> > a) it densifies sparse matrices
> >
> > b) it is much more costly than solving a linear system
> >
> > Samsara is roughly memory based, but different back-ends will try to
spill
> > to disk if necessary. It is likely that the resulting degradation in
> > performance would be dramatic and thus unacceptable to most users.
> >
> >
> >
> >> On Fri, Oct 2, 2015 at 8:47 PM, go canal <go...@yahoo.com.invalid>
wrote:
> >>
> >> HiI saw some distributed matrix functions included in Samsara now.
> >> Wondering if we have a plan to support matrix inversion ?BTW, am I
correct
> >> that it is distributed memory based, not out-of-core ? thanks, canal
> >
> >
Re: matrix inversion in plan ?
Posted by Ted Dunning <te...@gmail.com>.
It is almost certain that starting with an inversion is a serious error.
Are you sure you don't want a matrix solver instead?
Sent from my iPhone
> On Oct 3, 2015, at 20:09, go canal <go...@yahoo.com.INVALID> wrote:
>
> oh, it is so unfortunate that the first step of my project requires the inversion of a very large matrix. will have to revert back to scalapack or MR based solutions I guess.
> thanks, canal
>
>
> On Saturday, October 3, 2015 11:31 PM, Ted Dunning <te...@gmail.com> wrote:
>
>
> I doubt seriously that Samsara will support matrix inversion per se. The
> problem is
>
> a) it densifies sparse matrices
>
> b) it is much more costly than solving a linear system
>
> Samsara is roughly memory based, but different back-ends will try to spill
> to disk if necessary. It is likely that the resulting degradation in
> performance would be dramatic and thus unacceptable to most users.
>
>
>
>> On Fri, Oct 2, 2015 at 8:47 PM, go canal <go...@yahoo.com.invalid> wrote:
>>
>> HiI saw some distributed matrix functions included in Samsara now.
>> Wondering if we have a plan to support matrix inversion ?BTW, am I correct
>> that it is distributed memory based, not out-of-core ? thanks, canal
>
>
Re: matrix inversion in plan ?
Posted by Andrew Musselman <an...@gmail.com>.
If there's a need for inversion that's good info; would love to know the
purpose to get a sense of how people want to use the product.
On Saturday, October 3, 2015, Allen McIntosh <am...@appcomsci.com>
wrote:
> Can you explain why you feel you must invert a very large matrix. This
> can be a Bad Idea.
>
> On 10/03/2015 08:09 PM, go canal wrote:
> > oh, it is so unfortunate that the first step of my project requires
> > the inversion of a very large matrix. will have to revert back to
> > scalapack or MR based solutions I guess. thanks, canal
> >
> >
> > On Saturday, October 3, 2015 11:31 PM, Ted Dunning
> > <ted.dunning@gmail.com <javascript:;>> wrote:
> >
> >
> > I doubt seriously that Samsara will support matrix inversion per se.
> > The problem is
> >
> > a) it densifies sparse matrices
> >
> > b) it is much more costly than solving a linear system
> >
> > Samsara is roughly memory based, but different back-ends will try to
> > spill to disk if necessary. It is likely that the resulting
> > degradation in performance would be dramatic and thus unacceptable to
> > most users.
> >
> >
> >
> > On Fri, Oct 2, 2015 at 8:47 PM, go canal <go...@yahoo.com.invalid>
> > wrote:
> >
> >> HiI saw some distributed matrix functions included in Samsara now.
> >> Wondering if we have a plan to support matrix inversion ?BTW, am I
> >> correct that it is distributed memory based, not out-of-core ?
> >> thanks, canal
> >
> >
> >
> >
>
>
Re: matrix inversion in plan ?
Posted by Allen McIntosh <am...@appcomsci.com>.
Can you explain why you feel you must invert a very large matrix. This
can be a Bad Idea.
On 10/03/2015 08:09 PM, go canal wrote:
> oh, it is so unfortunate that the first step of my project requires
> the inversion of a very large matrix. will have to revert back to
> scalapack or MR based solutions I guess. thanks, canal
>
>
> On Saturday, October 3, 2015 11:31 PM, Ted Dunning
> <te...@gmail.com> wrote:
>
>
> I doubt seriously that Samsara will support matrix inversion per se.
> The problem is
>
> a) it densifies sparse matrices
>
> b) it is much more costly than solving a linear system
>
> Samsara is roughly memory based, but different back-ends will try to
> spill to disk if necessary. It is likely that the resulting
> degradation in performance would be dramatic and thus unacceptable to
> most users.
>
>
>
> On Fri, Oct 2, 2015 at 8:47 PM, go canal <go...@yahoo.com.invalid>
> wrote:
>
>> HiI saw some distributed matrix functions included in Samsara now.
>> Wondering if we have a plan to support matrix inversion ?BTW, am I
>> correct that it is distributed memory based, not out-of-core ?
>> thanks, canal
>
>
>
>
Re: matrix inversion in plan ?
Posted by go canal <go...@yahoo.com.INVALID>.
oh, it is so unfortunate that the first step of my project requires the inversion of a very large matrix. will have to revert back to scalapack or MR based solutions I guess.
thanks, canal
On Saturday, October 3, 2015 11:31 PM, Ted Dunning <te...@gmail.com> wrote:
I doubt seriously that Samsara will support matrix inversion per se. The
problem is
a) it densifies sparse matrices
b) it is much more costly than solving a linear system
Samsara is roughly memory based, but different back-ends will try to spill
to disk if necessary. It is likely that the resulting degradation in
performance would be dramatic and thus unacceptable to most users.
On Fri, Oct 2, 2015 at 8:47 PM, go canal <go...@yahoo.com.invalid> wrote:
> HiI saw some distributed matrix functions included in Samsara now.
> Wondering if we have a plan to support matrix inversion ?BTW, am I correct
> that it is distributed memory based, not out-of-core ? thanks, canal
Re: matrix inversion in plan ?
Posted by Ted Dunning <te...@gmail.com>.
I doubt seriously that Samsara will support matrix inversion per se. The
problem is
a) it densifies sparse matrices
b) it is much more costly than solving a linear system
Samsara is roughly memory based, but different back-ends will try to spill
to disk if necessary. It is likely that the resulting degradation in
performance would be dramatic and thus unacceptable to most users.
On Fri, Oct 2, 2015 at 8:47 PM, go canal <go...@yahoo.com.invalid> wrote:
> HiI saw some distributed matrix functions included in Samsara now.
> Wondering if we have a plan to support matrix inversion ?BTW, am I correct
> that it is distributed memory based, not out-of-core ? thanks, canal