You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Shannon Quinn <sq...@gatech.edu> on 2010/06/17 22:35:30 UTC

Multiple calls to DistributedRowMatrix.times()

Hi all,

I'm attempting to carry out the multiplication of two 
DistributedRowMatrix objects. It's similar to a SVD multiplication in 
that I have a matrix A and a matrix B, and the operation looks like BAB, or:

B.times(A.times(B))

When I run it, I get a FileAlreadyExistsException from Hadoop, since for 
both A and B the result of outputTmpBasePath.getParent() is the same (A 
is read from input, and B is calculated from A). Which would be a better 
approach:

-Modify DistributedRowMatrix.times() so "productWith" is instead 
"productWith" + (System.nanoTime() & 0xFF) .
-Change how the matrices are initialized and avoid the problem from 
within the Driver.

I only ask because transpose() employs the first technique, and I don't 
know how common (or uncommon) this particular situation is that might 
warrant the first change I mentioned.

Thank you!

Regards,
Shannon

Re: Multiple calls to DistributedRowMatrix.times()

Posted by Shannon Quinn <sq...@gatech.edu>.
A is symmetric, B is diagonal, both have exactly the same dimensions.

I could do this operation linearly as someone else had suggested. Where 
L = BAB, I could do L_ij = B_ii * A_ij * B_jj, and use a DenseVector for 
B instead. I just felt this was more straightforward code-wise.

On 6/17/2010 5:28 PM, Ted Dunning wrote:
> And symmetric?
>
> On Thu, Jun 17, 2010 at 2:28 PM, Shannon Quinn<sq...@gatech.edu>  wrote:
>
>    
>> A and B are both square matrices
>>      
>    


Re: Multiple calls to DistributedRowMatrix.times()

Posted by Ted Dunning <te...@gmail.com>.
And symmetric?

On Thu, Jun 17, 2010 at 2:28 PM, Shannon Quinn <sq...@gatech.edu> wrote:

> A and B are both square matrices

Re: Multiple calls to DistributedRowMatrix.times()

Posted by Shannon Quinn <sq...@gatech.edu>.
Good question, actually - forgot to mention to mention that A and B are 
both square matrices. Sorry about that.

On 6/17/2010 5:18 PM, Ted Dunning wrote:
> Off topic, but do you mean B' A B?
>
>
> On Thu, Jun 17, 2010 at 1:35 PM, Shannon Quinn<sq...@gatech.edu>  wrote:
>
>    
>> I'm attempting to carry out the multiplication of two DistributedRowMatrix
>> objects. It's similar to a SVD multiplication in that I have a matrix A and
>> a matrix B, and the operation looks like BAB
>>      
>    


Re: Multiple calls to DistributedRowMatrix.times()

Posted by Ted Dunning <te...@gmail.com>.
Off topic, but do you mean B' A B?


On Thu, Jun 17, 2010 at 1:35 PM, Shannon Quinn <sq...@gatech.edu> wrote:

> I'm attempting to carry out the multiplication of two DistributedRowMatrix
> objects. It's similar to a SVD multiplication in that I have a matrix A and
> a matrix B, and the operation looks like BAB

Re: Multiple calls to DistributedRowMatrix.times()

Posted by Jake Mannix <ja...@gmail.com>.
Hey Shannon,

  The whole useage of temp directories in DistributedRowMatrix is...
"confused" at best.  It needs to be better managed, really.  But out of the
two approaches to this current problem,

-Modify DistributedRowMatrix.times() so "productWith" is instead
"productWith" + (System.nanoTime() & 0xFF) .

I think I like this approach, yes.

  -jake

On Thu, Jun 17, 2010 at 1:35 PM, Shannon Quinn <sq...@gatech.edu> wrote:

> Hi all,
>
> I'm attempting to carry out the multiplication of two DistributedRowMatrix
> objects. It's similar to a SVD multiplication in that I have a matrix A and
> a matrix B, and the operation looks like BAB, or:
>
> B.times(A.times(B))
>
> When I run it, I get a FileAlreadyExistsException from Hadoop, since for
> both A and B the result of outputTmpBasePath.getParent() is the same (A is
> read from input, and B is calculated from A). Which would be a better
> approach:
>
> -Modify DistributedRowMatrix.times() so "productWith" is instead
> "productWith" + (System.nanoTime() & 0xFF) .
> -Change how the matrices are initialized and avoid the problem from within
> the Driver.
>
> I only ask because transpose() employs the first technique, and I don't
> know how common (or uncommon) this particular situation is that might
> warrant the first change I mentioned.
>
> Thank you!
>
> Regards,
> Shannon
>