You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com> on 2010/11/19 09:17:51 UTC
Lanczos Algorithm
Dear Mahout developers,
I'm a Computer Science student from the National University of Distance Education in Spain. I'm currently developing my final year project which is about Diffusion Maps.
This method is used for dimensionality reduction and it uses the Lanczos algorithm during its operations. The method is already implemented in the last release version
of Mahout in the LanczosSolver class but we foresee the need to use the algorithm with distributed calculations. This implementation of Diffusion Maps has to deal
with extremely large matrices and the distributed calculation is critical for me.
I have noticed that there is a DistributedLanzcosSolver class implemented in the Mahout library but I can’t have access to the source code because it isn't in the
last release version of Mahout.
Could you please let me know if I could have access to the source code of this class?Also I would like to ask you about how the LanczosSolver implementation works. I have made some test between this class and other
program which has been implemented in R. This program is using a library called Arpack, which also uses the Lanczos algorithm. When I calculate the eigenvalues and the eigenvectors of a symmetric matrix. I haven’t the same results. For example:
For this matrix:
4.42282138 1.51744077 0.07690571 0.93650042 2.19609401
1.51744077 1.73849477 -0.11856149 0.76555191 1.3673608
0.07690571 -0.11856149 0.55065932 1.72163263 -0.2283693
0.93650042 0.76555191 1.72163263 0.09470345 -1.16626194
2.19609401 1.3673608 -0.2283693 -1.16626194 -0.37321311
Results for R:
Eigenvalues
-0.6442398 1.1084103 2.3946915 6.2018925
Eigenvectors [,1] [,2] [,3] [,4]
[1,] -0.17050824 0.46631043 -0.010360993 0.83660453
[2,] -0.06455473 -0.87762807 -0.008814402 0.40939079
[3,] 0.68602882 0.04706265 -0.666429293 0.02602181
[4,] -0.39567054 -0.07491643 -0.670834157 0.12161492
[5,] 0.58272541 -0.06705358 0.325066897 0.34208875
Results for Java:
Eigenvalues
0.0 0.007869004183962289 0.023293016691817894 0.10872358093523908 0.13087002850143611
I never get the same eigenvalues, I think this is because the documentation of the class says:
To avoid floating point overflow problems which arise in power-methods like Lanczos, an initial pass is made through the input matrix to generate a good
starting seed vector by summing all the rows of the input matrix, and compute the trace(inputMatrixt*matrix)
This latter value, being the sum of all of the singular values, is used to rescale the entire matrix, effectively forcing the largest singular value to be strictly
less than one, and transforming floating point overflow problems into floating point underflow (ie, very small singular values will become invisible, as they will
appear to be zero and the algorithm will terminate).
Is it possible to return the eigenvalues to theirs original value?
Eigenvectors
-0.83660453 0.23122937 0.010360993 0.46631043 -0.17050824
-0.40939079 0.24067227 0.008814402 -0.87762807 -0.06455473
-0.02602181 0.28695718 0.666429293 0.04706265 0.68602882
-0.12161492 -0.61075665 0.670834157 -0.07491643 -0.39567054
-0.34208875 -0.65821099 -0.325066897 -0.06705358 0.58272541
Always happens the same. I have to force the calculation of N vectors (with an N x N matrix) to obtain the same values for the eigenvectors,
except in the sign of some of the values, which is acceptable. I thought this implementation of the algorithm should return the eigenvectors sorted but all the time I’m obtaining a vector which I don’t want to calculate between them.
In the example above it’s the second one starting from the left.Why is this happen?
Thanks in advance.
K.r.Pedro
Re: Lanczos Algorithm
Posted by Jake Mannix <ja...@gmail.com>.
On Wed, Nov 24, 2010 at 11:24 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
pmjimenez1983@hotmail.com> wrote:
>
> Hi Jake,
>
> Sorry I was trying to say less than 1 not 0, but anyway I have negative
> values:
>
Ahah! Your input matrix has negative values, so of course for a symmetric
matrix with negative entries you
can have an output with negative eigenvalues. This makes total sense, and
is expected: what are the
eigenvalues of the matrix: [[-1, 0], [0,1]]? {1, -1} of course.
If you were inputting an asymmetric matrix, and getting singular vectors,
then the singular values will
always be positive, but this is not your current case.
-jake
RE: Lanczos Algorithm
Posted by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>.
Hi Jake,
Sorry I was trying to say less than 1 not 0, but anyway I have negative values:
lanczos.solve(matrix, 6, vectores, lista, true);
Matrix
4.42282138 1.51744077 0.07690571 0.93650042 2.19609401
1.51744077 1.73849477 -0.11856149 0.76555191 1.3673608
0.07690571 -0.11856149 0.55065932 1.72163263 -0.2283693
0.93650042 0.76555191 1.72163263 0.09470345 -1.16626194
2.19609401 1.3673608 -0.2283693 -1.16626194 -0.37321311
Vectors
-0.836604532811076 0.010360993366570937 0.4663104348755353 -0.21103053642067893 0.17050824456194816
-0.40939078852782784 0.008814403593603212 -0.8776280713949454 0.39622059899392775 0.06455472788249345
-0.0260218119292915 0.6664292935945985 0.047062649256241984 -0.19380355385572554 -0.6860288199629306
-0.12161491901667112 0.6708341567012993 -0.07491642748270017 0.8613491282476691 0.3956705425673934
-0.34208875270170686 -0.3250668961115665 -0.06705357979997184 -0.13781586051962705 -0.5827254118099825
Values
-0.04981181942708407 -0.012214401586606135 0.0 0.021014795857212765 0.045401915968274364
Maybe I'm doing something wrong..
Thanks.
Pedro.
----------------------------------------
> From: jake.mannix@gmail.com
> Date: Tue, 23 Nov 2010 13:23:36 -0800
> Subject: Re: Lanczos Algorithm
> To: user@mahout.apache.org
>
> On Tue, Nov 23, 2010 at 12:00 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> pmjimenez1983@hotmail.com> wrote:
> >
> > And because of that all the eigenValues I get are less than 0. Is it
> > possible to get te real values?
> >
>
> You should not be getting negative eigenValues. That's a bug in the system
> if it's happening. Can you
> send the output of the LanczosSolver when you see negative eigenvalues
> coming out?
>
> -jake
Re: Lanczos Algorithm
Posted by Jake Mannix <ja...@gmail.com>.
On Tue, Nov 23, 2010 at 12:00 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
pmjimenez1983@hotmail.com> wrote:
>
> And because of that all the eigenValues I get are less than 0. Is it
> possible to get te real values?
>
You should not be getting negative eigenValues. That's a bug in the system
if it's happening. Can you
send the output of the LanczosSolver when you see negative eigenvalues
coming out?
-jake
RE: Lanczos Algorithm
Posted by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>.
Hi Jake,
Thanks a lot for your reply.
I just have another question, the LanzcosSolver documentation says:
To avoid floating point overflow problems which arise in power-methods like Lanczos, an initial pass is made
through the input matrix to
generate a good starting seed vector by summing all the rows of the input matrix, andcompute the trace(inputMatrixt*matrix)
This latter value, being the sum of all of the singular values, is used to rescale the entire matrix, effectively
forcing the largest singular value to be strictly less than one, and transforming floating point overflow
problems into floating point underflow (ie, very small singular values will become invisible, as they
will appear to be zero and the algorithm will terminate).
And because of that all the eigenValues I get are less than 0. Is it possible to get te real values?
K.R.
Pedro.
----------------------------------------
> From: jake.mannix@gmail.com
> Date: Mon, 22 Nov 2010 14:01:31 -0800
> Subject: Re: Lanczos Algorithm
> To: user@mahout.apache.org
>
> Hi Pedro,
>
> I wrote the Lanczos stuff in Mahout, but I've been swamped with my new job
> the past few months, and haven't been able to interact on the list much, but
> I've got a minute or two to try and answer your questions.
>
> First: the LanczosSolver does indeed leave you with eigenvalues in the
> opposite order than you expect (ascending order, instead of descending). It
> should be noted that in general, the Lanczos method *does not* give you the
> "largest N eigenvalues" - if you ask for a rank-N decomposition via Lanczos,
> you'll get the smallest eigenvalue for sure, then somewhat less than N/2 of
> the other small eigenvalues (with the last few being right in the middle of
> the spectrum), then somewhat less than N/2 of the largest eigenvalues (with
> the first few being in the middle of the spectrum), ending with the largest
> eigenvalue.
>
> If you really want the top N eigenvalues, then you should ask for somewhere
> between 3N and 4N rank decomposition, and take the last N of these. You
> won't be guaranteed to get *exactly all* of the top N eigenvalues,
> especially if you have a very rapidly decreasing eigenvalue spectrum (as is
> usually the case with real data), and you reach the "plateau" of middling
> values before N. In this case, if there are K
> get all K of these, and then N-K's worth of a sampling of the middling
> eigenvalues.
>
> One other thing you need to remember about the LanczosSolver: if your input
> matrix is symmetric, pass in the boolean isSymmetric=true to the solve()
> method, or else you'll get wrong values.
>
> -jake
>
> On Mon, Nov 22, 2010 at 1:07 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> pmjimenez1983@hotmail.com> wrote:
>
> >
> > Hi Ted,
> >
> > I can't give you an exact amount but more or less it could be around 10^5
> > non-zero elements per row.
> >
> > Could you please let me know, why the lanzcos algorithm is not always
> > returning the values in a decreasing order?
> >
> > Thanks.
> >
> > Pedro.
> >
> > ----------------------------------------
> > > From: ted.dunning@gmail.com
> > > Date: Fri, 19 Nov 2010 13:34:19 -0800
> > > Subject: Re: Lanczos Algorithm
> > > To: user@mahout.apache.org
> > >
> > > How many non-zero elements?
> > >
> > > On Fri, Nov 19, 2010 at 12:34 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > > pmjimenez1983@hotmail.com> wrote:
> > >
> > > >
> > > >
> > > > I was talking about 10^9 rows and 10^9 columns
> > > >
> > > > ----------------------------------------
> > > > > From: ted.dunning@gmail.com
> > > > > Date: Fri, 19 Nov 2010 12:07:16 -0800
> > > > > Subject: Re: Lanczos Algorithm
> > > > > To: user@mahout.apache.org
> > > > >
> > > > > On Fri, Nov 19, 2010 at 11:17 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > > > > pmjimenez1983@hotmail.com> wrote:
> > > > >
> > > > > > In this project I would have to work with matrix of 10^9, which
> > have a
> > > > very
> > > > > > sparse data.
> > > > >
> > > > >
> > > > > I think you mean 10^9 rows and 10^9 columns with much fewer 10^18
> > > > non-zero
> > > > > elements.
> > > > >
> > > > > Is that correct?
> > > > >
> > > > > Can you say how many non-zero elements?
> > > >
> > > >
> >
> >
Re: Lanczos Algorithm
Posted by Derek O'Callaghan <de...@ucd.ie>.
Hi Jake,
Thanks for the clarification regarding S-1. FYI the EigenVerificationJob
is already included in one of the DistributedLanczosSolver run()
methods. I see the EigenVector instances being written in
EigenVerificationJob.saveCleanEigens(), however, when they're read back
in TestClusterDumper.testKmeansSVD(), the vectors are actually
DenseVector instances, not EigenVectors, and so the associated
eigenValue is lost as it's currently encapsulated in EigenVector.name. I
think VectorWritable is just persisting DenseVectors and isn't aware of
EigenVectors, but I'd need to dig a bit deeper to confirm.
I just wanted to confirm that S should be constructed using the sqrts of
the eigenvalues generated by Lanczos/EigenVerificationJob?
Thanks again,
Derek
On 23/11/10 22:03, Jake Mannix wrote:
> Not sure about what is done in TestClusterDumper, but in general, to
> project
> the original rows of your matrix onto the reduced space defined by the
> decomposition, you do want to rescale by S^-1, or else you'll basically find
> that all of your rows seem to point in the direction of the largest
> eigenvector (that's why it's the largest eigenvector: most of the matrix
> points in it's direction!).
>
>
>
>> and the reduced matrix X can then be passed to KMeans for clustering. I
>> wanted to confirm if this is correct, and that the S (derived from the
>> Lanczos-generated eigenvalues) diagonal matrix can be ignored when
>> projecting the original matrix? Is this the reason why Lanczos only persists
>> the eigenvectors, and discards the eigenvalues
>> (DistributedLanczosSolver.serializeOutput())?
>>
>>
> I don't think so. I think you do want the eigenvalues as well. Because
> Lanczos can sometimes have stability issues, and end up with repeats of
> eigenvector/eigenvalue pairs, you need to do some checking on the output.
> This is done in the EigenVerificationJob class, which takes your original
> corpus, and the supposed eigenvectors (doesn't need the eigenvalues), and
> throws away any duplicates or incorrect vectors, and recomputes the
> eigenvalues/singular values and indeed stores them as well as the vectors
> (see the method saveCleanEigens() ).
>
> These recent discussions reminds me that the EigenVerificationJob needs to
> be just folded into the DistributedLanczosSolver, because it's confusing and
> nobody sees that they typically need to use it.
>
> -jake
>
>
Re: Lanczos Algorithm
Posted by Fernando Fernández <fe...@gmail.com>.
That helps a lot, Jake, I want to thank you so much for your patience
answering all of our questions.
Regarding the negative eigenvalues issue, I have obtained today a similar
result: the smallest eigenvalue negative (but very close to 0) and the
biggest one bigger than 1 (~1.02), but I must point that this happened with
artificially augmented data (copied and pasted the same 4 rows hundreds of
times to make some performance tests) so I think the problem can be related
to this. Pedro, are your data artificially generated?
Last, I think that it would be helpful for other people if these questions
are somewhat moved to the Mahout wiki (maybe some kind of FAQ) since I think
they are becoming so "frequent".
Thanks a lot again Jake.
Best,
Fernando.
2010/11/23 Jake Mannix <ja...@gmail.com>
> On Tue, Nov 23, 2010 at 9:58 AM, Derek O'Callaghan
> <de...@ucd.ie>wrote:
>
> > Hi Jake,
> >
> > I have some related questions about the usage of the eigenvectors and
> > eigenvalues generated by Lanczos, they're more or less on-topic so I
> thought
> > it'd be okay to post them here, but I can start a new thread if you like.
> > I've been going through some of the mails on the dev list regarding the
> > projection of a matrix onto an SVD basis which is generated by Lanczos,
> in
> > order to reduce the dimensionality of the matrix columns. The new matrix
> is
> > then passed to KMeans for clustering.
> >
>
> Ok, sounds good.
>
>
> > From Jeff's mail above, and the code in TestClusterDumper, it seems like
> > the second multiplication by S^-1 step is not performed/required, i.e.
> the
> > only step to project the original matrix A is:
> >
> > Reduced matrix X = A . V (or A . P using Jeff's notation)
> >
>
> Not sure about what is done in TestClusterDumper, but in general, to
> project
> the original rows of your matrix onto the reduced space defined by the
> decomposition, you do want to rescale by S^-1, or else you'll basically
> find
> that all of your rows seem to point in the direction of the largest
> eigenvector (that's why it's the largest eigenvector: most of the matrix
> points in it's direction!).
>
>
> > and the reduced matrix X can then be passed to KMeans for clustering. I
> > wanted to confirm if this is correct, and that the S (derived from the
> > Lanczos-generated eigenvalues) diagonal matrix can be ignored when
> > projecting the original matrix? Is this the reason why Lanczos only
> persists
> > the eigenvectors, and discards the eigenvalues
> > (DistributedLanczosSolver.serializeOutput())?
> >
>
> I don't think so. I think you do want the eigenvalues as well. Because
> Lanczos can sometimes have stability issues, and end up with repeats of
> eigenvector/eigenvalue pairs, you need to do some checking on the output.
> This is done in the EigenVerificationJob class, which takes your original
> corpus, and the supposed eigenvectors (doesn't need the eigenvalues), and
> throws away any duplicates or incorrect vectors, and recomputes the
> eigenvalues/singular values and indeed stores them as well as the vectors
> (see the method saveCleanEigens() ).
>
> These recent discussions reminds me that the EigenVerificationJob needs to
> be just folded into the DistributedLanczosSolver, because it's confusing
> and
> nobody sees that they typically need to use it.
>
> -jake
>
Re: Lanczos Algorithm
Posted by Jake Mannix <ja...@gmail.com>.
On Tue, Nov 23, 2010 at 9:58 AM, Derek O'Callaghan
<de...@ucd.ie>wrote:
> Hi Jake,
>
> I have some related questions about the usage of the eigenvectors and
> eigenvalues generated by Lanczos, they're more or less on-topic so I thought
> it'd be okay to post them here, but I can start a new thread if you like.
> I've been going through some of the mails on the dev list regarding the
> projection of a matrix onto an SVD basis which is generated by Lanczos, in
> order to reduce the dimensionality of the matrix columns. The new matrix is
> then passed to KMeans for clustering.
>
Ok, sounds good.
> From Jeff's mail above, and the code in TestClusterDumper, it seems like
> the second multiplication by S^-1 step is not performed/required, i.e. the
> only step to project the original matrix A is:
>
> Reduced matrix X = A . V (or A . P using Jeff's notation)
>
Not sure about what is done in TestClusterDumper, but in general, to project
the original rows of your matrix onto the reduced space defined by the
decomposition, you do want to rescale by S^-1, or else you'll basically find
that all of your rows seem to point in the direction of the largest
eigenvector (that's why it's the largest eigenvector: most of the matrix
points in it's direction!).
> and the reduced matrix X can then be passed to KMeans for clustering. I
> wanted to confirm if this is correct, and that the S (derived from the
> Lanczos-generated eigenvalues) diagonal matrix can be ignored when
> projecting the original matrix? Is this the reason why Lanczos only persists
> the eigenvectors, and discards the eigenvalues
> (DistributedLanczosSolver.serializeOutput())?
>
I don't think so. I think you do want the eigenvalues as well. Because
Lanczos can sometimes have stability issues, and end up with repeats of
eigenvector/eigenvalue pairs, you need to do some checking on the output.
This is done in the EigenVerificationJob class, which takes your original
corpus, and the supposed eigenvectors (doesn't need the eigenvalues), and
throws away any duplicates or incorrect vectors, and recomputes the
eigenvalues/singular values and indeed stores them as well as the vectors
(see the method saveCleanEigens() ).
These recent discussions reminds me that the EigenVerificationJob needs to
be just folded into the DistributedLanczosSolver, because it's confusing and
nobody sees that they typically need to use it.
-jake
Re: Lanczos Algorithm
Posted by Derek O'Callaghan <de...@ucd.ie>.
Hi Jake,
I have some related questions about the usage of the eigenvectors and
eigenvalues generated by Lanczos, they're more or less on-topic so I
thought it'd be okay to post them here, but I can start a new thread if
you like. I've been going through some of the mails on the dev list
regarding the projection of a matrix onto an SVD basis which is
generated by Lanczos, in order to reduce the dimensionality of the
matrix columns. The new matrix is then passed to KMeans for clustering.
(see "Re: Using SVD with Canopy/KMeans" thread on the dev list)
On 03/09/10 04:28, Jeff Eastman wrote:
"...A P is the projection of the data set onto the eigenvector basis:
A = original data matrix ([15x39])
P = eigenvector column matrix ([39x7])
D = eigenvalue diagonal matrix
Then:
A P = P D => A = P D P' "
where the projection of A is achieved with: A times P (see
TestClusterDumper.testKmeansSVD())
On 12/09/10 04:21, Jake Mannix wrote:
"To adapt this to do SVD, you just substitute A = C' * C, for any
(non-square) matrix C. You get a set of eigenvectors of C' * C, which in
turn are the right-singular vectors of C (what is called V, if C = U D V')."
This seems fine so far, in that the P in A times P is the V from the SVD
definition in Jake's mail.
However, from an older thread ("distributed svd" on the user list):
On 24/07/10 00:09, Jake Mannix wrote:
"What you want to do here is actually take the A matrix, represented by a
DistributedRowMatrix, and the V_k matrix (you actually get basically V_k's
transposed from the SVD job), represented by that same class, and then just
compute A . V . S^-1, where S is the diagonal matrix of singular values
(note that these are the sqrts of the eigenvalues you get out of Lancos)."
From Jeff's mail above, and the code in TestClusterDumper, it seems
like the second multiplication by S^-1 step is not performed/required,
i.e. the only step to project the original matrix A is:
Reduced matrix X = A . V (or A . P using Jeff's notation)
and the reduced matrix X can then be passed to KMeans for clustering. I
wanted to confirm if this is correct, and that the S (derived from the
Lanczos-generated eigenvalues) diagonal matrix can be ignored when
projecting the original matrix? Is this the reason why Lanczos only
persists the eigenvectors, and discards the eigenvalues
(DistributedLanczosSolver.serializeOutput())?
Thanks,
Derek
On 22/11/10 22:01, Jake Mannix wrote:
> Hi Pedro,
>
> I wrote the Lanczos stuff in Mahout, but I've been swamped with my new job
> the past few months, and haven't been able to interact on the list
> much, but
> I've got a minute or two to try and answer your questions.
>
> First: the LanczosSolver does indeed leave you with eigenvalues in the
> opposite order than you expect (ascending order, instead of
> descending). It
> should be noted that in general, the Lanczos method *does not* give
> you the
> "largest N eigenvalues" - if you ask for a rank-N decomposition via
> Lanczos,
> you'll get the smallest eigenvalue for sure, then somewhat less than
> N/2 of
> the other small eigenvalues (with the last few being right in the
> middle of
> the spectrum), then somewhat less than N/2 of the largest eigenvalues
> (with
> the first few being in the middle of the spectrum), ending with the
> largest
> eigenvalue.
>
> If you really want the top N eigenvalues, then you should ask for
> somewhere
> between 3N and 4N rank decomposition, and take the last N of these. You
> won't be guaranteed to get *exactly all* of the top N eigenvalues,
> especially if you have a very rapidly decreasing eigenvalue spectrum
> (as is
> usually the case with real data), and you reach the "plateau" of middling
> values before N. In this case, if there are K<N "large" eigenvalues,
> you'll
> get all K of these, and then N-K's worth of a sampling of the middling
> eigenvalues.
>
> One other thing you need to remember about the LanczosSolver: if your
> input
> matrix is symmetric, pass in the boolean isSymmetric=true to the solve()
> method, or else you'll get wrong values.
>
> -jake
>
> On Mon, Nov 22, 2010 at 1:07 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> pmjimenez1983@hotmail.com> wrote:
>
>>
>> Hi Ted,
>>
>> I can't give you an exact amount but more or less it could be around 10^5
>> non-zero elements per row.
>>
>> Could you please let me know, why the lanzcos algorithm is not always
>> returning the values in a decreasing order?
>>
>> Thanks.
>>
>> Pedro.
>>
>> ----------------------------------------
>>> From: ted.dunning@gmail.com
>>> Date: Fri, 19 Nov 2010 13:34:19 -0800
>>> Subject: Re: Lanczos Algorithm
>>> To: user@mahout.apache.org
>>>
>>> How many non-zero elements?
>>>
>>> On Fri, Nov 19, 2010 at 12:34 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
>>> pmjimenez1983@hotmail.com> wrote:
>>>
>>>>
>>>>
>>>> I was talking about 10^9 rows and 10^9 columns
>>>>
>>>> ----------------------------------------
>>>>> From: ted.dunning@gmail.com
>>>>> Date: Fri, 19 Nov 2010 12:07:16 -0800
>>>>> Subject: Re: Lanczos Algorithm
>>>>> To: user@mahout.apache.org
>>>>>
>>>>> On Fri, Nov 19, 2010 at 11:17 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
>>>>> pmjimenez1983@hotmail.com> wrote:
>>>>>
>>>>>> In this project I would have to work with matrix of 10^9, which
>> have a
>>>> very
>>>>>> sparse data.
>>>>>
>>>>>
>>>>> I think you mean 10^9 rows and 10^9 columns with much fewer 10^18
>>>> non-zero
>>>>> elements.
>>>>>
>>>>> Is that correct?
>>>>>
>>>>> Can you say how many non-zero elements?
>>>>
>>>>
>>
>>
>
Re: Lanczos Algorithm
Posted by Jake Mannix <ja...@gmail.com>.
Hi Pedro,
I wrote the Lanczos stuff in Mahout, but I've been swamped with my new job
the past few months, and haven't been able to interact on the list much, but
I've got a minute or two to try and answer your questions.
First: the LanczosSolver does indeed leave you with eigenvalues in the
opposite order than you expect (ascending order, instead of descending). It
should be noted that in general, the Lanczos method *does not* give you the
"largest N eigenvalues" - if you ask for a rank-N decomposition via Lanczos,
you'll get the smallest eigenvalue for sure, then somewhat less than N/2 of
the other small eigenvalues (with the last few being right in the middle of
the spectrum), then somewhat less than N/2 of the largest eigenvalues (with
the first few being in the middle of the spectrum), ending with the largest
eigenvalue.
If you really want the top N eigenvalues, then you should ask for somewhere
between 3N and 4N rank decomposition, and take the last N of these. You
won't be guaranteed to get *exactly all* of the top N eigenvalues,
especially if you have a very rapidly decreasing eigenvalue spectrum (as is
usually the case with real data), and you reach the "plateau" of middling
values before N. In this case, if there are K<N "large" eigenvalues, you'll
get all K of these, and then N-K's worth of a sampling of the middling
eigenvalues.
One other thing you need to remember about the LanczosSolver: if your input
matrix is symmetric, pass in the boolean isSymmetric=true to the solve()
method, or else you'll get wrong values.
-jake
On Mon, Nov 22, 2010 at 1:07 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
pmjimenez1983@hotmail.com> wrote:
>
> Hi Ted,
>
> I can't give you an exact amount but more or less it could be around 10^5
> non-zero elements per row.
>
> Could you please let me know, why the lanzcos algorithm is not always
> returning the values in a decreasing order?
>
> Thanks.
>
> Pedro.
>
> ----------------------------------------
> > From: ted.dunning@gmail.com
> > Date: Fri, 19 Nov 2010 13:34:19 -0800
> > Subject: Re: Lanczos Algorithm
> > To: user@mahout.apache.org
> >
> > How many non-zero elements?
> >
> > On Fri, Nov 19, 2010 at 12:34 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > pmjimenez1983@hotmail.com> wrote:
> >
> > >
> > >
> > > I was talking about 10^9 rows and 10^9 columns
> > >
> > > ----------------------------------------
> > > > From: ted.dunning@gmail.com
> > > > Date: Fri, 19 Nov 2010 12:07:16 -0800
> > > > Subject: Re: Lanczos Algorithm
> > > > To: user@mahout.apache.org
> > > >
> > > > On Fri, Nov 19, 2010 at 11:17 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > > > pmjimenez1983@hotmail.com> wrote:
> > > >
> > > > > In this project I would have to work with matrix of 10^9, which
> have a
> > > very
> > > > > sparse data.
> > > >
> > > >
> > > > I think you mean 10^9 rows and 10^9 columns with much fewer 10^18
> > > non-zero
> > > > elements.
> > > >
> > > > Is that correct?
> > > >
> > > > Can you say how many non-zero elements?
> > >
> > >
>
>
Re: Lanczos Algorithm
Posted by Jake Mannix <ja...@gmail.com>.
2010/11/23 Fernando Fernández <fe...@gmail.com>
> Hi Jake,
>
> You say that LanczosSolver leaves the eigenvalues in the reverse order that
> I expect, but, at least the eigenvectors are given in the same order,
> aren't
> they? I mean, If I want the top N eigenvalues and eigenvectors, and i ask
> for 4N, I should take the last N eigenvalues and the N eigenvectors, right?
>
The eigenvectors should be paired up with their values, yes. If you look at
the class org.apache.mahout.math.hadoop.decomposer.EigenVerificationJob,
you'll find the way I typically run the algorithm. I notice in looking at
that,
that it's not set up to handle symmetric matrices properly - it always
assumes
the input matrix is asymmetric and you're looking for singular vectors (not
eigenvectors of a symmetric matrix).
> I read also that there is an issue regarding the number of eigenvalues and
> eigenvectors that are returned when you ask LanczosSolver for N of them,
> and
> I have checked that if I ask for 40, for example, I get only 39. Will this
> leave out the biggest eigenvector and eigenvalue or the smallest one, or
> none of these? I spent the last weeks reading abouth this in the
> discussions
> but it's not very clear for me yet.
>
Your last eigenvector will be the largest one, and the first will be the
smallest
one. You are getting one rank less than you asked for, however, and that
seems to be because the code is treating your initial guess vector (that you
feed into the algorithm to start iterative matrix multiplication) as one
of the possible eigenvectors (which is most certainly is not). That might
not
be the issue, but either way, I think if you want the biggest 10
eigenvectors,
if you ask for 30 or 40, and take the last 10, you'll have the biggest 10
(not
the 2nd biggest through 11th biggest).
> Could you give us some help?
>
> By the way, I'm interested in having Uk and also V*S(-1) so i'm able to
> project new rows into the U space...
>
That's a simple matter of doing some extra matrix multiplications, right?
The
output of the current implementation of Lanczos is K vectors with
dimensionality
equal to M = numColumns(inputMatrix), and thus naturally form a dense,
row-partitioned K x M matrix = V^t. If you scale each of these singular
vectors
by 1/(their singular value), you'll have a set of K vectors suitable for
projecting
new rows onto the reduced dimensional space (of course, you also need to
project all of the original rows onto this space too, because you have V^t,
not
U. In fact, if ever matrix is being represented as a row-matrix, you don't
really want U, because that's a matrix of K singular vectors, each having
numRows(inputMatrix) values - you want numRows vectors, each having K
values: U^t).
Does that help?
-jake
Re: Lanczos Algorithm
Posted by Fernando Fernández <fe...@gmail.com>.
Hi Jake,
You say that LanczosSolver leaves the eigenvalues in the reverse order that
I expect, but, at least the eigenvectors are given in the same order, aren't
they? I mean, If I want the top N eigenvalues and eigenvectors, and i ask
for 4N, I should take the last N eigenvalues and the N eigenvectors, right?
I read also that there is an issue regarding the number of eigenvalues and
eigenvectors that are returned when you ask LanczosSolver for N of them, and
I have checked that if I ask for 40, for example, I get only 39. Will this
leave out the biggest eigenvector and eigenvalue or the smallest one, or
none of these? I spent the last weeks reading abouth this in the discussions
but it's not very clear for me yet.
Could you give us some help?
By the way, I'm interested in having Uk and also V*S(-1) so i'm able to
project new rows into the U space...
Thanks in advance!!
2010/11/22 Ted Dunning <te...@gmail.com>
> That seems like a lot. That would mean that have 10^14 = 100 trillion
> nonzero elements which would take 10PB to store with one bit per non-zero
> element.
>
> Are there many totally zero rows?
>
> Can you estimate how many non-zero elements you have in all?
>
> On Mon, Nov 22, 2010 at 1:07 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> pmjimenez1983@hotmail.com> wrote:
>
> >
> > Hi Ted,
> >
> > I can't give you an exact amount but more or less it could be around 10^5
> > non-zero elements per row.
> >
> > Could you please let me know, why the lanzcos algorithm is not always
> > returning the values in a decreasing order?
> >
> > Thanks.
> >
> > Pedro.
> >
> > ----------------------------------------
> > > From: ted.dunning@gmail.com
> > > Date: Fri, 19 Nov 2010 13:34:19 -0800
> > > Subject: Re: Lanczos Algorithm
> > > To: user@mahout.apache.org
> > >
> > > How many non-zero elements?
> > >
> > > On Fri, Nov 19, 2010 at 12:34 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > > pmjimenez1983@hotmail.com> wrote:
> > >
> > > >
> > > >
> > > > I was talking about 10^9 rows and 10^9 columns
> > > >
> > > > ----------------------------------------
> > > > > From: ted.dunning@gmail.com
> > > > > Date: Fri, 19 Nov 2010 12:07:16 -0800
> > > > > Subject: Re: Lanczos Algorithm
> > > > > To: user@mahout.apache.org
> > > > >
> > > > > On Fri, Nov 19, 2010 at 11:17 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > > > > pmjimenez1983@hotmail.com> wrote:
> > > > >
> > > > > > In this project I would have to work with matrix of 10^9, which
> > have a
> > > > very
> > > > > > sparse data.
> > > > >
> > > > >
> > > > > I think you mean 10^9 rows and 10^9 columns with much fewer 10^18
> > > > non-zero
> > > > > elements.
> > > > >
> > > > > Is that correct?
> > > > >
> > > > > Can you say how many non-zero elements?
> > > >
> > > >
> >
> >
>
Re: Lanczos Algorithm
Posted by Fernando Fernández <fe...@gmail.com>.
Hi everyone again
Can anyone confirm this? Is The V matrix persisted by the Lanczos algorithm
already scaled by (1/eigenvalues)?
Thanks in advance,
Fernando.
El 2 de diciembre de 2010 17:54, Fernando Fernández <
fernando.fernandez.gonzalez@gmail.com> escribió:
> Hi again,
>
>
>
> I have a question regarding the output of the solver. Is it really the V
> matrix or is it already V*S(-1). Jake said that I needed to scale the
> vectors by (1/eigenvalue), but when I do this programatically and serialize
> the output to the hdfs and check the results, I see that the V*S(-1) vectors
> that I built and the V that the solve method (or mahout svd command)
> persists has almost the same values, so it seems to be already scaled by
> (1/eigenvalues). Is this true?
>
> For example:
>
> Eigenvectors persisted by the solve method to the hdfs (A portion):
>
> 2 elts: {00:0.00969523384177557, 01:-0.1509405313358042,
> 02:-0.15402428299469786,
>
> Eigenvalue asociated: 0.35597101584493984
>
>
> When I take the V matrix from the solve method at execution time and scale
> by (1/eigenvalue):
>
> 2 elts: {00:0.009036918423663949, 01:-0.15711902318788887,
> 02:-0.14878245692981582
>
> If I multiply the first element by the eigenvalue (re-scale to obtain V
> again) I would obtain something like 0.00321688, very far from the
> 0.00969523384177557 that is persisted to the hdfs by the solve method.
>
> Can someone throw some light on this?
>
>
> Best,
> Fernando.
>
> El 26 de noviembre de 2010 09:49, Fernando Fernández <
> fernando.fernandez.gonzalez@gmail.com> escribió:
>
> Yes, I'm checking the error today with some colleagues and seems that we
>> broke something trying to fix other errors... now SVD doesn't work even on
>> the command line, I'll write again if I get back to the situation where the
>> command line worked and I got ClassNotFoundExceptions...
>>
>> Thanks!!
>>
>> 2010/11/26 Ted Dunning <te...@gmail.com>
>>
>> Ahh... the problem is not that classes are missing. It is that the code
>>> that is reading the
>>> matrix wants to see a LongWritable for the row number and is getting an
>>> IntWritable instead.
>>>
>>> I think that your file containing your matrix was somehow created
>>> incorrectly. This isn't necessarily
>>> your error, but I think that the error exists.
>>>
>>> 2010/11/25 Fernando Fernández <fe...@gmail.com>
>>>
>>> > java.lang.IllegalStateException: java.io.IOException: wrong key class:
>>> > org.apache.hadoop.io.IntWritable is not class
>>> > org.apache.hadoop.io.LongWritable
>>> >
>>>
>>
>>
>
Re: Lanczos Algorithm
Posted by Fernando Fernández <fe...@gmail.com>.
Hi again,
I have a question regarding the output of the solver. Is it really the V
matrix or is it already V*S(-1). Jake said that I needed to scale the
vectors by (1/eigenvalue), but when I do this programatically and serialize
the output to the hdfs and check the results, I see that the V*S(-1) vectors
that I built and the V that the solve method (or mahout svd command)
persists has almost the same values, so it seems to be already scaled by
(1/eigenvalues). Is this true?
For example:
Eigenvectors persisted by the solve method to the hdfs (A portion):
2 elts: {00:0.00969523384177557, 01:-0.1509405313358042,
02:-0.15402428299469786,
Eigenvalue asociated: 0.35597101584493984
When I take the V matrix from the solve method at execution time and scale
by (1/eigenvalue):
2 elts: {00:0.009036918423663949, 01:-0.15711902318788887,
02:-0.14878245692981582
If I multiply the first element by the eigenvalue (re-scale to obtain V
again) I would obtain something like 0.00321688, very far from the
0.00969523384177557 that is persisted to the hdfs by the solve method.
Can someone throw some light on this?
Best,
Fernando.
El 26 de noviembre de 2010 09:49, Fernando Fernández <
fernando.fernandez.gonzalez@gmail.com> escribió:
> Yes, I'm checking the error today with some colleagues and seems that we
> broke something trying to fix other errors... now SVD doesn't work even on
> the command line, I'll write again if I get back to the situation where the
> command line worked and I got ClassNotFoundExceptions...
>
> Thanks!!
>
> 2010/11/26 Ted Dunning <te...@gmail.com>
>
> Ahh... the problem is not that classes are missing. It is that the code
>> that is reading the
>> matrix wants to see a LongWritable for the row number and is getting an
>> IntWritable instead.
>>
>> I think that your file containing your matrix was somehow created
>> incorrectly. This isn't necessarily
>> your error, but I think that the error exists.
>>
>> 2010/11/25 Fernando Fernández <fe...@gmail.com>
>>
>> > java.lang.IllegalStateException: java.io.IOException: wrong key class:
>> > org.apache.hadoop.io.IntWritable is not class
>> > org.apache.hadoop.io.LongWritable
>> >
>>
>
>
Re: Lanczos Algorithm
Posted by Fernando Fernández <fe...@gmail.com>.
Yes, I'm checking the error today with some colleagues and seems that we
broke something trying to fix other errors... now SVD doesn't work even on
the command line, I'll write again if I get back to the situation where the
command line worked and I got ClassNotFoundExceptions...
Thanks!!
2010/11/26 Ted Dunning <te...@gmail.com>
> Ahh... the problem is not that classes are missing. It is that the code
> that is reading the
> matrix wants to see a LongWritable for the row number and is getting an
> IntWritable instead.
>
> I think that your file containing your matrix was somehow created
> incorrectly. This isn't necessarily
> your error, but I think that the error exists.
>
> 2010/11/25 Fernando Fernández <fe...@gmail.com>
>
> > java.lang.IllegalStateException: java.io.IOException: wrong key class:
> > org.apache.hadoop.io.IntWritable is not class
> > org.apache.hadoop.io.LongWritable
> >
>
Re: Lanczos Algorithm
Posted by Ted Dunning <te...@gmail.com>.
Ahh... the problem is not that classes are missing. It is that the code
that is reading the
matrix wants to see a LongWritable for the row number and is getting an
IntWritable instead.
I think that your file containing your matrix was somehow created
incorrectly. This isn't necessarily
your error, but I think that the error exists.
2010/11/25 Fernando Fernández <fe...@gmail.com>
> java.lang.IllegalStateException: java.io.IOException: wrong key class:
> org.apache.hadoop.io.IntWritable is not class
> org.apache.hadoop.io.LongWritable
>
Re: Lanczos Algorithm
Posted by Fernando Fernández <fe...@gmail.com>.
Yes, I'm using maven, so I don't understand it either. Here's the last
exception I got:
10/11/25 20:29:28 INFO lanczos.LanczosSolver: Finding 10 singular vectors of
matrix with 6 rows, via Lanczos
10/11/25 20:29:28 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
10/11/25 20:29:28 INFO compress.CodecPool: Got brand-new decompressor
ERROR CAUSE: java.io.IOException: wrong key class:
org.apache.hadoop.io.IntWritable is not class
org.apache.hadoop.io.LongWritable
java.lang.IllegalStateException: java.io.IOException: wrong key class:
org.apache.hadoop.io.IntWritable is not class
org.apache.hadoop.io.LongWritable
at
org.apache.mahout.math.hadoop.DistributedRowMatrix$DistributedMatrixIterator.hasNext(DistributedRowMatrix.java:230)
at
org.apache.mahout.math.decomposer.lanczos.LanczosSolver.getInitialVector(LanczosSolver.java:194)
at
org.apache.mahout.math.decomposer.lanczos.LanczosSolver.solve(LanczosSolver.java:107)
at #########################################################
Caused by: java.io.IOException: wrong key class:
org.apache.hadoop.io.IntWritable is not class
org.apache.hadoop.io.LongWritable
at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1824)
at
org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1876)
at
org.apache.mahout.math.hadoop.DistributedRowMatrix$DistributedMatrixIterator.hasNext(DistributedRowMatrix.java:221)
... 3 more
The ###... line is just the name of my project.
2010/11/25 Ted Dunning <te...@gmail.com>
> Mahout-math is a separate dependency. If you are using maven, the second
> order dependency should be picked up automagically. If you are using
> maven,
> I don't understand why the build wouldn't pick it up automatically.
>
> 2010/11/25 Fernando Fernández <fe...@gmail.com>
>
> > Hi again Jake,
> >
> > I'm having a lot of problems trying to use the DistributedLanczosSolver
> > class directly. I get ClassNotFound exceptions for
> > org.apache.mahout.math.vector, but svd seems to execute perfectly when I
> > use
> > it on the command line (The only problem is that I only get printed
> > eigenvalues and I need them to do the extra multiplications you mentioned
> > before). Can you tell which .java file or source code generates the job
> > under "mahout svd" command so I can compare it to my code and see if i'm
> > missing something that causes the exception? If I have this code, I just
> > need to add the lines needed to persist eigenvalues.
> >
> > Best,
> > Fernando.
> >
> > 2010/11/24 Jake Mannix <ja...@gmail.com>
> >
> > > 2010/11/24 Fernando Fernández <fe...@gmail.com>
> > >
> > > > Sorry jake,
> > > >
> > > > What do you mean by "length", do you mean the norm of the vector?
> > > >
> > >
> > > yes.
> > >
> > > -jake
> > >
> >
>
Re: Lanczos Algorithm
Posted by Ted Dunning <te...@gmail.com>.
Mahout-math is a separate dependency. If you are using maven, the second
order dependency should be picked up automagically. If you are using maven,
I don't understand why the build wouldn't pick it up automatically.
2010/11/25 Fernando Fernández <fe...@gmail.com>
> Hi again Jake,
>
> I'm having a lot of problems trying to use the DistributedLanczosSolver
> class directly. I get ClassNotFound exceptions for
> org.apache.mahout.math.vector, but svd seems to execute perfectly when I
> use
> it on the command line (The only problem is that I only get printed
> eigenvalues and I need them to do the extra multiplications you mentioned
> before). Can you tell which .java file or source code generates the job
> under "mahout svd" command so I can compare it to my code and see if i'm
> missing something that causes the exception? If I have this code, I just
> need to add the lines needed to persist eigenvalues.
>
> Best,
> Fernando.
>
> 2010/11/24 Jake Mannix <ja...@gmail.com>
>
> > 2010/11/24 Fernando Fernández <fe...@gmail.com>
> >
> > > Sorry jake,
> > >
> > > What do you mean by "length", do you mean the norm of the vector?
> > >
> >
> > yes.
> >
> > -jake
> >
>
Re: Lanczos Algorithm
Posted by Fernando Fernández <fe...@gmail.com>.
Hi again Jake,
I'm having a lot of problems trying to use the DistributedLanczosSolver
class directly. I get ClassNotFound exceptions for
org.apache.mahout.math.vector, but svd seems to execute perfectly when I use
it on the command line (The only problem is that I only get printed
eigenvalues and I need them to do the extra multiplications you mentioned
before). Can you tell which .java file or source code generates the job
under "mahout svd" command so I can compare it to my code and see if i'm
missing something that causes the exception? If I have this code, I just
need to add the lines needed to persist eigenvalues.
Best,
Fernando.
2010/11/24 Jake Mannix <ja...@gmail.com>
> 2010/11/24 Fernando Fernández <fe...@gmail.com>
>
> > Sorry jake,
> >
> > What do you mean by "length", do you mean the norm of the vector?
> >
>
> yes.
>
> -jake
>
Re: Lanczos Algorithm
Posted by Jake Mannix <ja...@gmail.com>.
2010/11/24 Fernando Fernández <fe...@gmail.com>
> Sorry jake,
>
> What do you mean by "length", do you mean the norm of the vector?
>
yes.
-jake
Re: Lanczos Algorithm
Posted by Fernando Fernández <fe...@gmail.com>.
Sorry jake,
What do you mean by "length", do you mean the norm of the vector?
2010/11/24 Jake Mannix <ja...@gmail.com>
> On Wed, Nov 24, 2010 at 11:39 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> pmjimenez1983@hotmail.com> wrote:
>
> >
> > This is in the worst case. I haved asked for the amount of data and
> > that's what somebody told me. Anyway I'm not going to make any test with
> > that amount of data.
> >
>
> Ok.
>
>
> > Is it possible to undone the changes in the original eigenvalues to get
> the
> > original ones?
> >
>
> If you take your input matrix, and multiply it by an eigenvector, the
> length
> of the resultant
> vector will be your eigenvalue.
>
> -jake
>
Re: Lanczos Algorithm
Posted by Jake Mannix <ja...@gmail.com>.
On Wed, Nov 24, 2010 at 11:39 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
pmjimenez1983@hotmail.com> wrote:
>
> This is in the worst case. I haved asked for the amount of data and
> that's what somebody told me. Anyway I'm not going to make any test with
> that amount of data.
>
Ok.
> Is it possible to undone the changes in the original eigenvalues to get the
> original ones?
>
If you take your input matrix, and multiply it by an eigenvector, the length
of the resultant
vector will be your eigenvalue.
-jake
RE: Lanczos Algorithm
Posted by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>.
Hi Jake,
This is in the worst case. I haved asked for the amount of data and
that's what somebody told me. Anyway I'm not going to make any test with
that amount of data.
I'm doing my final year project and I just have to make this works with a
reasonable amount of data. So I just need to know how the algorithm works.
Is it possible to undone the changes in the original eigenvalues to get the original ones?
Thanks.
Pedro.
----------------------------------------
> From: jake.mannix@gmail.com
> Date: Tue, 23 Nov 2010 13:35:32 -0800
> Subject: Re: Lanczos Algorithm
> To: user@mahout.apache.org
>
> What is the expected size of this matrix you have, not the worst case size?
> I have never heard of anyone doing a matrix decomposition on a
> 100-trillion non-zero element matrix before. Are you sure the *average*
> size of the rows would be 10^5 nonzero elements?
>
> On Tue, Nov 23, 2010 at 11:55 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> pmjimenez1983@hotmail.com> wrote:
>
> >
> > Well, this is in the worst case but it could be possible.
> >
> > I'm not going to make any tests with this amount of data because for me is
> > impossible but this project is part of a bigger one and they would have
> > enough space to deal with this amount of data.
> >
> >
> > ----------------------------------------
> > > From: ted.dunning@gmail.com
> > > Date: Mon, 22 Nov 2010 14:46:20 -0800
> > > Subject: Re: Lanczos Algorithm
> > > To: user@mahout.apache.org
> > >
> > > That seems like a lot. That would mean that have 10^14 = 100 trillion
> > > nonzero elements which would take 10PB to store with one bit per non-zero
> > > element.
> > >
> > > Are there many totally zero rows?
> > >
> > > Can you estimate how many non-zero elements you have in all?
> > >
> > > On Mon, Nov 22, 2010 at 1:07 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > > pmjimenez1983@hotmail.com> wrote:
> > >
> > > >
> > > > Hi Ted,
> > > >
> > > > I can't give you an exact amount but more or less it could be around
> > 10^5
> > > > non-zero elements per row.
> > > >
> > > > Could you please let me know, why the lanzcos algorithm is not always
> > > > returning the values in a decreasing order?
> > > >
> > > > Thanks.
> > > >
> > > > Pedro.
> > > >
> > > > ----------------------------------------
> > > > > From: ted.dunning@gmail.com
> > > > > Date: Fri, 19 Nov 2010 13:34:19 -0800
> > > > > Subject: Re: Lanczos Algorithm
> > > > > To: user@mahout.apache.org
> > > > >
> > > > > How many non-zero elements?
> > > > >
> > > > > On Fri, Nov 19, 2010 at 12:34 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > > > > pmjimenez1983@hotmail.com> wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > I was talking about 10^9 rows and 10^9 columns
> > > > > >
> > > > > > ----------------------------------------
> > > > > > > From: ted.dunning@gmail.com
> > > > > > > Date: Fri, 19 Nov 2010 12:07:16 -0800
> > > > > > > Subject: Re: Lanczos Algorithm
> > > > > > > To: user@mahout.apache.org
> > > > > > >
> > > > > > > On Fri, Nov 19, 2010 at 11:17 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ
> > <
> > > > > > > pmjimenez1983@hotmail.com> wrote:
> > > > > > >
> > > > > > > > In this project I would have to work with matrix of 10^9, which
> > > > have a
> > > > > > very
> > > > > > > > sparse data.
> > > > > > >
> > > > > > >
> > > > > > > I think you mean 10^9 rows and 10^9 columns with much fewer 10^18
> > > > > > non-zero
> > > > > > > elements.
> > > > > > >
> > > > > > > Is that correct?
> > > > > > >
> > > > > > > Can you say how many non-zero elements?
> > > > > >
> > > > > >
> > > >
> > > >
> >
> >
Re: Lanczos Algorithm
Posted by Jake Mannix <ja...@gmail.com>.
What is the expected size of this matrix you have, not the worst case size?
I have never heard of anyone doing a matrix decomposition on a
100-trillion non-zero element matrix before. Are you sure the *average*
size of the rows would be 10^5 nonzero elements?
On Tue, Nov 23, 2010 at 11:55 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
pmjimenez1983@hotmail.com> wrote:
>
> Well, this is in the worst case but it could be possible.
>
> I'm not going to make any tests with this amount of data because for me is
> impossible but this project is part of a bigger one and they would have
> enough space to deal with this amount of data.
>
>
> ----------------------------------------
> > From: ted.dunning@gmail.com
> > Date: Mon, 22 Nov 2010 14:46:20 -0800
> > Subject: Re: Lanczos Algorithm
> > To: user@mahout.apache.org
> >
> > That seems like a lot. That would mean that have 10^14 = 100 trillion
> > nonzero elements which would take 10PB to store with one bit per non-zero
> > element.
> >
> > Are there many totally zero rows?
> >
> > Can you estimate how many non-zero elements you have in all?
> >
> > On Mon, Nov 22, 2010 at 1:07 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > pmjimenez1983@hotmail.com> wrote:
> >
> > >
> > > Hi Ted,
> > >
> > > I can't give you an exact amount but more or less it could be around
> 10^5
> > > non-zero elements per row.
> > >
> > > Could you please let me know, why the lanzcos algorithm is not always
> > > returning the values in a decreasing order?
> > >
> > > Thanks.
> > >
> > > Pedro.
> > >
> > > ----------------------------------------
> > > > From: ted.dunning@gmail.com
> > > > Date: Fri, 19 Nov 2010 13:34:19 -0800
> > > > Subject: Re: Lanczos Algorithm
> > > > To: user@mahout.apache.org
> > > >
> > > > How many non-zero elements?
> > > >
> > > > On Fri, Nov 19, 2010 at 12:34 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > > > pmjimenez1983@hotmail.com> wrote:
> > > >
> > > > >
> > > > >
> > > > > I was talking about 10^9 rows and 10^9 columns
> > > > >
> > > > > ----------------------------------------
> > > > > > From: ted.dunning@gmail.com
> > > > > > Date: Fri, 19 Nov 2010 12:07:16 -0800
> > > > > > Subject: Re: Lanczos Algorithm
> > > > > > To: user@mahout.apache.org
> > > > > >
> > > > > > On Fri, Nov 19, 2010 at 11:17 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ
> <
> > > > > > pmjimenez1983@hotmail.com> wrote:
> > > > > >
> > > > > > > In this project I would have to work with matrix of 10^9, which
> > > have a
> > > > > very
> > > > > > > sparse data.
> > > > > >
> > > > > >
> > > > > > I think you mean 10^9 rows and 10^9 columns with much fewer 10^18
> > > > > non-zero
> > > > > > elements.
> > > > > >
> > > > > > Is that correct?
> > > > > >
> > > > > > Can you say how many non-zero elements?
> > > > >
> > > > >
> > >
> > >
>
>
RE: Lanczos Algorithm
Posted by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>.
Hi Ted,
This is in the worst case. I haved asked for the amount of data and that's what somebody told me. Anyway I'm not going to make any test with that amount of data.
I'm doing my final year project and I just have to make this works with a reasonable amount of data. So I just need to know how the algorithm it's working and why I'm getting different values when I compare the results with another implementation in R.
----------------------------------------
> From: ted.dunning@gmail.com
> Date: Tue, 23 Nov 2010 14:55:00 -0800
> Subject: Re: Lanczos Algorithm
> To: user@mahout.apache.org
>
> I seriously doubt that, actually. 10^14 is a very large number.
>
> As far as I know, the record for computing an SVD of a large sparse matrix
> started with about 2-3 x 10^9 non-zero elements. You are saying that your
> problem is 100,000 times larger than this. I think that you are going to
> have to
> wait for another 15 compute speed doubling times before this becomes a
> feasible computation.
>
> On Tue, Nov 23, 2010 at 11:55 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> pmjimenez1983@hotmail.com> wrote:
>
> > Well, this is in the worst case but it could be possible.
> >
> > I'm not going to make any tests with this amount of data because for me is
> > impossible but this project is part of a bigger one and they would have
> > enough space to deal with this amount of data.
> >
> >
> > ----------------------------------------
> > > From: ted.dunning@gmail.com
> > > Date: Mon, 22 Nov 2010 14:46:20 -0800
> > > Subject: Re: Lanczos Algorithm
> > > To: user@mahout.apache.org
> > >
> > > That seems like a lot. That would mean that have 10^14 = 100 trillion
> > > nonzero elements which would take 10PB to store with one bit per non-zero
> > > element.
> > >
> > > Are there many totally zero rows?
> > >
> > > Can you estimate how many non-zero elements you have in all?
> > >
> >
Re: Lanczos Algorithm
Posted by Ted Dunning <te...@gmail.com>.
I seriously doubt that, actually. 10^14 is a very large number.
As far as I know, the record for computing an SVD of a large sparse matrix
started with about 2-3 x 10^9 non-zero elements. You are saying that your
problem is 100,000 times larger than this. I think that you are going to
have to
wait for another 15 compute speed doubling times before this becomes a
feasible computation.
On Tue, Nov 23, 2010 at 11:55 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
pmjimenez1983@hotmail.com> wrote:
> Well, this is in the worst case but it could be possible.
>
> I'm not going to make any tests with this amount of data because for me is
> impossible but this project is part of a bigger one and they would have
> enough space to deal with this amount of data.
>
>
> ----------------------------------------
> > From: ted.dunning@gmail.com
> > Date: Mon, 22 Nov 2010 14:46:20 -0800
> > Subject: Re: Lanczos Algorithm
> > To: user@mahout.apache.org
> >
> > That seems like a lot. That would mean that have 10^14 = 100 trillion
> > nonzero elements which would take 10PB to store with one bit per non-zero
> > element.
> >
> > Are there many totally zero rows?
> >
> > Can you estimate how many non-zero elements you have in all?
> >
>
RE: Lanczos Algorithm
Posted by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>.
Well, this is in the worst case but it could be possible.
I'm not going to make any tests with this amount of data because for me is impossible but this project is part of a bigger one and they would have enough space to deal with this amount of data.
----------------------------------------
> From: ted.dunning@gmail.com
> Date: Mon, 22 Nov 2010 14:46:20 -0800
> Subject: Re: Lanczos Algorithm
> To: user@mahout.apache.org
>
> That seems like a lot. That would mean that have 10^14 = 100 trillion
> nonzero elements which would take 10PB to store with one bit per non-zero
> element.
>
> Are there many totally zero rows?
>
> Can you estimate how many non-zero elements you have in all?
>
> On Mon, Nov 22, 2010 at 1:07 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> pmjimenez1983@hotmail.com> wrote:
>
> >
> > Hi Ted,
> >
> > I can't give you an exact amount but more or less it could be around 10^5
> > non-zero elements per row.
> >
> > Could you please let me know, why the lanzcos algorithm is not always
> > returning the values in a decreasing order?
> >
> > Thanks.
> >
> > Pedro.
> >
> > ----------------------------------------
> > > From: ted.dunning@gmail.com
> > > Date: Fri, 19 Nov 2010 13:34:19 -0800
> > > Subject: Re: Lanczos Algorithm
> > > To: user@mahout.apache.org
> > >
> > > How many non-zero elements?
> > >
> > > On Fri, Nov 19, 2010 at 12:34 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > > pmjimenez1983@hotmail.com> wrote:
> > >
> > > >
> > > >
> > > > I was talking about 10^9 rows and 10^9 columns
> > > >
> > > > ----------------------------------------
> > > > > From: ted.dunning@gmail.com
> > > > > Date: Fri, 19 Nov 2010 12:07:16 -0800
> > > > > Subject: Re: Lanczos Algorithm
> > > > > To: user@mahout.apache.org
> > > > >
> > > > > On Fri, Nov 19, 2010 at 11:17 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > > > > pmjimenez1983@hotmail.com> wrote:
> > > > >
> > > > > > In this project I would have to work with matrix of 10^9, which
> > have a
> > > > very
> > > > > > sparse data.
> > > > >
> > > > >
> > > > > I think you mean 10^9 rows and 10^9 columns with much fewer 10^18
> > > > non-zero
> > > > > elements.
> > > > >
> > > > > Is that correct?
> > > > >
> > > > > Can you say how many non-zero elements?
> > > >
> > > >
> >
> >
Re: Lanczos Algorithm
Posted by Ted Dunning <te...@gmail.com>.
That seems like a lot. That would mean that have 10^14 = 100 trillion
nonzero elements which would take 10PB to store with one bit per non-zero
element.
Are there many totally zero rows?
Can you estimate how many non-zero elements you have in all?
On Mon, Nov 22, 2010 at 1:07 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
pmjimenez1983@hotmail.com> wrote:
>
> Hi Ted,
>
> I can't give you an exact amount but more or less it could be around 10^5
> non-zero elements per row.
>
> Could you please let me know, why the lanzcos algorithm is not always
> returning the values in a decreasing order?
>
> Thanks.
>
> Pedro.
>
> ----------------------------------------
> > From: ted.dunning@gmail.com
> > Date: Fri, 19 Nov 2010 13:34:19 -0800
> > Subject: Re: Lanczos Algorithm
> > To: user@mahout.apache.org
> >
> > How many non-zero elements?
> >
> > On Fri, Nov 19, 2010 at 12:34 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > pmjimenez1983@hotmail.com> wrote:
> >
> > >
> > >
> > > I was talking about 10^9 rows and 10^9 columns
> > >
> > > ----------------------------------------
> > > > From: ted.dunning@gmail.com
> > > > Date: Fri, 19 Nov 2010 12:07:16 -0800
> > > > Subject: Re: Lanczos Algorithm
> > > > To: user@mahout.apache.org
> > > >
> > > > On Fri, Nov 19, 2010 at 11:17 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > > > pmjimenez1983@hotmail.com> wrote:
> > > >
> > > > > In this project I would have to work with matrix of 10^9, which
> have a
> > > very
> > > > > sparse data.
> > > >
> > > >
> > > > I think you mean 10^9 rows and 10^9 columns with much fewer 10^18
> > > non-zero
> > > > elements.
> > > >
> > > > Is that correct?
> > > >
> > > > Can you say how many non-zero elements?
> > >
> > >
>
>
RE: Lanczos Algorithm
Posted by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>.
Hi Ted,
I can't give you an exact amount but more or less it could be around 10^5 non-zero elements per row.
Could you please let me know, why the lanzcos algorithm is not always returning the values in a decreasing order?
Thanks.
Pedro.
----------------------------------------
> From: ted.dunning@gmail.com
> Date: Fri, 19 Nov 2010 13:34:19 -0800
> Subject: Re: Lanczos Algorithm
> To: user@mahout.apache.org
>
> How many non-zero elements?
>
> On Fri, Nov 19, 2010 at 12:34 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> pmjimenez1983@hotmail.com> wrote:
>
> >
> >
> > I was talking about 10^9 rows and 10^9 columns
> >
> > ----------------------------------------
> > > From: ted.dunning@gmail.com
> > > Date: Fri, 19 Nov 2010 12:07:16 -0800
> > > Subject: Re: Lanczos Algorithm
> > > To: user@mahout.apache.org
> > >
> > > On Fri, Nov 19, 2010 at 11:17 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > > pmjimenez1983@hotmail.com> wrote:
> > >
> > > > In this project I would have to work with matrix of 10^9, which have a
> > very
> > > > sparse data.
> > >
> > >
> > > I think you mean 10^9 rows and 10^9 columns with much fewer 10^18
> > non-zero
> > > elements.
> > >
> > > Is that correct?
> > >
> > > Can you say how many non-zero elements?
> >
> >
Re: Lanczos Algorithm
Posted by Ted Dunning <te...@gmail.com>.
How many non-zero elements?
On Fri, Nov 19, 2010 at 12:34 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
pmjimenez1983@hotmail.com> wrote:
>
>
> I was talking about 10^9 rows and 10^9 columns
>
> ----------------------------------------
> > From: ted.dunning@gmail.com
> > Date: Fri, 19 Nov 2010 12:07:16 -0800
> > Subject: Re: Lanczos Algorithm
> > To: user@mahout.apache.org
> >
> > On Fri, Nov 19, 2010 at 11:17 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> > pmjimenez1983@hotmail.com> wrote:
> >
> > > In this project I would have to work with matrix of 10^9, which have a
> very
> > > sparse data.
> >
> >
> > I think you mean 10^9 rows and 10^9 columns with much fewer 10^18
> non-zero
> > elements.
> >
> > Is that correct?
> >
> > Can you say how many non-zero elements?
>
>
RE: Lanczos Algorithm
Posted by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>.
I was talking about 10^9 rows and 10^9 columns
----------------------------------------
> From: ted.dunning@gmail.com
> Date: Fri, 19 Nov 2010 12:07:16 -0800
> Subject: Re: Lanczos Algorithm
> To: user@mahout.apache.org
>
> On Fri, Nov 19, 2010 at 11:17 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
> pmjimenez1983@hotmail.com> wrote:
>
> > In this project I would have to work with matrix of 10^9, which have a very
> > sparse data.
>
>
> I think you mean 10^9 rows and 10^9 columns with much fewer 10^18 non-zero
> elements.
>
> Is that correct?
>
> Can you say how many non-zero elements?
Re: Lanczos Algorithm
Posted by Ted Dunning <te...@gmail.com>.
On Fri, Nov 19, 2010 at 11:17 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ <
pmjimenez1983@hotmail.com> wrote:
> In this project I would have to work with matrix of 10^9, which have a very
> sparse data.
I think you mean 10^9 rows and 10^9 columns with much fewer 10^18 non-zero
elements.
Is that correct?
Can you say how many non-zero elements?
RE: Lanczos Algorithm
Posted by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>.
Hi,
First of all, thanks for your replies.
Fernando,
I'm using 0.4 version but I couldn't find the source code. I tried to get to the .java of this package:
org.apache.mahout.math.hadoop.decomposer.DistributedLanczosSolver
But I couln't find the file.
Ted,
In this project I would have to work with matrix of 10^9, which have a very sparse data. I'm just in the begining, trying to see how can make this work, so I don't have a real example of data to calculate. I'm just checking how the LanczosSolver works because the amount of data that I would have to manage is extremely big. I would need something like the DistributedLanzcosSolver to calculate the eigenValues and eigenVectors. Which is the data I need to work in Diffusion Maps.
K.R.
Pedro.
> From: ted.dunning@gmail.com
> Date: Fri, 19 Nov 2010 08:08:31 -0800
> Subject: Re: Lanczos Algorithm
> To: user@mahout.apache.org
>
> First, the Lanczos decomposition in Mahout is pretty new. It has passed
> some tests, but has a very reasonable probability of significant bugs,
> especially if you are giving it a matrix that does not have a strongly
> decreasing sequence of singular values.
>
> Remember also that the singular values are not necessarily returned in the
> order you might expect.
>
> Can you say a bit more about your data? How large? How sparse?
>
> This may also be of interest to you:
> https://issues.apache.org/jira/browse/MAHOUT-376
>
> I see this stochastic decomposition as the way forward to larger
> decomposition than can be done with the Lanczos solver.
>
> 2010/11/19 Fernando Fernández <fe...@gmail.com>
>
> > Hi Pedro,
> >
> > Which version of Mahout are you working with? I think
> > DistributedLanczosSolver is available from from version 0.4. I have more or
> > less the same doubts about the results as you have, so i'm willing to read
> > the answers to these questions :)
> >
> >
> > Best.
> > Fernando.
> >
> > 2010/11/19 PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>
> >
> > >
> > > Dear Mahout developers,
> > >
> > > I'm a Computer Science student from the National University of Distance
> > > Education in Spain. I'm currently developing my final year project which
> > is
> > > about Diffusion Maps.
> > >
> > > This method is used for dimensionality reduction and it uses the Lanczos
> > > algorithm during its operations. The method is already implemented in the
> > > last release version
> > > of Mahout in the LanczosSolver class but we foresee the need to use the
> > > algorithm with distributed calculations. This implementation of Diffusion
> > > Maps has to deal
> > > with extremely large matrices and the distributed calculation is critical
> > > for me.
> > >
> > > I have noticed that there is a DistributedLanzcosSolver class implemented
> > > in the Mahout library but I can’t have access to the source code because
> > it
> > > isn't in the
> > > last release version of Mahout.
> > >
> > >
> > > Could you please let me know if I could have access to the source code of
> > > this class?Also I would like to ask you about how the LanczosSolver
> > > implementation works. I have made some test between this class and other
> > > program which has been implemented in R. This program is using a library
> > > called Arpack, which also uses the Lanczos algorithm. When I calculate
> > the
> > > eigenvalues and the eigenvectors of a symmetric matrix. I haven’t the
> > same
> > > results. For example:
> > > For this matrix:
> > >
> > >
> > > 4.42282138 1.51744077 0.07690571 0.93650042 2.19609401
> > > 1.51744077 1.73849477 -0.11856149 0.76555191 1.3673608
> > > 0.07690571 -0.11856149 0.55065932 1.72163263 -0.2283693
> > > 0.93650042 0.76555191 1.72163263 0.09470345 -1.16626194
> > > 2.19609401 1.3673608 -0.2283693 -1.16626194 -0.37321311
> > > Results for R:
> > >
> > >
> > > Eigenvalues
> > >
> > > -0.6442398 1.1084103 2.3946915 6.2018925
> > >
> > > Eigenvectors [,1] [,2] [,3] [,4]
> > >
> > > [1,] -0.17050824 0.46631043 -0.010360993 0.83660453
> > > [2,] -0.06455473 -0.87762807 -0.008814402 0.40939079
> > > [3,] 0.68602882 0.04706265 -0.666429293 0.02602181
> > > [4,] -0.39567054 -0.07491643 -0.670834157 0.12161492
> > > [5,] 0.58272541 -0.06705358 0.325066897 0.34208875
> > >
> > >
> > > Results for Java:
> > >
> > >
> > > Eigenvalues
> > >
> > > 0.0 0.007869004183962289 0.023293016691817894 0.10872358093523908
> > > 0.13087002850143611
> > > I never get the same eigenvalues, I think this is because the
> > documentation
> > > of the class says:
> > > To avoid floating point overflow problems which arise in power-methods
> > like
> > > Lanczos, an initial pass is made through the input matrix to generate a
> > good
> > > starting seed vector by summing all the rows of the input matrix, and
> > > compute the trace(inputMatrixt*matrix)
> > > This latter value, being the sum of all of the singular values, is used
> > to
> > > rescale the entire matrix, effectively forcing the largest singular value
> > to
> > > be strictly
> > > less than one, and transforming floating point overflow problems into
> > > floating point underflow (ie, very small singular values will become
> > > invisible, as they will
> > > appear to be zero and the algorithm will terminate).
> > > Is it possible to return the eigenvalues to theirs original value?
> > > Eigenvectors
> > >
> > > -0.83660453 0.23122937 0.010360993 0.46631043 -0.17050824
> > > -0.40939079 0.24067227 0.008814402 -0.87762807 -0.06455473
> > > -0.02602181 0.28695718 0.666429293 0.04706265 0.68602882
> > > -0.12161492 -0.61075665 0.670834157 -0.07491643 -0.39567054
> > > -0.34208875 -0.65821099 -0.325066897 -0.06705358 0.58272541
> > >
> > >
> > >
> > > Always happens the same. I have to force the calculation of N vectors
> > (with
> > > an N x N matrix) to obtain the same values for the eigenvectors,
> > > except in the sign of some of the values, which is acceptable. I thought
> > > this implementation of the algorithm should return the eigenvectors
> > sorted
> > > but all the time I’m obtaining a vector which I don’t want to calculate
> > > between them.
> > > In the example above it’s the second one starting from the left.Why is
> > this
> > > happen?
> > >
> > > Thanks in advance.
> > >
> > > K.r.Pedro
> >
Re: Lanczos Algorithm
Posted by Ted Dunning <te...@gmail.com>.
First, the Lanczos decomposition in Mahout is pretty new. It has passed
some tests, but has a very reasonable probability of significant bugs,
especially if you are giving it a matrix that does not have a strongly
decreasing sequence of singular values.
Remember also that the singular values are not necessarily returned in the
order you might expect.
Can you say a bit more about your data? How large? How sparse?
This may also be of interest to you:
https://issues.apache.org/jira/browse/MAHOUT-376
I see this stochastic decomposition as the way forward to larger
decomposition than can be done with the Lanczos solver.
2010/11/19 Fernando Fernández <fe...@gmail.com>
> Hi Pedro,
>
> Which version of Mahout are you working with? I think
> DistributedLanczosSolver is available from from version 0.4. I have more or
> less the same doubts about the results as you have, so i'm willing to read
> the answers to these questions :)
>
>
> Best.
> Fernando.
>
> 2010/11/19 PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>
>
> >
> > Dear Mahout developers,
> >
> > I'm a Computer Science student from the National University of Distance
> > Education in Spain. I'm currently developing my final year project which
> is
> > about Diffusion Maps.
> >
> > This method is used for dimensionality reduction and it uses the Lanczos
> > algorithm during its operations. The method is already implemented in the
> > last release version
> > of Mahout in the LanczosSolver class but we foresee the need to use the
> > algorithm with distributed calculations. This implementation of Diffusion
> > Maps has to deal
> > with extremely large matrices and the distributed calculation is critical
> > for me.
> >
> > I have noticed that there is a DistributedLanzcosSolver class implemented
> > in the Mahout library but I can’t have access to the source code because
> it
> > isn't in the
> > last release version of Mahout.
> >
> >
> > Could you please let me know if I could have access to the source code of
> > this class?Also I would like to ask you about how the LanczosSolver
> > implementation works. I have made some test between this class and other
> > program which has been implemented in R. This program is using a library
> > called Arpack, which also uses the Lanczos algorithm. When I calculate
> the
> > eigenvalues and the eigenvectors of a symmetric matrix. I haven’t the
> same
> > results. For example:
> > For this matrix:
> >
> >
> > 4.42282138 1.51744077 0.07690571 0.93650042 2.19609401
> > 1.51744077 1.73849477 -0.11856149 0.76555191 1.3673608
> > 0.07690571 -0.11856149 0.55065932 1.72163263 -0.2283693
> > 0.93650042 0.76555191 1.72163263 0.09470345 -1.16626194
> > 2.19609401 1.3673608 -0.2283693 -1.16626194 -0.37321311
> > Results for R:
> >
> >
> > Eigenvalues
> >
> > -0.6442398 1.1084103 2.3946915 6.2018925
> >
> > Eigenvectors [,1] [,2] [,3] [,4]
> >
> > [1,] -0.17050824 0.46631043 -0.010360993 0.83660453
> > [2,] -0.06455473 -0.87762807 -0.008814402 0.40939079
> > [3,] 0.68602882 0.04706265 -0.666429293 0.02602181
> > [4,] -0.39567054 -0.07491643 -0.670834157 0.12161492
> > [5,] 0.58272541 -0.06705358 0.325066897 0.34208875
> >
> >
> > Results for Java:
> >
> >
> > Eigenvalues
> >
> > 0.0 0.007869004183962289 0.023293016691817894 0.10872358093523908
> > 0.13087002850143611
> > I never get the same eigenvalues, I think this is because the
> documentation
> > of the class says:
> > To avoid floating point overflow problems which arise in power-methods
> like
> > Lanczos, an initial pass is made through the input matrix to generate a
> good
> > starting seed vector by summing all the rows of the input matrix, and
> > compute the trace(inputMatrixt*matrix)
> > This latter value, being the sum of all of the singular values, is used
> to
> > rescale the entire matrix, effectively forcing the largest singular value
> to
> > be strictly
> > less than one, and transforming floating point overflow problems into
> > floating point underflow (ie, very small singular values will become
> > invisible, as they will
> > appear to be zero and the algorithm will terminate).
> > Is it possible to return the eigenvalues to theirs original value?
> > Eigenvectors
> >
> > -0.83660453 0.23122937 0.010360993 0.46631043 -0.17050824
> > -0.40939079 0.24067227 0.008814402 -0.87762807 -0.06455473
> > -0.02602181 0.28695718 0.666429293 0.04706265 0.68602882
> > -0.12161492 -0.61075665 0.670834157 -0.07491643 -0.39567054
> > -0.34208875 -0.65821099 -0.325066897 -0.06705358 0.58272541
> >
> >
> >
> > Always happens the same. I have to force the calculation of N vectors
> (with
> > an N x N matrix) to obtain the same values for the eigenvectors,
> > except in the sign of some of the values, which is acceptable. I thought
> > this implementation of the algorithm should return the eigenvectors
> sorted
> > but all the time I’m obtaining a vector which I don’t want to calculate
> > between them.
> > In the example above it’s the second one starting from the left.Why is
> this
> > happen?
> >
> > Thanks in advance.
> >
> > K.r.Pedro
>
Re: Lanczos Algorithm
Posted by Fernando Fernández <fe...@gmail.com>.
Hi Pedro,
Which version of Mahout are you working with? I think
DistributedLanczosSolver is available from from version 0.4. I have more or
less the same doubts about the results as you have, so i'm willing to read
the answers to these questions :)
Best.
Fernando.
2010/11/19 PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>
>
> Dear Mahout developers,
>
> I'm a Computer Science student from the National University of Distance
> Education in Spain. I'm currently developing my final year project which is
> about Diffusion Maps.
>
> This method is used for dimensionality reduction and it uses the Lanczos
> algorithm during its operations. The method is already implemented in the
> last release version
> of Mahout in the LanczosSolver class but we foresee the need to use the
> algorithm with distributed calculations. This implementation of Diffusion
> Maps has to deal
> with extremely large matrices and the distributed calculation is critical
> for me.
>
> I have noticed that there is a DistributedLanzcosSolver class implemented
> in the Mahout library but I can’t have access to the source code because it
> isn't in the
> last release version of Mahout.
>
>
> Could you please let me know if I could have access to the source code of
> this class?Also I would like to ask you about how the LanczosSolver
> implementation works. I have made some test between this class and other
> program which has been implemented in R. This program is using a library
> called Arpack, which also uses the Lanczos algorithm. When I calculate the
> eigenvalues and the eigenvectors of a symmetric matrix. I haven’t the same
> results. For example:
> For this matrix:
>
>
> 4.42282138 1.51744077 0.07690571 0.93650042 2.19609401
> 1.51744077 1.73849477 -0.11856149 0.76555191 1.3673608
> 0.07690571 -0.11856149 0.55065932 1.72163263 -0.2283693
> 0.93650042 0.76555191 1.72163263 0.09470345 -1.16626194
> 2.19609401 1.3673608 -0.2283693 -1.16626194 -0.37321311
> Results for R:
>
>
> Eigenvalues
>
> -0.6442398 1.1084103 2.3946915 6.2018925
>
> Eigenvectors [,1] [,2] [,3] [,4]
>
> [1,] -0.17050824 0.46631043 -0.010360993 0.83660453
> [2,] -0.06455473 -0.87762807 -0.008814402 0.40939079
> [3,] 0.68602882 0.04706265 -0.666429293 0.02602181
> [4,] -0.39567054 -0.07491643 -0.670834157 0.12161492
> [5,] 0.58272541 -0.06705358 0.325066897 0.34208875
>
>
> Results for Java:
>
>
> Eigenvalues
>
> 0.0 0.007869004183962289 0.023293016691817894 0.10872358093523908
> 0.13087002850143611
> I never get the same eigenvalues, I think this is because the documentation
> of the class says:
> To avoid floating point overflow problems which arise in power-methods like
> Lanczos, an initial pass is made through the input matrix to generate a good
> starting seed vector by summing all the rows of the input matrix, and
> compute the trace(inputMatrixt*matrix)
> This latter value, being the sum of all of the singular values, is used to
> rescale the entire matrix, effectively forcing the largest singular value to
> be strictly
> less than one, and transforming floating point overflow problems into
> floating point underflow (ie, very small singular values will become
> invisible, as they will
> appear to be zero and the algorithm will terminate).
> Is it possible to return the eigenvalues to theirs original value?
> Eigenvectors
>
> -0.83660453 0.23122937 0.010360993 0.46631043 -0.17050824
> -0.40939079 0.24067227 0.008814402 -0.87762807 -0.06455473
> -0.02602181 0.28695718 0.666429293 0.04706265 0.68602882
> -0.12161492 -0.61075665 0.670834157 -0.07491643 -0.39567054
> -0.34208875 -0.65821099 -0.325066897 -0.06705358 0.58272541
>
>
>
> Always happens the same. I have to force the calculation of N vectors (with
> an N x N matrix) to obtain the same values for the eigenvectors,
> except in the sign of some of the values, which is acceptable. I thought
> this implementation of the algorithm should return the eigenvectors sorted
> but all the time I’m obtaining a vector which I don’t want to calculate
> between them.
> In the example above it’s the second one starting from the left.Why is this
> happen?
>
> Thanks in advance.
>
> K.r.Pedro
RE: DistributedRowMatrix - FileNotFoundException
Posted by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>.
Thanks everyone for reply. I would try to do what you have suggested me.
> Date: Wed, 7 Mar 2012 21:33:16 -0800
> Subject: Re: DistributedRowMatrix - FileNotFoundException
> From: goksron@gmail.com
> To: user@mahout.apache.org
>
> In examples/bin/asf-email-examples.sh it shows how the Bayes
> classifier takes raw text and creates int/vector sequence files. You
> can get a very small subset of the Apache mail archives.
>
> Try running this example and watch the different files as the script
> makes them. The Mahout job is seq2sparse.
>
> On Wed, Mar 7, 2012 at 8:20 PM, Paritosh Ranjan <pr...@xebia.com> wrote:
> > You will have to use org.apache.hadoop.io.SequenceFile.Writer to write a
> > sequence file which can be used as a input.
> >
> > Something like,
> >
> > Writer writer = new Writer(fileSystem, conf, pathToWrite, IntWritable.class,
> > VectorWritable.class);
> > //for all IntWritable, VectorWritable pairs
> > writer.append(new IntWritable(theIntValue), new VectorWritable(theVector));
> >
> > and then use this sequence file.
> >
> >
> > On 08-03-2012 02:57, Sean Owen wrote:
> >>
> >> DistributedRowMatrix operates on IntWritable,VectorWritable in a
> >> sequence file, and it looks like you're feeding text. No, it doesn't
> >> accept some text-based format.
> >>
> >> On Wed, Mar 7, 2012 at 8:41 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ
> >> <pm...@hotmail.com> wrote:
> >>>
> >>> Sorry but I can't understand how to do it.
> >>>
> >>> I have single separated-space text file with my input matrix. To run
> >>> DistributedRowMatrix with that file I need to convert data to seqFile
> >>> format.
> >>>
> >>> How I can do this with SequenceFileInputFormat? I have tried with
> >>> InputDriver but I didn't have success.
> >>>
> >>> Thanks for your help.
> >>>
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
Re: DistributedRowMatrix - FileNotFoundException
Posted by Lance Norskog <go...@gmail.com>.
In examples/bin/asf-email-examples.sh it shows how the Bayes
classifier takes raw text and creates int/vector sequence files. You
can get a very small subset of the Apache mail archives.
Try running this example and watch the different files as the script
makes them. The Mahout job is seq2sparse.
On Wed, Mar 7, 2012 at 8:20 PM, Paritosh Ranjan <pr...@xebia.com> wrote:
> You will have to use org.apache.hadoop.io.SequenceFile.Writer to write a
> sequence file which can be used as a input.
>
> Something like,
>
> Writer writer = new Writer(fileSystem, conf, pathToWrite, IntWritable.class,
> VectorWritable.class);
> //for all IntWritable, VectorWritable pairs
> writer.append(new IntWritable(theIntValue), new VectorWritable(theVector));
>
> and then use this sequence file.
>
>
> On 08-03-2012 02:57, Sean Owen wrote:
>>
>> DistributedRowMatrix operates on IntWritable,VectorWritable in a
>> sequence file, and it looks like you're feeding text. No, it doesn't
>> accept some text-based format.
>>
>> On Wed, Mar 7, 2012 at 8:41 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ
>> <pm...@hotmail.com> wrote:
>>>
>>> Sorry but I can't understand how to do it.
>>>
>>> I have single separated-space text file with my input matrix. To run
>>> DistributedRowMatrix with that file I need to convert data to seqFile
>>> format.
>>>
>>> How I can do this with SequenceFileInputFormat? I have tried with
>>> InputDriver but I didn't have success.
>>>
>>> Thanks for your help.
>>>
>
--
Lance Norskog
goksron@gmail.com
Re: DistributedRowMatrix - FileNotFoundException
Posted by Paritosh Ranjan <pr...@xebia.com>.
You will have to use org.apache.hadoop.io.SequenceFile.Writer to write a
sequence file which can be used as a input.
Something like,
Writer writer = new Writer(fileSystem, conf, pathToWrite,
IntWritable.class, VectorWritable.class);
//for all IntWritable, VectorWritable pairs
writer.append(new IntWritable(theIntValue), new VectorWritable(theVector));
and then use this sequence file.
On 08-03-2012 02:57, Sean Owen wrote:
> DistributedRowMatrix operates on IntWritable,VectorWritable in a
> sequence file, and it looks like you're feeding text. No, it doesn't
> accept some text-based format.
>
> On Wed, Mar 7, 2012 at 8:41 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ
> <pm...@hotmail.com> wrote:
>> Sorry but I can't understand how to do it.
>>
>> I have single separated-space text file with my input matrix. To run DistributedRowMatrix with that file I need to convert data to seqFile format.
>>
>> How I can do this with SequenceFileInputFormat? I have tried with InputDriver but I didn't have success.
>>
>> Thanks for your help.
>>
Re: DistributedRowMatrix - FileNotFoundException
Posted by Sean Owen <sr...@gmail.com>.
DistributedRowMatrix operates on IntWritable,VectorWritable in a
sequence file, and it looks like you're feeding text. No, it doesn't
accept some text-based format.
On Wed, Mar 7, 2012 at 8:41 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ
<pm...@hotmail.com> wrote:
>
> Sorry but I can't understand how to do it.
>
> I have single separated-space text file with my input matrix. To run DistributedRowMatrix with that file I need to convert data to seqFile format.
>
> How I can do this with SequenceFileInputFormat? I have tried with InputDriver but I didn't have success.
>
> Thanks for your help.
>
RE: DistributedRowMatrix - FileNotFoundException
Posted by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>.
Sorry but I can't understand how to do it.
I have single separated-space text file with my input matrix. To run DistributedRowMatrix with that file I need to convert data to seqFile format.
How I can do this with SequenceFileInputFormat? I have tried with InputDriver but I didn't have success.
Thanks for your help.
> Date: Tue, 6 Mar 2012 19:31:58 +0000
> Subject: Re: DistributedRowMatrix - FileNotFoundException
> From: srowen@gmail.com
> To: user@mahout.apache.org
>
> Your input is still text though, and I assume your'e trying to use
> TextInputFormat. You can't do this as it expects an IntWritable, and
> that means it expects input as a sequence file, via
> SequenceFileInputFormat.
>
> On Tue, Mar 6, 2012 at 7:21 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ
> <pm...@hotmail.com> wrote:
> >
> > Thanks for reply.
> >
> > I was doing something wrong. I have to convert my input file to a seqFile. Now I'm trying to convert it.
> >
> > The file looks like:
> >
> > 2323.03 994.45 87.....
> > 56.45 76.21 275.1 12.456......
> > ......
> >
> > Each line represents a matrix row. And each column is separated by space.
> >
> >
> > So I executed the following command to get the seqFile
> >
> > bin/mahout seqdirectory -i /home/pedro/input -o /home/pedro/diffuse/output -c UTF-8
> >
> > And I try to run my program whith the generated file. Getting the following error:
> >
> > java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.IntWritable
> > at org.apache.mahout.math.hadoop.TransposeJob$TransposeMapper.map(TransposeJob.java:100)
> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
> > at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> > at org.apache.hadoop.mapred.Child.main(Child.java:253)
> >
> > Do I have to change the input file to another format?
> >
> > Thanks.
> >
> >
> >
> >> Date: Sun, 4 Mar 2012 17:48:56 -0800
> >> Subject: Re: DistributedRowMatrix - FileNotFoundException
> >> From: goksron@gmail.com
> >> To: user@mahout.apache.org
> >>
> >> This could be a problem with the DRM code or HDFS management. Try
> >> running it without HDFS or a Hadoop cluster, with local files and in
> >> pseudo-distributed mode. This way you can narrow the problem to one of
> >> the above.
> >>
> >> On Sat, Mar 3, 2012 at 10:13 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ
> >> <pm...@hotmail.com> wrote:
> >> >
> >> > Hi everyone!
> >> >
> >> > I'm trying to use DistributedRowMatrix in my class code but I'm getting the same error all the time: "FileNotFoundException"
> >> >
> >> > I have put a file in my hdfs directory under /user/hduser/diffuse. And I run the progam with "diffuse" as input and output directory. The code looks like:
> >> >
> >> > Configuration originalConfig = getConf();
> >> > DistributedRowMatrix matrix = new
> >> > DistributedRowMatrix(inputPath,
> >> > outputPath,
> >> > numRows,
> >> > numCols);
> >> >
> >> > JobConf conf = new JobConf(originalConfig);
> >> > matrix.configure(conf);
> >> >
> >> > DistributedRowMatrix t1 = matrix.transpose();
> >> >
> >> > 12/03/03 18:55:13 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
> >> > 12/03/03 18:55:14 INFO mapred.FileInputFormat: Total input paths to process : 7
> >> > 12/03/03 18:55:14 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost:54310/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201203031751_0007
> >> > Exception in thread "main" java.io.FileNotFoundException: File does not exist: hdfs://localhost:54310/user/hduser/diffuse/7476429391099/data
> >> > at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:525)
> >> > at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:51)
> >> > at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:211)
> >> > at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:929)
> >> > at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:921)
> >> > at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
> >> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:838)
> >> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791)
> >> > at java.security.AccessController.doPrivileged(Native Method)
> >> > at javax.security.auth.Subject.doAs(Subject.java:396)
> >> > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> >> > at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:791)
> >> > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:765)
> >> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1200)
> >> > at org.apache.mahout.math.hadoop.DistributedRowMatrix.transpose(DistributedRowMatrix.java:159)
> >> > at Distributed.MatrixTransposeJob.run(MatrixTransposeJob.java:51)
> >> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> >> > at Distributed.MatrixTransposeJob.main(MatrixTransposeJob.java:58)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >> >
> >> > What I'm doing wrong?
> >> >
> >> > Everytime I try to run the code I'm obtaining a path like this one:
> >> >
> >> > FileNotFoundException: File does not exist: hdfs://localhost:54310/user/hduser/diffuse/7476429391099/data
> >> >
> >> > Thanks a lot.
> >> >
> >> > Pedro.
> >> >
> >>
> >>
> >>
> >> --
> >> Lance Norskog
> >> goksron@gmail.com
> >
Re: DistributedRowMatrix - FileNotFoundException
Posted by Sean Owen <sr...@gmail.com>.
Your input is still text though, and I assume your'e trying to use
TextInputFormat. You can't do this as it expects an IntWritable, and
that means it expects input as a sequence file, via
SequenceFileInputFormat.
On Tue, Mar 6, 2012 at 7:21 PM, PEDRO MANUEL JIMENEZ RODRIGUEZ
<pm...@hotmail.com> wrote:
>
> Thanks for reply.
>
> I was doing something wrong. I have to convert my input file to a seqFile. Now I'm trying to convert it.
>
> The file looks like:
>
> 2323.03 994.45 87.....
> 56.45 76.21 275.1 12.456......
> ......
>
> Each line represents a matrix row. And each column is separated by space.
>
>
> So I executed the following command to get the seqFile
>
> bin/mahout seqdirectory -i /home/pedro/input -o /home/pedro/diffuse/output -c UTF-8
>
> And I try to run my program whith the generated file. Getting the following error:
>
> java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.IntWritable
> at org.apache.mahout.math.hadoop.TransposeJob$TransposeMapper.map(TransposeJob.java:100)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:253)
>
> Do I have to change the input file to another format?
>
> Thanks.
>
>
>
>> Date: Sun, 4 Mar 2012 17:48:56 -0800
>> Subject: Re: DistributedRowMatrix - FileNotFoundException
>> From: goksron@gmail.com
>> To: user@mahout.apache.org
>>
>> This could be a problem with the DRM code or HDFS management. Try
>> running it without HDFS or a Hadoop cluster, with local files and in
>> pseudo-distributed mode. This way you can narrow the problem to one of
>> the above.
>>
>> On Sat, Mar 3, 2012 at 10:13 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ
>> <pm...@hotmail.com> wrote:
>> >
>> > Hi everyone!
>> >
>> > I'm trying to use DistributedRowMatrix in my class code but I'm getting the same error all the time: "FileNotFoundException"
>> >
>> > I have put a file in my hdfs directory under /user/hduser/diffuse. And I run the progam with "diffuse" as input and output directory. The code looks like:
>> >
>> > Configuration originalConfig = getConf();
>> > DistributedRowMatrix matrix = new
>> > DistributedRowMatrix(inputPath,
>> > outputPath,
>> > numRows,
>> > numCols);
>> >
>> > JobConf conf = new JobConf(originalConfig);
>> > matrix.configure(conf);
>> >
>> > DistributedRowMatrix t1 = matrix.transpose();
>> >
>> > 12/03/03 18:55:13 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
>> > 12/03/03 18:55:14 INFO mapred.FileInputFormat: Total input paths to process : 7
>> > 12/03/03 18:55:14 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost:54310/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201203031751_0007
>> > Exception in thread "main" java.io.FileNotFoundException: File does not exist: hdfs://localhost:54310/user/hduser/diffuse/7476429391099/data
>> > at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:525)
>> > at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:51)
>> > at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:211)
>> > at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:929)
>> > at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:921)
>> > at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
>> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:838)
>> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791)
>> > at java.security.AccessController.doPrivileged(Native Method)
>> > at javax.security.auth.Subject.doAs(Subject.java:396)
>> > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>> > at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:791)
>> > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:765)
>> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1200)
>> > at org.apache.mahout.math.hadoop.DistributedRowMatrix.transpose(DistributedRowMatrix.java:159)
>> > at Distributed.MatrixTransposeJob.run(MatrixTransposeJob.java:51)
>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> > at Distributed.MatrixTransposeJob.main(MatrixTransposeJob.java:58)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> >
>> > What I'm doing wrong?
>> >
>> > Everytime I try to run the code I'm obtaining a path like this one:
>> >
>> > FileNotFoundException: File does not exist: hdfs://localhost:54310/user/hduser/diffuse/7476429391099/data
>> >
>> > Thanks a lot.
>> >
>> > Pedro.
>> >
>>
>>
>>
>> --
>> Lance Norskog
>> goksron@gmail.com
>
RE: DistributedRowMatrix - FileNotFoundException
Posted by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>.
Thanks for reply.
I was doing something wrong. I have to convert my input file to a seqFile. Now I'm trying to convert it.
The file looks like:
2323.03 994.45 87.....
56.45 76.21 275.1 12.456......
......
Each line represents a matrix row. And each column is separated by space.
So I executed the following command to get the seqFile
bin/mahout seqdirectory -i /home/pedro/input -o /home/pedro/diffuse/output -c UTF-8
And I try to run my program whith the generated file. Getting the following error:
java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.IntWritable
at org.apache.mahout.math.hadoop.TransposeJob$TransposeMapper.map(TransposeJob.java:100)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Do I have to change the input file to another format?
Thanks.
> Date: Sun, 4 Mar 2012 17:48:56 -0800
> Subject: Re: DistributedRowMatrix - FileNotFoundException
> From: goksron@gmail.com
> To: user@mahout.apache.org
>
> This could be a problem with the DRM code or HDFS management. Try
> running it without HDFS or a Hadoop cluster, with local files and in
> pseudo-distributed mode. This way you can narrow the problem to one of
> the above.
>
> On Sat, Mar 3, 2012 at 10:13 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ
> <pm...@hotmail.com> wrote:
> >
> > Hi everyone!
> >
> > I'm trying to use DistributedRowMatrix in my class code but I'm getting the same error all the time: "FileNotFoundException"
> >
> > I have put a file in my hdfs directory under /user/hduser/diffuse. And I run the progam with "diffuse" as input and output directory. The code looks like:
> >
> > Configuration originalConfig = getConf();
> > DistributedRowMatrix matrix = new
> > DistributedRowMatrix(inputPath,
> > outputPath,
> > numRows,
> > numCols);
> >
> > JobConf conf = new JobConf(originalConfig);
> > matrix.configure(conf);
> >
> > DistributedRowMatrix t1 = matrix.transpose();
> >
> > 12/03/03 18:55:13 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
> > 12/03/03 18:55:14 INFO mapred.FileInputFormat: Total input paths to process : 7
> > 12/03/03 18:55:14 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost:54310/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201203031751_0007
> > Exception in thread "main" java.io.FileNotFoundException: File does not exist: hdfs://localhost:54310/user/hduser/diffuse/7476429391099/data
> > at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:525)
> > at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:51)
> > at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:211)
> > at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:929)
> > at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:921)
> > at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:838)
> > at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> > at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:791)
> > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:765)
> > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1200)
> > at org.apache.mahout.math.hadoop.DistributedRowMatrix.transpose(DistributedRowMatrix.java:159)
> > at Distributed.MatrixTransposeJob.run(MatrixTransposeJob.java:51)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> > at Distributed.MatrixTransposeJob.main(MatrixTransposeJob.java:58)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >
> > What I'm doing wrong?
> >
> > Everytime I try to run the code I'm obtaining a path like this one:
> >
> > FileNotFoundException: File does not exist: hdfs://localhost:54310/user/hduser/diffuse/7476429391099/data
> >
> > Thanks a lot.
> >
> > Pedro.
> >
>
>
>
> --
> Lance Norskog
> goksron@gmail.com
Re: DistributedRowMatrix - FileNotFoundException
Posted by Lance Norskog <go...@gmail.com>.
This could be a problem with the DRM code or HDFS management. Try
running it without HDFS or a Hadoop cluster, with local files and in
pseudo-distributed mode. This way you can narrow the problem to one of
the above.
On Sat, Mar 3, 2012 at 10:13 AM, PEDRO MANUEL JIMENEZ RODRIGUEZ
<pm...@hotmail.com> wrote:
>
> Hi everyone!
>
> I'm trying to use DistributedRowMatrix in my class code but I'm getting the same error all the time: "FileNotFoundException"
>
> I have put a file in my hdfs directory under /user/hduser/diffuse. And I run the progam with "diffuse" as input and output directory. The code looks like:
>
> Configuration originalConfig = getConf();
> DistributedRowMatrix matrix = new
> DistributedRowMatrix(inputPath,
> outputPath,
> numRows,
> numCols);
>
> JobConf conf = new JobConf(originalConfig);
> matrix.configure(conf);
>
> DistributedRowMatrix t1 = matrix.transpose();
>
> 12/03/03 18:55:13 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
> 12/03/03 18:55:14 INFO mapred.FileInputFormat: Total input paths to process : 7
> 12/03/03 18:55:14 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost:54310/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201203031751_0007
> Exception in thread "main" java.io.FileNotFoundException: File does not exist: hdfs://localhost:54310/user/hduser/diffuse/7476429391099/data
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:525)
> at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:51)
> at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:211)
> at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:929)
> at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:921)
> at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:838)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:791)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:765)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1200)
> at org.apache.mahout.math.hadoop.DistributedRowMatrix.transpose(DistributedRowMatrix.java:159)
> at Distributed.MatrixTransposeJob.run(MatrixTransposeJob.java:51)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at Distributed.MatrixTransposeJob.main(MatrixTransposeJob.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> What I'm doing wrong?
>
> Everytime I try to run the code I'm obtaining a path like this one:
>
> FileNotFoundException: File does not exist: hdfs://localhost:54310/user/hduser/diffuse/7476429391099/data
>
> Thanks a lot.
>
> Pedro.
>
--
Lance Norskog
goksron@gmail.com
DistributedRowMatrix - FileNotFoundException
Posted by PEDRO MANUEL JIMENEZ RODRIGUEZ <pm...@hotmail.com>.
Hi everyone!
I'm trying to use DistributedRowMatrix in my class code but I'm getting the same error all the time: "FileNotFoundException"
I have put a file in my hdfs directory under /user/hduser/diffuse. And I run the progam with "diffuse" as input and output directory. The code looks like:
Configuration originalConfig = getConf();
DistributedRowMatrix matrix = new
DistributedRowMatrix(inputPath,
outputPath,
numRows,
numCols);
JobConf conf = new JobConf(originalConfig);
matrix.configure(conf);
DistributedRowMatrix t1 = matrix.transpose();
12/03/03 18:55:13 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
12/03/03 18:55:14 INFO mapred.FileInputFormat: Total input paths to process : 7
12/03/03 18:55:14 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost:54310/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201203031751_0007
Exception in thread "main" java.io.FileNotFoundException: File does not exist: hdfs://localhost:54310/user/hduser/diffuse/7476429391099/data
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:525)
at org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:51)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:211)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:929)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:921)
at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:838)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:791)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:791)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:765)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1200)
at org.apache.mahout.math.hadoop.DistributedRowMatrix.transpose(DistributedRowMatrix.java:159)
at Distributed.MatrixTransposeJob.run(MatrixTransposeJob.java:51)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at Distributed.MatrixTransposeJob.main(MatrixTransposeJob.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
What I'm doing wrong?
Everytime I try to run the code I'm obtaining a path like this one:
FileNotFoundException: File does not exist: hdfs://localhost:54310/user/hduser/diffuse/7476429391099/data
Thanks a lot.
Pedro.