You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Stuti Awasthi <st...@hcl.com> on 2013/07/31 14:09:01 UTC
How to SSVD output to generate Clusters
Hi All,
I wanted to group the documents with same context but which belongs to one single domain together. I have tried KMeans and LDA provided in Mahout to perform the clustering but the groups which are generated are not very good. Hence I thought to use LSA to indentify the context related to the word and then perform the Clustering.
I am able to run SSVD of Mahout and generated 3 files : Sigma,U,V as output of SSVD.
I am not sure how to use the output of SSVD to fed to the Clustering Algorithm so that we can generate the clusters of the documents which might be talking about same context.
Any pointers how can I achieve this ?
Regards
Stuti Awasthi
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
Re: How to SSVD output to generate Clusters
Posted by Ted Dunning <te...@gmail.com>.
Use the U matrix for clustering.
Note that you are replicating what the spectral clustering already does.
On Wed, Jul 31, 2013 at 5:09 AM, Stuti Awasthi <st...@hcl.com> wrote:
> Hi All,
>
> I wanted to group the documents with same context but which belongs to one
> single domain together. I have tried KMeans and LDA provided in Mahout to
> perform the clustering but the groups which are generated are not very
> good. Hence I thought to use LSA to indentify the context related to the
> word and then perform the Clustering.
>
> I am able to run SSVD of Mahout and generated 3 files : Sigma,U,V as
> output of SSVD.
> I am not sure how to use the output of SSVD to fed to the Clustering
> Algorithm so that we can generate the clusters of the documents which might
> be talking about same context.
>
> Any pointers how can I achieve this ?
>
> Regards
> Stuti Awasthi
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
RE: How to SSVD output to generate Clusters
Posted by Stuti Awasthi <st...@hcl.com>.
Hey Sunil,
Initially I was working with Mahout-0.7 distro but when faced this error and got to know that issue is resolved in Mahout 0.8 , hence I downloaded Mahout 0.8, copied the " mahout-core-0.8-job.jar " inside the same directory of Mahout 0.7.
But in Hadoop command Im using " mahout-core-0.8-job.jar " to call " org.apache.mahout.math.hadoop.stochasticsvd.SSVDCli " class.
I believe "mahout-core-0.8-job.jar" is sufficient to run/test the updated code as SSVDCLI class is packaged under this jar.
Thanks
Stuti Awasthi
-----Original Message-----
From: Suneel Marthi [mailto:suneel_marthi@yahoo.com]
Sent: Wednesday, August 07, 2013 2:49 PM
To: user@mahout.apache.org
Subject: Re: How to SSVD output to generate Clusters
R u working off of Mahout 0.8 or 0.7?
"hadoop jar mahout-distribution-0.7/mahout-core-0.8-job.jar" ????
________________________________
From: Stuti Awasthi <st...@hcl.com>
To: "user@mahout.apache.org" <us...@mahout.apache.org>
Sent: Wednesday, August 7, 2013 5:12 AM
Subject: RE: How to SSVD output to generate Clusters
I have not used -q option while running ssvd. Here are the commands which I run :
Seq2sparse :
mahout-distribution-0.7/bin/mahout seq2sparse -i /stuti/SSVD/data-seq -o /stuti/SSVD/data-vectors -s 5 -ml 50 -nv -ng 3 -n 2 -x 70
SSVD
hadoop jar mahout-distribution-0.7/mahout-core-0.8-job.jar org.apache.mahout.math.hadoop.stochasticsvd.SSVDCli -i /stuti/SSVD/data-vectors/tf-vectors -o /stuti/SSVD/Output -k 90 -U true -V true --reduceTasks 1
Thanks
Stuti Awasthi
-----Original Message-----
From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
Sent: Wednesday, August 07, 2013 2:14 PM
To: user@mahout.apache.org
Subject: RE: How to SSVD output to generate Clusters
Thanks Stuti. Yes it looks like it is not there. Let me run a test. One
question . Did you use -q 0 or 1 or sometni g elw se?
On Aug 7, 2013 12:18 AM, "Stuti Awasthi" <st...@hcl.com> wrote:
> Hey Dmitriy,
>
> Sorry for replying late.
> I have re-run the steps to generate U. Here are the output:
>
> 1. "seq2sparse" Output: Contains Named Vector
> Key: /Description_10: Value:
> /Description_10:{554:1.0,54:1.0,514:1.0,322:1.0,247:1.0,91:1.0,127:1.0
> ,456:1.0,480:1.0,713:1.0,674:1.0,117:1.0,461:1.0,595:2.0,446:1.0,296:2
> .0}
>
> 2. "ssvd" Output of U: Not Contained NamedVector
> Key: /Description_10: Value:
> {0:0.010564205396743468,1:-0.01989403962804719,2:0.015640314765729225,3:0.04031717183780774,4:-0.03325995251075869,5:0.0294201152018514,6:
>
> 0.03834130611889856,7:-0.008686421005328312,8:-0.06164515883823538,9:0
> .03752875772953153,10:0.04739786931946798,11:-0.07912744917669134,12:0
> .020078421275704143,13:-0.0
>
> 4017504785907734,14:0.012539132502559502,15:0.0733073647645918,16:-0.0
> 2111033727307056,17:0.0799478317610193,18:-0.08481960414593219,19:-0.0
> 6987848875856222,20:0.03693
>
> 2920091059446,21:-0.06949180571421532,22:-0.03447267994522256,23:-0.07
> 104196347181493,24:0.022262180555421562,25:-0.0485632586340187,26:-0.0
> 5380823388650383,27:0.09299
>
> 533887785207,28:0.0019344239524856396,29:0.002936116541403362,30:-0.07
> 249587007236825,31:0.0016026176038041033,32:-0.0711115256224166,33:0.0
> 6603931206284432,34:0.01922
>
> 6806201249697,35:0.13972781245330326,36:0.0787696939450401,37:0.070653
> 56340476747,38:0.08437107545490818,39:0.06381670380272558,40:0.0464059
> 64753673735,41:0.0601332388
>
> 594578,42:-0.12996454299711707,43:0.10779361589915878,44:-0.0652470275
> 4474347,45:0.014785171162887613,46:-0.036630574690084586,47:-0.1506665
> 6149902793,48:0.16190482591
> 405958,49:-0.00869851116149916}
>
> As you said that propagation of keys to keys or/and names to names
> should happen . Any idea what's going wrong or if there is a mistake from my side ?
>
> Thanks
> Stuti Awasthi
>
> -----Original Message-----
> From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
> Sent: Friday, August 02, 2013 11:39 PM
> To: user@mahout.apache.org
> Subject: Re: How to SSVD output to generate Clusters
>
> by eyeballing the code, i think i don' t see a problem. if rows of A
> are named values, then row of U (or U*Sigma or U*Sigma^1/2) would also
> retain names from values of rows of A. Output would not contain
> NamedVector values for rows that were not NamedVector values
> themselves in the input. Does seq2sparse output create NamedVectors as values ?
>
> Note that if what you want is to have *keys* from from seq2sparse
> (such as
> 'file_1') propagated to name of named vector value in U, that much is
> not happening. The algorithm propagates keys to keys or/and names to
> names (but not any other combinations of those).
>
> -d
>
>
> On Fri, Aug 2, 2013 at 10:42 AM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
> > it should. i worked on the issue and last time it was checked it was
> > still working with name propagation. if not, then it is a bug
> >
> >
> > On Fri, Aug 2, 2013 at 3:33 AM, Stuti Awasthi <stutiawasthi@hcl.com
> >wrote:
> >
> >> Hey Ted,
> >>
> >> As suggested, I tried SSVD with Mahout 0.8. I think the issue of
> >> NamedVector not propagating to the output U ,still persists.
> >> Here is what I have done :
> >>
> >> 1. Created featureVector using "seq2sparse" with -nv option.
> >> Checked the output, named vector created.
> >> 2. Provided this featureVector to "ssvd" with params " -k 100 -U
> >> true -V true". After execution, 3 output got generated namely :
> >> sigma, U, V 3. I dumped "U" to check the output if it contained namedVectors:
> >>
> >> mahout-distribution-0.7/bin/mahout seqdumper -i
> >> /stuti/SSVD/Output/U
> >> | more
> >> Output:
> >> Key: /File_1: Value:
> >> {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.
> >> ....}
> >>
> >> I did not see the NamedVector getting created in the output of ssvd.
> >> Please point out if I have missed any step in between.
> >>
> >> As I wanted to perform the Clustering, I took the output of "U" and
> >> generated the NamedVector with custom code. The output looks like this :
> >> Key: /File_1: Value:
> >> /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.03343115
> >> 00
> >> 858222,.....}
> >>
> >> Then I fed this namedvector file to KMeans to generate 10 Clusters.
> >> In this I have used Random Centroid selection with KMeans.
> >> Finally I dumped the ClusterOutput as :
> >> <ClusterId>,<DocumentID1>,<DocumentId2>.....
> >>
> >> Please let me know if I have performed any mistake in the end to
> >> end execution as well Im not sure Why SSVD output is not generating
> >> the named vectors as the issue id fixed..
> >>
> >> Please suggest
> >>
> >> Regards
> >> Stuti Awasthi
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Ted Dunning [mailto:ted.dunning@gmail.com]
> >> Sent: Thursday, August 01, 2013 8:37 PM
> >> To: user@mahout.apache.org
> >> Subject: Re: How to SSVD output to generate Clusters
> >>
> >> On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi
> >> <st...@hcl.com>
> >> wrote:
> >>
> >> > I think there is a problem because of NamedVector as after some
> >> > search I get this Jira.
> >> > https://issues.apache.org/jira/browse/MAHOUT-1067
> >> >
> >>
> >> Note also that this bug is fixed in 0.8
> >>
> >>
> >> ::DISCLAIMER::
> >>
> >> -------------------------------------------------------------------
> >> --
> >> -------------------------------------------------------------------
> >> --
> >> ----------
> >>
> >> The contents of this e-mail and any attachment(s) are confidential
> >> and intended for the named recipient(s) only.
> >> E-mail transmission is not guaranteed to be secure or error-free as
> >> information could be intercepted, corrupted, lost, destroyed,
> >> arrive late or incomplete, or may contain viruses in transmission.
> >> The e mail and its contents (with or without referred errors) shall
> >> therefore not attach any liability on the originator or HCL or its
> >> affiliates.
> >> Views or opinions, if any, presented in this email are solely those
> >> of the author and may not necessarily reflect the views or opinions
> >> of HCL or its affiliates. Any form of reproduction, dissemination,
> >> copying, disclosure, modification, distribution and / or
> >> publication of this message without the prior written consent of
> >> authorized representative of HCL is strictly prohibited. If you
> >> have received this email in error please delete it and notify the
> >> sender immediately.
> >> Before opening any email and/or attachments, please check them for
> >> viruses and other defects.
> >>
> >>
> >> -------------------------------------------------------------------
> >> --
> >> -------------------------------------------------------------------
> >> --
> >> ----------
> >>
> >
> >
>
Re: How to SSVD output to generate Clusters
Posted by Suneel Marthi <su...@yahoo.com>.
R u working off of Mahout 0.8 or 0.7?
"hadoop jar mahout-distribution-0.7/mahout-core-0.8-job.jar" ????
________________________________
From: Stuti Awasthi <st...@hcl.com>
To: "user@mahout.apache.org" <us...@mahout.apache.org>
Sent: Wednesday, August 7, 2013 5:12 AM
Subject: RE: How to SSVD output to generate Clusters
I have not used -q option while running ssvd. Here are the commands which I run :
Seq2sparse :
mahout-distribution-0.7/bin/mahout seq2sparse -i /stuti/SSVD/data-seq -o /stuti/SSVD/data-vectors -s 5 -ml 50 -nv -ng 3 -n 2 -x 70
SSVD
hadoop jar mahout-distribution-0.7/mahout-core-0.8-job.jar org.apache.mahout.math.hadoop.stochasticsvd.SSVDCli -i /stuti/SSVD/data-vectors/tf-vectors -o /stuti/SSVD/Output -k 90 -U true -V true --reduceTasks 1
Thanks
Stuti Awasthi
-----Original Message-----
From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
Sent: Wednesday, August 07, 2013 2:14 PM
To: user@mahout.apache.org
Subject: RE: How to SSVD output to generate Clusters
Thanks Stuti. Yes it looks like it is not there. Let me run a test. One
question . Did you use -q 0 or 1 or sometni g elw se?
On Aug 7, 2013 12:18 AM, "Stuti Awasthi" <st...@hcl.com> wrote:
> Hey Dmitriy,
>
> Sorry for replying late.
> I have re-run the steps to generate U. Here are the output:
>
> 1. "seq2sparse" Output: Contains Named Vector
> Key: /Description_10: Value:
> /Description_10:{554:1.0,54:1.0,514:1.0,322:1.0,247:1.0,91:1.0,127:1.0
> ,456:1.0,480:1.0,713:1.0,674:1.0,117:1.0,461:1.0,595:2.0,446:1.0,296:2
> .0}
>
> 2. "ssvd" Output of U: Not Contained NamedVector
> Key: /Description_10: Value:
> {0:0.010564205396743468,1:-0.01989403962804719,2:0.015640314765729225,3:0.04031717183780774,4:-0.03325995251075869,5:0.0294201152018514,6:
>
> 0.03834130611889856,7:-0.008686421005328312,8:-0.06164515883823538,9:0
> .03752875772953153,10:0.04739786931946798,11:-0.07912744917669134,12:0
> .020078421275704143,13:-0.0
>
> 4017504785907734,14:0.012539132502559502,15:0.0733073647645918,16:-0.0
> 2111033727307056,17:0.0799478317610193,18:-0.08481960414593219,19:-0.0
> 6987848875856222,20:0.03693
>
> 2920091059446,21:-0.06949180571421532,22:-0.03447267994522256,23:-0.07
> 104196347181493,24:0.022262180555421562,25:-0.0485632586340187,26:-0.0
> 5380823388650383,27:0.09299
>
> 533887785207,28:0.0019344239524856396,29:0.002936116541403362,30:-0.07
> 249587007236825,31:0.0016026176038041033,32:-0.0711115256224166,33:0.0
> 6603931206284432,34:0.01922
>
> 6806201249697,35:0.13972781245330326,36:0.0787696939450401,37:0.070653
> 56340476747,38:0.08437107545490818,39:0.06381670380272558,40:0.0464059
> 64753673735,41:0.0601332388
>
> 594578,42:-0.12996454299711707,43:0.10779361589915878,44:-0.0652470275
> 4474347,45:0.014785171162887613,46:-0.036630574690084586,47:-0.1506665
> 6149902793,48:0.16190482591
> 405958,49:-0.00869851116149916}
>
> As you said that propagation of keys to keys or/and names to names
> should happen . Any idea what's going wrong or if there is a mistake from my side ?
>
> Thanks
> Stuti Awasthi
>
> -----Original Message-----
> From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
> Sent: Friday, August 02, 2013 11:39 PM
> To: user@mahout.apache.org
> Subject: Re: How to SSVD output to generate Clusters
>
> by eyeballing the code, i think i don' t see a problem. if rows of A
> are named values, then row of U (or U*Sigma or U*Sigma^1/2) would also
> retain names from values of rows of A. Output would not contain
> NamedVector values for rows that were not NamedVector values
> themselves in the input. Does seq2sparse output create NamedVectors as values ?
>
> Note that if what you want is to have *keys* from from seq2sparse
> (such as
> 'file_1') propagated to name of named vector value in U, that much is
> not happening. The algorithm propagates keys to keys or/and names to
> names (but not any other combinations of those).
>
> -d
>
>
> On Fri, Aug 2, 2013 at 10:42 AM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
> > it should. i worked on the issue and last time it was checked it was
> > still working with name propagation. if not, then it is a bug
> >
> >
> > On Fri, Aug 2, 2013 at 3:33 AM, Stuti Awasthi <stutiawasthi@hcl.com
> >wrote:
> >
> >> Hey Ted,
> >>
> >> As suggested, I tried SSVD with Mahout 0.8. I think the issue of
> >> NamedVector not propagating to the output U ,still persists.
> >> Here is what I have done :
> >>
> >> 1. Created featureVector using "seq2sparse" with -nv option.
> >> Checked the output, named vector created.
> >> 2. Provided this featureVector to "ssvd" with params " -k 100 -U
> >> true -V true". After execution, 3 output got generated namely :
> >> sigma, U, V 3. I dumped "U" to check the output if it contained namedVectors:
> >>
> >> mahout-distribution-0.7/bin/mahout seqdumper -i
> >> /stuti/SSVD/Output/U
> >> | more
> >> Output:
> >> Key: /File_1: Value:
> >> {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.
> >> ....}
> >>
> >> I did not see the NamedVector getting created in the output of ssvd.
> >> Please point out if I have missed any step in between.
> >>
> >> As I wanted to perform the Clustering, I took the output of "U" and
> >> generated the NamedVector with custom code. The output looks like this :
> >> Key: /File_1: Value:
> >> /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.03343115
> >> 00
> >> 858222,.....}
> >>
> >> Then I fed this namedvector file to KMeans to generate 10 Clusters.
> >> In this I have used Random Centroid selection with KMeans.
> >> Finally I dumped the ClusterOutput as :
> >> <ClusterId>,<DocumentID1>,<DocumentId2>.....
> >>
> >> Please let me know if I have performed any mistake in the end to
> >> end execution as well Im not sure Why SSVD output is not generating
> >> the named vectors as the issue id fixed..
> >>
> >> Please suggest
> >>
> >> Regards
> >> Stuti Awasthi
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Ted Dunning [mailto:ted.dunning@gmail.com]
> >> Sent: Thursday, August 01, 2013 8:37 PM
> >> To: user@mahout.apache.org
> >> Subject: Re: How to SSVD output to generate Clusters
> >>
> >> On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi
> >> <st...@hcl.com>
> >> wrote:
> >>
> >> > I think there is a problem because of NamedVector as after some
> >> > search I get this Jira.
> >> > https://issues.apache.org/jira/browse/MAHOUT-1067
> >> >
> >>
> >> Note also that this bug is fixed in 0.8
> >>
> >>
> >> ::DISCLAIMER::
> >>
> >> -------------------------------------------------------------------
> >> --
> >> -------------------------------------------------------------------
> >> --
> >> ----------
> >>
> >> The contents of this e-mail and any attachment(s) are confidential
> >> and intended for the named recipient(s) only.
> >> E-mail transmission is not guaranteed to be secure or error-free as
> >> information could be intercepted, corrupted, lost, destroyed,
> >> arrive late or incomplete, or may contain viruses in transmission.
> >> The e mail and its contents (with or without referred errors) shall
> >> therefore not attach any liability on the originator or HCL or its
> >> affiliates.
> >> Views or opinions, if any, presented in this email are solely those
> >> of the author and may not necessarily reflect the views or opinions
> >> of HCL or its affiliates. Any form of reproduction, dissemination,
> >> copying, disclosure, modification, distribution and / or
> >> publication of this message without the prior written consent of
> >> authorized representative of HCL is strictly prohibited. If you
> >> have received this email in error please delete it and notify the
> >> sender immediately.
> >> Before opening any email and/or attachments, please check them for
> >> viruses and other defects.
> >>
> >>
> >> -------------------------------------------------------------------
> >> --
> >> -------------------------------------------------------------------
> >> --
> >> ----------
> >>
> >
> >
>
Re: How to SSVD output to generate Clusters
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
On Wed, Aug 7, 2013 at 10:44 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> @Stuti:
>
> Ok, so I added additional tests (there was a small bug in the test) and
> actually in local tests the name keys and named vector names are both
> propagating on all execution paths. So i cannot confirm problem of
> no-propagation. (this is on 0.9 trunk).
>
This test mod is in MAHOUT-1306, already pushed to trunk.
I know this probably doesn't resolve your particular situation but I don't
see a problem with SSVD itself wrt name propagation --ether key or named
vector values. I know Pat was doing this successfully.Let me know if you
find something.
> > Sent: Friday, August 02, 2013 11:39 PM
>>>>> > To: user@mahout.apache.org
>>>>> > Subject: Re: How to SSVD output to generate Clusters
>>>>> >
>>>>> > by eyeballing the code, i think i don' t see a problem. if rows of A
>>>>> > are named values, then row of U (or U*Sigma or U*Sigma^1/2) would
>>>>> also
>>>>> > retain names from values of rows of A. Output would not contain
>>>>> > NamedVector values for rows that were not NamedVector values
>>>>> > themselves in the input. Does seq2sparse output create NamedVectors
>>>>> as values ?
>>>>> >
>>>>> > Note that if what you want is to have *keys* from from seq2sparse
>>>>> > (such as
>>>>> > 'file_1') propagated to name of named vector value in U, that much is
>>>>> > not happening. The algorithm propagates keys to keys or/and names to
>>>>> > names (but not any other combinations of those).
>>>>> >
>>>>> > -d
>>>>> >
>>>>> >
>>>>> > On Fri, Aug 2, 2013 at 10:42 AM, Dmitriy Lyubimov <dlieu.7@gmail.com
>>>>> >
>>>>> > wrote:
>>>>> >
>>>>> > > it should. i worked on the issue and last time it was checked it
>>>>> was
>>>>> > > still working with name propagation. if not, then it is a bug
>>>>> > >
>>>>> > >
>>>>> > > On Fri, Aug 2, 2013 at 3:33 AM, Stuti Awasthi <
>>>>> stutiawasthi@hcl.com
>>>>> > >wrote:
>>>>> > >
>>>>> > >> Hey Ted,
>>>>> > >>
>>>>> > >> As suggested, I tried SSVD with Mahout 0.8. I think the issue of
>>>>> > >> NamedVector not propagating to the output U ,still persists.
>>>>> > >> Here is what I have done :
>>>>> > >>
>>>>> > >> 1. Created featureVector using "seq2sparse" with -nv option.
>>>>> > >> Checked the output, named vector created.
>>>>> > >> 2. Provided this featureVector to "ssvd" with params " -k 100 -U
>>>>> > >> true -V true". After execution, 3 output got generated namely :
>>>>> > >> sigma, U, V 3. I dumped "U" to check the output if it contained
>>>>> namedVectors:
>>>>> > >>
>>>>> > >> mahout-distribution-0.7/bin/mahout seqdumper -i
>>>>> > >> /stuti/SSVD/Output/U
>>>>> > >> | more
>>>>> > >> Output:
>>>>> > >> Key: /File_1: Value:
>>>>> > >>
>>>>> {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.
>>>>> > >> ....}
>>>>> > >>
>>>>> > >> I did not see the NamedVector getting created in the output of
>>>>> ssvd.
>>>>> > >> Please point out if I have missed any step in between.
>>>>> > >>
>>>>> > >> As I wanted to perform the Clustering, I took the output of "U"
>>>>> and
>>>>> > >> generated the NamedVector with custom code. The output looks like
>>>>> this :
>>>>> > >> Key: /File_1: Value:
>>>>> > >>
>>>>> /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.03343115
>>>>> > >> 00
>>>>> > >> 858222,.....}
>>>>> > >>
>>>>> > >> Then I fed this namedvector file to KMeans to generate 10
>>>>> Clusters.
>>>>> > >> In this I have used Random Centroid selection with KMeans.
>>>>> > >> Finally I dumped the ClusterOutput as :
>>>>> > >> <ClusterId>,<DocumentID1>,<DocumentId2>.....
>>>>> > >>
>>>>> > >> Please let me know if I have performed any mistake in the end to
>>>>> > >> end execution as well Im not sure Why SSVD output is not
>>>>> generating
>>>>> > >> the named vectors as the issue id fixed..
>>>>> > >>
>>>>> > >> Please suggest
>>>>> > >>
>>>>> > >> Regards
>>>>> > >> Stuti Awasthi
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> > >> -----Original Message-----
>>>>> > >> From: Ted Dunning [mailto:ted.dunning@gmail.com]
>>>>> > >> Sent: Thursday, August 01, 2013 8:37 PM
>>>>> > >> To: user@mahout.apache.org
>>>>> > >> Subject: Re: How to SSVD output to generate Clusters
>>>>> > >>
>>>>> > >> On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi
>>>>> > >> <st...@hcl.com>
>>>>> > >> wrote:
>>>>> > >>
>>>>> > >> > I think there is a problem because of NamedVector as after some
>>>>> > >> > search I get this Jira.
>>>>> > >> > https://issues.apache.org/jira/browse/MAHOUT-1067
>>>>> > >> >
>>>>> > >>
>>>>> > >> Note also that this bug is fixed in 0.8
>>>>> > >>
>>>>> > >>
>>>>> > >> ::DISCLAIMER::
>>>>> > >>
>>>>> > >>
>>>>> -------------------------------------------------------------------
>>>>> > >> --
>>>>> > >>
>>>>> -------------------------------------------------------------------
>>>>> > >> --
>>>>> > >> ----------
>>>>> > >>
>>>>> > >> The contents of this e-mail and any attachment(s) are confidential
>>>>> > >> and intended for the named recipient(s) only.
>>>>> > >> E-mail transmission is not guaranteed to be secure or error-free
>>>>> as
>>>>> > >> information could be intercepted, corrupted, lost, destroyed,
>>>>> > >> arrive late or incomplete, or may contain viruses in transmission.
>>>>> > >> The e mail and its contents (with or without referred errors)
>>>>> shall
>>>>> > >> therefore not attach any liability on the originator or HCL or its
>>>>> > >> affiliates.
>>>>> > >> Views or opinions, if any, presented in this email are solely
>>>>> those
>>>>> > >> of the author and may not necessarily reflect the views or
>>>>> opinions
>>>>> > >> of HCL or its affiliates. Any form of reproduction, dissemination,
>>>>> > >> copying, disclosure, modification, distribution and / or
>>>>> > >> publication of this message without the prior written consent of
>>>>> > >> authorized representative of HCL is strictly prohibited. If you
>>>>> > >> have received this email in error please delete it and notify the
>>>>> > >> sender immediately.
>>>>> > >> Before opening any email and/or attachments, please check them for
>>>>> > >> viruses and other defects.
>>>>> > >>
>>>>> > >>
>>>>> > >>
>>>>> -------------------------------------------------------------------
>>>>> > >> --
>>>>> > >>
>>>>> -------------------------------------------------------------------
>>>>> > >> --
>>>>> > >> ----------
>>>>> > >>
>>>>> > >
>>>>> > >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
Re: How to SSVD output to generate Clusters
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
@Stuti:
Ok, so I added additional tests (there was a small bug in the test) and
actually in local tests the name keys and named vector names are both
propagating on all execution paths. So i cannot confirm problem of
no-propagation. (this is on 0.9 trunk).
I did find a problem DB Tsai mentioned to me earlier which is affecting
correctness of PCA computation if overwrite=true is given (basically, job
cleanup accidentally wipes some vector information passed from the frontend
in this case and the job does not assert its existence). Again, this is
only specific to PCA and only if -ow (cleanup of target directory) is
requested. I also extended tests and cleaned up some architectural issues.
I will commit a small fix, but i am still not sure what casues problems in
Stuti's case.
-d
On Wed, Aug 7, 2013 at 2:51 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> yes. i think it got broken after replacing side loaders
> by SequenceFileDirValueIterator stuff which accepts glob but for some
> reason direct file name does not find the file...
>
> (i would assume if glob is something like /path/file* than a direct name
> in a form /path/file should also be a valid glob but it seems not to pick
> the file).
>
>
>
> On Wed, Aug 7, 2013 at 1:09 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>
>> also i think I am seeing the issues DB mentioned to me yesterday at
>> sparkML meetup (or something similar).
>>
>>
>> On Wed, Aug 7, 2013 at 1:01 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>>
>>> Thanks, Stuti.
>>>
>>> ok, i think there's something indeed going on with PCA stuff. it may
>>> require a patch.
>>>
>>> -d
>>>
>>>
>>> On Wed, Aug 7, 2013 at 2:12 AM, Stuti Awasthi <st...@hcl.com>wrote:
>>>
>>>> I have not used -q option while running ssvd. Here are the commands
>>>> which I run :
>>>>
>>>> Seq2sparse :
>>>> mahout-distribution-0.7/bin/mahout seq2sparse -i /stuti/SSVD/data-seq
>>>> -o /stuti/SSVD/data-vectors -s 5 -ml 50 -nv -ng 3 -n 2 -x 70
>>>>
>>>> SSVD
>>>> hadoop jar mahout-distribution-0.7/mahout-core-0.8-job.jar
>>>> org.apache.mahout.math.hadoop.stochasticsvd.SSVDCli -i
>>>> /stuti/SSVD/data-vectors/tf-vectors -o /stuti/SSVD/Output -k 90 -U true -V
>>>> true --reduceTasks 1
>>>>
>>>> Thanks
>>>> Stuti Awasthi
>>>>
>>>> -----Original Message-----
>>>> From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
>>>> Sent: Wednesday, August 07, 2013 2:14 PM
>>>> To: user@mahout.apache.org
>>>> Subject: RE: How to SSVD output to generate Clusters
>>>>
>>>> Thanks Stuti. Yes it looks like it is not there. Let me run a test. One
>>>> question . Did you use -q 0 or 1 or sometni g elw se?
>>>> On Aug 7, 2013 12:18 AM, "Stuti Awasthi" <st...@hcl.com> wrote:
>>>>
>>>> > Hey Dmitriy,
>>>> >
>>>> > Sorry for replying late.
>>>> > I have re-run the steps to generate U. Here are the output:
>>>> >
>>>> > 1. "seq2sparse" Output: Contains Named Vector
>>>> > Key: /Description_10: Value:
>>>> > /Description_10:{554:1.0,54:1.0,514:1.0,322:1.0,247:1.0,91:1.0,127:1.0
>>>> > ,456:1.0,480:1.0,713:1.0,674:1.0,117:1.0,461:1.0,595:2.0,446:1.0,296:2
>>>> > .0}
>>>> >
>>>> > 2. "ssvd" Output of U: Not Contained NamedVector
>>>> > Key: /Description_10: Value:
>>>> >
>>>> {0:0.010564205396743468,1:-0.01989403962804719,2:0.015640314765729225,3:0.04031717183780774,4:-0.03325995251075869,5:0.0294201152018514,6:
>>>> >
>>>> > 0.03834130611889856,7:-0.008686421005328312,8:-0.06164515883823538,9:0
>>>> > .03752875772953153,10:0.04739786931946798,11:-0.07912744917669134,12:0
>>>> > .020078421275704143,13:-0.0
>>>> >
>>>> > 4017504785907734,14:0.012539132502559502,15:0.0733073647645918,16:-0.0
>>>> > 2111033727307056,17:0.0799478317610193,18:-0.08481960414593219,19:-0.0
>>>> > 6987848875856222,20:0.03693
>>>> >
>>>> > 2920091059446,21:-0.06949180571421532,22:-0.03447267994522256,23:-0.07
>>>> > 104196347181493,24:0.022262180555421562,25:-0.0485632586340187,26:-0.0
>>>> > 5380823388650383,27:0.09299
>>>> >
>>>> > 533887785207,28:0.0019344239524856396,29:0.002936116541403362,30:-0.07
>>>> > 249587007236825,31:0.0016026176038041033,32:-0.0711115256224166,33:0.0
>>>> > 6603931206284432,34:0.01922
>>>> >
>>>> > 6806201249697,35:0.13972781245330326,36:0.0787696939450401,37:0.070653
>>>> > 56340476747,38:0.08437107545490818,39:0.06381670380272558,40:0.0464059
>>>> > 64753673735,41:0.0601332388
>>>> >
>>>> > 594578,42:-0.12996454299711707,43:0.10779361589915878,44:-0.0652470275
>>>> > 4474347,45:0.014785171162887613,46:-0.036630574690084586,47:-0.1506665
>>>> > 6149902793,48:0.16190482591
>>>> > 405958,49:-0.00869851116149916}
>>>> >
>>>> > As you said that propagation of keys to keys or/and names to names
>>>> > should happen . Any idea what's going wrong or if there is a mistake
>>>> from my side ?
>>>> >
>>>> > Thanks
>>>> > Stuti Awasthi
>>>> >
>>>> > -----Original Message-----
>>>> > From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
>>>> > Sent: Friday, August 02, 2013 11:39 PM
>>>> > To: user@mahout.apache.org
>>>> > Subject: Re: How to SSVD output to generate Clusters
>>>> >
>>>> > by eyeballing the code, i think i don' t see a problem. if rows of A
>>>> > are named values, then row of U (or U*Sigma or U*Sigma^1/2) would also
>>>> > retain names from values of rows of A. Output would not contain
>>>> > NamedVector values for rows that were not NamedVector values
>>>> > themselves in the input. Does seq2sparse output create NamedVectors
>>>> as values ?
>>>> >
>>>> > Note that if what you want is to have *keys* from from seq2sparse
>>>> > (such as
>>>> > 'file_1') propagated to name of named vector value in U, that much is
>>>> > not happening. The algorithm propagates keys to keys or/and names to
>>>> > names (but not any other combinations of those).
>>>> >
>>>> > -d
>>>> >
>>>> >
>>>> > On Fri, Aug 2, 2013 at 10:42 AM, Dmitriy Lyubimov <dl...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > it should. i worked on the issue and last time it was checked it was
>>>> > > still working with name propagation. if not, then it is a bug
>>>> > >
>>>> > >
>>>> > > On Fri, Aug 2, 2013 at 3:33 AM, Stuti Awasthi <stutiawasthi@hcl.com
>>>> > >wrote:
>>>> > >
>>>> > >> Hey Ted,
>>>> > >>
>>>> > >> As suggested, I tried SSVD with Mahout 0.8. I think the issue of
>>>> > >> NamedVector not propagating to the output U ,still persists.
>>>> > >> Here is what I have done :
>>>> > >>
>>>> > >> 1. Created featureVector using "seq2sparse" with -nv option.
>>>> > >> Checked the output, named vector created.
>>>> > >> 2. Provided this featureVector to "ssvd" with params " -k 100 -U
>>>> > >> true -V true". After execution, 3 output got generated namely :
>>>> > >> sigma, U, V 3. I dumped "U" to check the output if it contained
>>>> namedVectors:
>>>> > >>
>>>> > >> mahout-distribution-0.7/bin/mahout seqdumper -i
>>>> > >> /stuti/SSVD/Output/U
>>>> > >> | more
>>>> > >> Output:
>>>> > >> Key: /File_1: Value:
>>>> > >>
>>>> {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.
>>>> > >> ....}
>>>> > >>
>>>> > >> I did not see the NamedVector getting created in the output of
>>>> ssvd.
>>>> > >> Please point out if I have missed any step in between.
>>>> > >>
>>>> > >> As I wanted to perform the Clustering, I took the output of "U" and
>>>> > >> generated the NamedVector with custom code. The output looks like
>>>> this :
>>>> > >> Key: /File_1: Value:
>>>> > >> /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.03343115
>>>> > >> 00
>>>> > >> 858222,.....}
>>>> > >>
>>>> > >> Then I fed this namedvector file to KMeans to generate 10 Clusters.
>>>> > >> In this I have used Random Centroid selection with KMeans.
>>>> > >> Finally I dumped the ClusterOutput as :
>>>> > >> <ClusterId>,<DocumentID1>,<DocumentId2>.....
>>>> > >>
>>>> > >> Please let me know if I have performed any mistake in the end to
>>>> > >> end execution as well Im not sure Why SSVD output is not generating
>>>> > >> the named vectors as the issue id fixed..
>>>> > >>
>>>> > >> Please suggest
>>>> > >>
>>>> > >> Regards
>>>> > >> Stuti Awasthi
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >>
>>>> > >> -----Original Message-----
>>>> > >> From: Ted Dunning [mailto:ted.dunning@gmail.com]
>>>> > >> Sent: Thursday, August 01, 2013 8:37 PM
>>>> > >> To: user@mahout.apache.org
>>>> > >> Subject: Re: How to SSVD output to generate Clusters
>>>> > >>
>>>> > >> On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi
>>>> > >> <st...@hcl.com>
>>>> > >> wrote:
>>>> > >>
>>>> > >> > I think there is a problem because of NamedVector as after some
>>>> > >> > search I get this Jira.
>>>> > >> > https://issues.apache.org/jira/browse/MAHOUT-1067
>>>> > >> >
>>>> > >>
>>>> > >> Note also that this bug is fixed in 0.8
>>>> > >>
>>>> > >>
>>>> > >> ::DISCLAIMER::
>>>> > >>
>>>> > >> -------------------------------------------------------------------
>>>> > >> --
>>>> > >> -------------------------------------------------------------------
>>>> > >> --
>>>> > >> ----------
>>>> > >>
>>>> > >> The contents of this e-mail and any attachment(s) are confidential
>>>> > >> and intended for the named recipient(s) only.
>>>> > >> E-mail transmission is not guaranteed to be secure or error-free as
>>>> > >> information could be intercepted, corrupted, lost, destroyed,
>>>> > >> arrive late or incomplete, or may contain viruses in transmission.
>>>> > >> The e mail and its contents (with or without referred errors) shall
>>>> > >> therefore not attach any liability on the originator or HCL or its
>>>> > >> affiliates.
>>>> > >> Views or opinions, if any, presented in this email are solely those
>>>> > >> of the author and may not necessarily reflect the views or opinions
>>>> > >> of HCL or its affiliates. Any form of reproduction, dissemination,
>>>> > >> copying, disclosure, modification, distribution and / or
>>>> > >> publication of this message without the prior written consent of
>>>> > >> authorized representative of HCL is strictly prohibited. If you
>>>> > >> have received this email in error please delete it and notify the
>>>> > >> sender immediately.
>>>> > >> Before opening any email and/or attachments, please check them for
>>>> > >> viruses and other defects.
>>>> > >>
>>>> > >>
>>>> > >> -------------------------------------------------------------------
>>>> > >> --
>>>> > >> -------------------------------------------------------------------
>>>> > >> --
>>>> > >> ----------
>>>> > >>
>>>> > >
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>
Re: How to SSVD output to generate Clusters
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
yes. i think it got broken after replacing side loaders
by SequenceFileDirValueIterator stuff which accepts glob but for some
reason direct file name does not find the file...
(i would assume if glob is something like /path/file* than a direct name in
a form /path/file should also be a valid glob but it seems not to pick the
file).
On Wed, Aug 7, 2013 at 1:09 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> also i think I am seeing the issues DB mentioned to me yesterday at
> sparkML meetup (or something similar).
>
>
> On Wed, Aug 7, 2013 at 1:01 PM, Dmitriy Lyubimov <dl...@gmail.com>wrote:
>
>> Thanks, Stuti.
>>
>> ok, i think there's something indeed going on with PCA stuff. it may
>> require a patch.
>>
>> -d
>>
>>
>> On Wed, Aug 7, 2013 at 2:12 AM, Stuti Awasthi <st...@hcl.com>wrote:
>>
>>> I have not used -q option while running ssvd. Here are the commands
>>> which I run :
>>>
>>> Seq2sparse :
>>> mahout-distribution-0.7/bin/mahout seq2sparse -i /stuti/SSVD/data-seq -o
>>> /stuti/SSVD/data-vectors -s 5 -ml 50 -nv -ng 3 -n 2 -x 70
>>>
>>> SSVD
>>> hadoop jar mahout-distribution-0.7/mahout-core-0.8-job.jar
>>> org.apache.mahout.math.hadoop.stochasticsvd.SSVDCli -i
>>> /stuti/SSVD/data-vectors/tf-vectors -o /stuti/SSVD/Output -k 90 -U true -V
>>> true --reduceTasks 1
>>>
>>> Thanks
>>> Stuti Awasthi
>>>
>>> -----Original Message-----
>>> From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
>>> Sent: Wednesday, August 07, 2013 2:14 PM
>>> To: user@mahout.apache.org
>>> Subject: RE: How to SSVD output to generate Clusters
>>>
>>> Thanks Stuti. Yes it looks like it is not there. Let me run a test. One
>>> question . Did you use -q 0 or 1 or sometni g elw se?
>>> On Aug 7, 2013 12:18 AM, "Stuti Awasthi" <st...@hcl.com> wrote:
>>>
>>> > Hey Dmitriy,
>>> >
>>> > Sorry for replying late.
>>> > I have re-run the steps to generate U. Here are the output:
>>> >
>>> > 1. "seq2sparse" Output: Contains Named Vector
>>> > Key: /Description_10: Value:
>>> > /Description_10:{554:1.0,54:1.0,514:1.0,322:1.0,247:1.0,91:1.0,127:1.0
>>> > ,456:1.0,480:1.0,713:1.0,674:1.0,117:1.0,461:1.0,595:2.0,446:1.0,296:2
>>> > .0}
>>> >
>>> > 2. "ssvd" Output of U: Not Contained NamedVector
>>> > Key: /Description_10: Value:
>>> >
>>> {0:0.010564205396743468,1:-0.01989403962804719,2:0.015640314765729225,3:0.04031717183780774,4:-0.03325995251075869,5:0.0294201152018514,6:
>>> >
>>> > 0.03834130611889856,7:-0.008686421005328312,8:-0.06164515883823538,9:0
>>> > .03752875772953153,10:0.04739786931946798,11:-0.07912744917669134,12:0
>>> > .020078421275704143,13:-0.0
>>> >
>>> > 4017504785907734,14:0.012539132502559502,15:0.0733073647645918,16:-0.0
>>> > 2111033727307056,17:0.0799478317610193,18:-0.08481960414593219,19:-0.0
>>> > 6987848875856222,20:0.03693
>>> >
>>> > 2920091059446,21:-0.06949180571421532,22:-0.03447267994522256,23:-0.07
>>> > 104196347181493,24:0.022262180555421562,25:-0.0485632586340187,26:-0.0
>>> > 5380823388650383,27:0.09299
>>> >
>>> > 533887785207,28:0.0019344239524856396,29:0.002936116541403362,30:-0.07
>>> > 249587007236825,31:0.0016026176038041033,32:-0.0711115256224166,33:0.0
>>> > 6603931206284432,34:0.01922
>>> >
>>> > 6806201249697,35:0.13972781245330326,36:0.0787696939450401,37:0.070653
>>> > 56340476747,38:0.08437107545490818,39:0.06381670380272558,40:0.0464059
>>> > 64753673735,41:0.0601332388
>>> >
>>> > 594578,42:-0.12996454299711707,43:0.10779361589915878,44:-0.0652470275
>>> > 4474347,45:0.014785171162887613,46:-0.036630574690084586,47:-0.1506665
>>> > 6149902793,48:0.16190482591
>>> > 405958,49:-0.00869851116149916}
>>> >
>>> > As you said that propagation of keys to keys or/and names to names
>>> > should happen . Any idea what's going wrong or if there is a mistake
>>> from my side ?
>>> >
>>> > Thanks
>>> > Stuti Awasthi
>>> >
>>> > -----Original Message-----
>>> > From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
>>> > Sent: Friday, August 02, 2013 11:39 PM
>>> > To: user@mahout.apache.org
>>> > Subject: Re: How to SSVD output to generate Clusters
>>> >
>>> > by eyeballing the code, i think i don' t see a problem. if rows of A
>>> > are named values, then row of U (or U*Sigma or U*Sigma^1/2) would also
>>> > retain names from values of rows of A. Output would not contain
>>> > NamedVector values for rows that were not NamedVector values
>>> > themselves in the input. Does seq2sparse output create NamedVectors as
>>> values ?
>>> >
>>> > Note that if what you want is to have *keys* from from seq2sparse
>>> > (such as
>>> > 'file_1') propagated to name of named vector value in U, that much is
>>> > not happening. The algorithm propagates keys to keys or/and names to
>>> > names (but not any other combinations of those).
>>> >
>>> > -d
>>> >
>>> >
>>> > On Fri, Aug 2, 2013 at 10:42 AM, Dmitriy Lyubimov <dl...@gmail.com>
>>> > wrote:
>>> >
>>> > > it should. i worked on the issue and last time it was checked it was
>>> > > still working with name propagation. if not, then it is a bug
>>> > >
>>> > >
>>> > > On Fri, Aug 2, 2013 at 3:33 AM, Stuti Awasthi <stutiawasthi@hcl.com
>>> > >wrote:
>>> > >
>>> > >> Hey Ted,
>>> > >>
>>> > >> As suggested, I tried SSVD with Mahout 0.8. I think the issue of
>>> > >> NamedVector not propagating to the output U ,still persists.
>>> > >> Here is what I have done :
>>> > >>
>>> > >> 1. Created featureVector using "seq2sparse" with -nv option.
>>> > >> Checked the output, named vector created.
>>> > >> 2. Provided this featureVector to "ssvd" with params " -k 100 -U
>>> > >> true -V true". After execution, 3 output got generated namely :
>>> > >> sigma, U, V 3. I dumped "U" to check the output if it contained
>>> namedVectors:
>>> > >>
>>> > >> mahout-distribution-0.7/bin/mahout seqdumper -i
>>> > >> /stuti/SSVD/Output/U
>>> > >> | more
>>> > >> Output:
>>> > >> Key: /File_1: Value:
>>> > >>
>>> {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.
>>> > >> ....}
>>> > >>
>>> > >> I did not see the NamedVector getting created in the output of ssvd.
>>> > >> Please point out if I have missed any step in between.
>>> > >>
>>> > >> As I wanted to perform the Clustering, I took the output of "U" and
>>> > >> generated the NamedVector with custom code. The output looks like
>>> this :
>>> > >> Key: /File_1: Value:
>>> > >> /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.03343115
>>> > >> 00
>>> > >> 858222,.....}
>>> > >>
>>> > >> Then I fed this namedvector file to KMeans to generate 10 Clusters.
>>> > >> In this I have used Random Centroid selection with KMeans.
>>> > >> Finally I dumped the ClusterOutput as :
>>> > >> <ClusterId>,<DocumentID1>,<DocumentId2>.....
>>> > >>
>>> > >> Please let me know if I have performed any mistake in the end to
>>> > >> end execution as well Im not sure Why SSVD output is not generating
>>> > >> the named vectors as the issue id fixed..
>>> > >>
>>> > >> Please suggest
>>> > >>
>>> > >> Regards
>>> > >> Stuti Awasthi
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >>
>>> > >> -----Original Message-----
>>> > >> From: Ted Dunning [mailto:ted.dunning@gmail.com]
>>> > >> Sent: Thursday, August 01, 2013 8:37 PM
>>> > >> To: user@mahout.apache.org
>>> > >> Subject: Re: How to SSVD output to generate Clusters
>>> > >>
>>> > >> On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi
>>> > >> <st...@hcl.com>
>>> > >> wrote:
>>> > >>
>>> > >> > I think there is a problem because of NamedVector as after some
>>> > >> > search I get this Jira.
>>> > >> > https://issues.apache.org/jira/browse/MAHOUT-1067
>>> > >> >
>>> > >>
>>> > >> Note also that this bug is fixed in 0.8
>>> > >>
>>> > >>
>>> > >> ::DISCLAIMER::
>>> > >>
>>> > >> -------------------------------------------------------------------
>>> > >> --
>>> > >> -------------------------------------------------------------------
>>> > >> --
>>> > >> ----------
>>> > >>
>>> > >> The contents of this e-mail and any attachment(s) are confidential
>>> > >> and intended for the named recipient(s) only.
>>> > >> E-mail transmission is not guaranteed to be secure or error-free as
>>> > >> information could be intercepted, corrupted, lost, destroyed,
>>> > >> arrive late or incomplete, or may contain viruses in transmission.
>>> > >> The e mail and its contents (with or without referred errors) shall
>>> > >> therefore not attach any liability on the originator or HCL or its
>>> > >> affiliates.
>>> > >> Views or opinions, if any, presented in this email are solely those
>>> > >> of the author and may not necessarily reflect the views or opinions
>>> > >> of HCL or its affiliates. Any form of reproduction, dissemination,
>>> > >> copying, disclosure, modification, distribution and / or
>>> > >> publication of this message without the prior written consent of
>>> > >> authorized representative of HCL is strictly prohibited. If you
>>> > >> have received this email in error please delete it and notify the
>>> > >> sender immediately.
>>> > >> Before opening any email and/or attachments, please check them for
>>> > >> viruses and other defects.
>>> > >>
>>> > >>
>>> > >> -------------------------------------------------------------------
>>> > >> --
>>> > >> -------------------------------------------------------------------
>>> > >> --
>>> > >> ----------
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>
>>
>
Re: How to SSVD output to generate Clusters
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
also i think I am seeing the issues DB mentioned to me yesterday at sparkML
meetup (or something similar).
On Wed, Aug 7, 2013 at 1:01 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> Thanks, Stuti.
>
> ok, i think there's something indeed going on with PCA stuff. it may
> require a patch.
>
> -d
>
>
> On Wed, Aug 7, 2013 at 2:12 AM, Stuti Awasthi <st...@hcl.com>wrote:
>
>> I have not used -q option while running ssvd. Here are the commands which
>> I run :
>>
>> Seq2sparse :
>> mahout-distribution-0.7/bin/mahout seq2sparse -i /stuti/SSVD/data-seq -o
>> /stuti/SSVD/data-vectors -s 5 -ml 50 -nv -ng 3 -n 2 -x 70
>>
>> SSVD
>> hadoop jar mahout-distribution-0.7/mahout-core-0.8-job.jar
>> org.apache.mahout.math.hadoop.stochasticsvd.SSVDCli -i
>> /stuti/SSVD/data-vectors/tf-vectors -o /stuti/SSVD/Output -k 90 -U true -V
>> true --reduceTasks 1
>>
>> Thanks
>> Stuti Awasthi
>>
>> -----Original Message-----
>> From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
>> Sent: Wednesday, August 07, 2013 2:14 PM
>> To: user@mahout.apache.org
>> Subject: RE: How to SSVD output to generate Clusters
>>
>> Thanks Stuti. Yes it looks like it is not there. Let me run a test. One
>> question . Did you use -q 0 or 1 or sometni g elw se?
>> On Aug 7, 2013 12:18 AM, "Stuti Awasthi" <st...@hcl.com> wrote:
>>
>> > Hey Dmitriy,
>> >
>> > Sorry for replying late.
>> > I have re-run the steps to generate U. Here are the output:
>> >
>> > 1. "seq2sparse" Output: Contains Named Vector
>> > Key: /Description_10: Value:
>> > /Description_10:{554:1.0,54:1.0,514:1.0,322:1.0,247:1.0,91:1.0,127:1.0
>> > ,456:1.0,480:1.0,713:1.0,674:1.0,117:1.0,461:1.0,595:2.0,446:1.0,296:2
>> > .0}
>> >
>> > 2. "ssvd" Output of U: Not Contained NamedVector
>> > Key: /Description_10: Value:
>> >
>> {0:0.010564205396743468,1:-0.01989403962804719,2:0.015640314765729225,3:0.04031717183780774,4:-0.03325995251075869,5:0.0294201152018514,6:
>> >
>> > 0.03834130611889856,7:-0.008686421005328312,8:-0.06164515883823538,9:0
>> > .03752875772953153,10:0.04739786931946798,11:-0.07912744917669134,12:0
>> > .020078421275704143,13:-0.0
>> >
>> > 4017504785907734,14:0.012539132502559502,15:0.0733073647645918,16:-0.0
>> > 2111033727307056,17:0.0799478317610193,18:-0.08481960414593219,19:-0.0
>> > 6987848875856222,20:0.03693
>> >
>> > 2920091059446,21:-0.06949180571421532,22:-0.03447267994522256,23:-0.07
>> > 104196347181493,24:0.022262180555421562,25:-0.0485632586340187,26:-0.0
>> > 5380823388650383,27:0.09299
>> >
>> > 533887785207,28:0.0019344239524856396,29:0.002936116541403362,30:-0.07
>> > 249587007236825,31:0.0016026176038041033,32:-0.0711115256224166,33:0.0
>> > 6603931206284432,34:0.01922
>> >
>> > 6806201249697,35:0.13972781245330326,36:0.0787696939450401,37:0.070653
>> > 56340476747,38:0.08437107545490818,39:0.06381670380272558,40:0.0464059
>> > 64753673735,41:0.0601332388
>> >
>> > 594578,42:-0.12996454299711707,43:0.10779361589915878,44:-0.0652470275
>> > 4474347,45:0.014785171162887613,46:-0.036630574690084586,47:-0.1506665
>> > 6149902793,48:0.16190482591
>> > 405958,49:-0.00869851116149916}
>> >
>> > As you said that propagation of keys to keys or/and names to names
>> > should happen . Any idea what's going wrong or if there is a mistake
>> from my side ?
>> >
>> > Thanks
>> > Stuti Awasthi
>> >
>> > -----Original Message-----
>> > From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
>> > Sent: Friday, August 02, 2013 11:39 PM
>> > To: user@mahout.apache.org
>> > Subject: Re: How to SSVD output to generate Clusters
>> >
>> > by eyeballing the code, i think i don' t see a problem. if rows of A
>> > are named values, then row of U (or U*Sigma or U*Sigma^1/2) would also
>> > retain names from values of rows of A. Output would not contain
>> > NamedVector values for rows that were not NamedVector values
>> > themselves in the input. Does seq2sparse output create NamedVectors as
>> values ?
>> >
>> > Note that if what you want is to have *keys* from from seq2sparse
>> > (such as
>> > 'file_1') propagated to name of named vector value in U, that much is
>> > not happening. The algorithm propagates keys to keys or/and names to
>> > names (but not any other combinations of those).
>> >
>> > -d
>> >
>> >
>> > On Fri, Aug 2, 2013 at 10:42 AM, Dmitriy Lyubimov <dl...@gmail.com>
>> > wrote:
>> >
>> > > it should. i worked on the issue and last time it was checked it was
>> > > still working with name propagation. if not, then it is a bug
>> > >
>> > >
>> > > On Fri, Aug 2, 2013 at 3:33 AM, Stuti Awasthi <stutiawasthi@hcl.com
>> > >wrote:
>> > >
>> > >> Hey Ted,
>> > >>
>> > >> As suggested, I tried SSVD with Mahout 0.8. I think the issue of
>> > >> NamedVector not propagating to the output U ,still persists.
>> > >> Here is what I have done :
>> > >>
>> > >> 1. Created featureVector using "seq2sparse" with -nv option.
>> > >> Checked the output, named vector created.
>> > >> 2. Provided this featureVector to "ssvd" with params " -k 100 -U
>> > >> true -V true". After execution, 3 output got generated namely :
>> > >> sigma, U, V 3. I dumped "U" to check the output if it contained
>> namedVectors:
>> > >>
>> > >> mahout-distribution-0.7/bin/mahout seqdumper -i
>> > >> /stuti/SSVD/Output/U
>> > >> | more
>> > >> Output:
>> > >> Key: /File_1: Value:
>> > >> {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.
>> > >> ....}
>> > >>
>> > >> I did not see the NamedVector getting created in the output of ssvd.
>> > >> Please point out if I have missed any step in between.
>> > >>
>> > >> As I wanted to perform the Clustering, I took the output of "U" and
>> > >> generated the NamedVector with custom code. The output looks like
>> this :
>> > >> Key: /File_1: Value:
>> > >> /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.03343115
>> > >> 00
>> > >> 858222,.....}
>> > >>
>> > >> Then I fed this namedvector file to KMeans to generate 10 Clusters.
>> > >> In this I have used Random Centroid selection with KMeans.
>> > >> Finally I dumped the ClusterOutput as :
>> > >> <ClusterId>,<DocumentID1>,<DocumentId2>.....
>> > >>
>> > >> Please let me know if I have performed any mistake in the end to
>> > >> end execution as well Im not sure Why SSVD output is not generating
>> > >> the named vectors as the issue id fixed..
>> > >>
>> > >> Please suggest
>> > >>
>> > >> Regards
>> > >> Stuti Awasthi
>> > >>
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> -----Original Message-----
>> > >> From: Ted Dunning [mailto:ted.dunning@gmail.com]
>> > >> Sent: Thursday, August 01, 2013 8:37 PM
>> > >> To: user@mahout.apache.org
>> > >> Subject: Re: How to SSVD output to generate Clusters
>> > >>
>> > >> On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi
>> > >> <st...@hcl.com>
>> > >> wrote:
>> > >>
>> > >> > I think there is a problem because of NamedVector as after some
>> > >> > search I get this Jira.
>> > >> > https://issues.apache.org/jira/browse/MAHOUT-1067
>> > >> >
>> > >>
>> > >> Note also that this bug is fixed in 0.8
>> > >>
>> > >>
>> > >> ::DISCLAIMER::
>> > >>
>> > >> -------------------------------------------------------------------
>> > >> --
>> > >> -------------------------------------------------------------------
>> > >> --
>> > >> ----------
>> > >>
>> > >> The contents of this e-mail and any attachment(s) are confidential
>> > >> and intended for the named recipient(s) only.
>> > >> E-mail transmission is not guaranteed to be secure or error-free as
>> > >> information could be intercepted, corrupted, lost, destroyed,
>> > >> arrive late or incomplete, or may contain viruses in transmission.
>> > >> The e mail and its contents (with or without referred errors) shall
>> > >> therefore not attach any liability on the originator or HCL or its
>> > >> affiliates.
>> > >> Views or opinions, if any, presented in this email are solely those
>> > >> of the author and may not necessarily reflect the views or opinions
>> > >> of HCL or its affiliates. Any form of reproduction, dissemination,
>> > >> copying, disclosure, modification, distribution and / or
>> > >> publication of this message without the prior written consent of
>> > >> authorized representative of HCL is strictly prohibited. If you
>> > >> have received this email in error please delete it and notify the
>> > >> sender immediately.
>> > >> Before opening any email and/or attachments, please check them for
>> > >> viruses and other defects.
>> > >>
>> > >>
>> > >> -------------------------------------------------------------------
>> > >> --
>> > >> -------------------------------------------------------------------
>> > >> --
>> > >> ----------
>> > >>
>> > >
>> > >
>> >
>>
>
>
Re: How to SSVD output to generate Clusters
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Thanks, Stuti.
ok, i think there's something indeed going on with PCA stuff. it may
require a patch.
-d
On Wed, Aug 7, 2013 at 2:12 AM, Stuti Awasthi <st...@hcl.com> wrote:
> I have not used -q option while running ssvd. Here are the commands which
> I run :
>
> Seq2sparse :
> mahout-distribution-0.7/bin/mahout seq2sparse -i /stuti/SSVD/data-seq -o
> /stuti/SSVD/data-vectors -s 5 -ml 50 -nv -ng 3 -n 2 -x 70
>
> SSVD
> hadoop jar mahout-distribution-0.7/mahout-core-0.8-job.jar
> org.apache.mahout.math.hadoop.stochasticsvd.SSVDCli -i
> /stuti/SSVD/data-vectors/tf-vectors -o /stuti/SSVD/Output -k 90 -U true -V
> true --reduceTasks 1
>
> Thanks
> Stuti Awasthi
>
> -----Original Message-----
> From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
> Sent: Wednesday, August 07, 2013 2:14 PM
> To: user@mahout.apache.org
> Subject: RE: How to SSVD output to generate Clusters
>
> Thanks Stuti. Yes it looks like it is not there. Let me run a test. One
> question . Did you use -q 0 or 1 or sometni g elw se?
> On Aug 7, 2013 12:18 AM, "Stuti Awasthi" <st...@hcl.com> wrote:
>
> > Hey Dmitriy,
> >
> > Sorry for replying late.
> > I have re-run the steps to generate U. Here are the output:
> >
> > 1. "seq2sparse" Output: Contains Named Vector
> > Key: /Description_10: Value:
> > /Description_10:{554:1.0,54:1.0,514:1.0,322:1.0,247:1.0,91:1.0,127:1.0
> > ,456:1.0,480:1.0,713:1.0,674:1.0,117:1.0,461:1.0,595:2.0,446:1.0,296:2
> > .0}
> >
> > 2. "ssvd" Output of U: Not Contained NamedVector
> > Key: /Description_10: Value:
> >
> {0:0.010564205396743468,1:-0.01989403962804719,2:0.015640314765729225,3:0.04031717183780774,4:-0.03325995251075869,5:0.0294201152018514,6:
> >
> > 0.03834130611889856,7:-0.008686421005328312,8:-0.06164515883823538,9:0
> > .03752875772953153,10:0.04739786931946798,11:-0.07912744917669134,12:0
> > .020078421275704143,13:-0.0
> >
> > 4017504785907734,14:0.012539132502559502,15:0.0733073647645918,16:-0.0
> > 2111033727307056,17:0.0799478317610193,18:-0.08481960414593219,19:-0.0
> > 6987848875856222,20:0.03693
> >
> > 2920091059446,21:-0.06949180571421532,22:-0.03447267994522256,23:-0.07
> > 104196347181493,24:0.022262180555421562,25:-0.0485632586340187,26:-0.0
> > 5380823388650383,27:0.09299
> >
> > 533887785207,28:0.0019344239524856396,29:0.002936116541403362,30:-0.07
> > 249587007236825,31:0.0016026176038041033,32:-0.0711115256224166,33:0.0
> > 6603931206284432,34:0.01922
> >
> > 6806201249697,35:0.13972781245330326,36:0.0787696939450401,37:0.070653
> > 56340476747,38:0.08437107545490818,39:0.06381670380272558,40:0.0464059
> > 64753673735,41:0.0601332388
> >
> > 594578,42:-0.12996454299711707,43:0.10779361589915878,44:-0.0652470275
> > 4474347,45:0.014785171162887613,46:-0.036630574690084586,47:-0.1506665
> > 6149902793,48:0.16190482591
> > 405958,49:-0.00869851116149916}
> >
> > As you said that propagation of keys to keys or/and names to names
> > should happen . Any idea what's going wrong or if there is a mistake
> from my side ?
> >
> > Thanks
> > Stuti Awasthi
> >
> > -----Original Message-----
> > From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
> > Sent: Friday, August 02, 2013 11:39 PM
> > To: user@mahout.apache.org
> > Subject: Re: How to SSVD output to generate Clusters
> >
> > by eyeballing the code, i think i don' t see a problem. if rows of A
> > are named values, then row of U (or U*Sigma or U*Sigma^1/2) would also
> > retain names from values of rows of A. Output would not contain
> > NamedVector values for rows that were not NamedVector values
> > themselves in the input. Does seq2sparse output create NamedVectors as
> values ?
> >
> > Note that if what you want is to have *keys* from from seq2sparse
> > (such as
> > 'file_1') propagated to name of named vector value in U, that much is
> > not happening. The algorithm propagates keys to keys or/and names to
> > names (but not any other combinations of those).
> >
> > -d
> >
> >
> > On Fri, Aug 2, 2013 at 10:42 AM, Dmitriy Lyubimov <dl...@gmail.com>
> > wrote:
> >
> > > it should. i worked on the issue and last time it was checked it was
> > > still working with name propagation. if not, then it is a bug
> > >
> > >
> > > On Fri, Aug 2, 2013 at 3:33 AM, Stuti Awasthi <stutiawasthi@hcl.com
> > >wrote:
> > >
> > >> Hey Ted,
> > >>
> > >> As suggested, I tried SSVD with Mahout 0.8. I think the issue of
> > >> NamedVector not propagating to the output U ,still persists.
> > >> Here is what I have done :
> > >>
> > >> 1. Created featureVector using "seq2sparse" with -nv option.
> > >> Checked the output, named vector created.
> > >> 2. Provided this featureVector to "ssvd" with params " -k 100 -U
> > >> true -V true". After execution, 3 output got generated namely :
> > >> sigma, U, V 3. I dumped "U" to check the output if it contained
> namedVectors:
> > >>
> > >> mahout-distribution-0.7/bin/mahout seqdumper -i
> > >> /stuti/SSVD/Output/U
> > >> | more
> > >> Output:
> > >> Key: /File_1: Value:
> > >> {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.
> > >> ....}
> > >>
> > >> I did not see the NamedVector getting created in the output of ssvd.
> > >> Please point out if I have missed any step in between.
> > >>
> > >> As I wanted to perform the Clustering, I took the output of "U" and
> > >> generated the NamedVector with custom code. The output looks like
> this :
> > >> Key: /File_1: Value:
> > >> /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.03343115
> > >> 00
> > >> 858222,.....}
> > >>
> > >> Then I fed this namedvector file to KMeans to generate 10 Clusters.
> > >> In this I have used Random Centroid selection with KMeans.
> > >> Finally I dumped the ClusterOutput as :
> > >> <ClusterId>,<DocumentID1>,<DocumentId2>.....
> > >>
> > >> Please let me know if I have performed any mistake in the end to
> > >> end execution as well Im not sure Why SSVD output is not generating
> > >> the named vectors as the issue id fixed..
> > >>
> > >> Please suggest
> > >>
> > >> Regards
> > >> Stuti Awasthi
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> -----Original Message-----
> > >> From: Ted Dunning [mailto:ted.dunning@gmail.com]
> > >> Sent: Thursday, August 01, 2013 8:37 PM
> > >> To: user@mahout.apache.org
> > >> Subject: Re: How to SSVD output to generate Clusters
> > >>
> > >> On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi
> > >> <st...@hcl.com>
> > >> wrote:
> > >>
> > >> > I think there is a problem because of NamedVector as after some
> > >> > search I get this Jira.
> > >> > https://issues.apache.org/jira/browse/MAHOUT-1067
> > >> >
> > >>
> > >> Note also that this bug is fixed in 0.8
> > >>
> > >>
> > >> ::DISCLAIMER::
> > >>
> > >> -------------------------------------------------------------------
> > >> --
> > >> -------------------------------------------------------------------
> > >> --
> > >> ----------
> > >>
> > >> The contents of this e-mail and any attachment(s) are confidential
> > >> and intended for the named recipient(s) only.
> > >> E-mail transmission is not guaranteed to be secure or error-free as
> > >> information could be intercepted, corrupted, lost, destroyed,
> > >> arrive late or incomplete, or may contain viruses in transmission.
> > >> The e mail and its contents (with or without referred errors) shall
> > >> therefore not attach any liability on the originator or HCL or its
> > >> affiliates.
> > >> Views or opinions, if any, presented in this email are solely those
> > >> of the author and may not necessarily reflect the views or opinions
> > >> of HCL or its affiliates. Any form of reproduction, dissemination,
> > >> copying, disclosure, modification, distribution and / or
> > >> publication of this message without the prior written consent of
> > >> authorized representative of HCL is strictly prohibited. If you
> > >> have received this email in error please delete it and notify the
> > >> sender immediately.
> > >> Before opening any email and/or attachments, please check them for
> > >> viruses and other defects.
> > >>
> > >>
> > >> -------------------------------------------------------------------
> > >> --
> > >> -------------------------------------------------------------------
> > >> --
> > >> ----------
> > >>
> > >
> > >
> >
>
RE: How to SSVD output to generate Clusters
Posted by Stuti Awasthi <st...@hcl.com>.
I have not used -q option while running ssvd. Here are the commands which I run :
Seq2sparse :
mahout-distribution-0.7/bin/mahout seq2sparse -i /stuti/SSVD/data-seq -o /stuti/SSVD/data-vectors -s 5 -ml 50 -nv -ng 3 -n 2 -x 70
SSVD
hadoop jar mahout-distribution-0.7/mahout-core-0.8-job.jar org.apache.mahout.math.hadoop.stochasticsvd.SSVDCli -i /stuti/SSVD/data-vectors/tf-vectors -o /stuti/SSVD/Output -k 90 -U true -V true --reduceTasks 1
Thanks
Stuti Awasthi
-----Original Message-----
From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
Sent: Wednesday, August 07, 2013 2:14 PM
To: user@mahout.apache.org
Subject: RE: How to SSVD output to generate Clusters
Thanks Stuti. Yes it looks like it is not there. Let me run a test. One
question . Did you use -q 0 or 1 or sometni g elw se?
On Aug 7, 2013 12:18 AM, "Stuti Awasthi" <st...@hcl.com> wrote:
> Hey Dmitriy,
>
> Sorry for replying late.
> I have re-run the steps to generate U. Here are the output:
>
> 1. "seq2sparse" Output: Contains Named Vector
> Key: /Description_10: Value:
> /Description_10:{554:1.0,54:1.0,514:1.0,322:1.0,247:1.0,91:1.0,127:1.0
> ,456:1.0,480:1.0,713:1.0,674:1.0,117:1.0,461:1.0,595:2.0,446:1.0,296:2
> .0}
>
> 2. "ssvd" Output of U: Not Contained NamedVector
> Key: /Description_10: Value:
> {0:0.010564205396743468,1:-0.01989403962804719,2:0.015640314765729225,3:0.04031717183780774,4:-0.03325995251075869,5:0.0294201152018514,6:
>
> 0.03834130611889856,7:-0.008686421005328312,8:-0.06164515883823538,9:0
> .03752875772953153,10:0.04739786931946798,11:-0.07912744917669134,12:0
> .020078421275704143,13:-0.0
>
> 4017504785907734,14:0.012539132502559502,15:0.0733073647645918,16:-0.0
> 2111033727307056,17:0.0799478317610193,18:-0.08481960414593219,19:-0.0
> 6987848875856222,20:0.03693
>
> 2920091059446,21:-0.06949180571421532,22:-0.03447267994522256,23:-0.07
> 104196347181493,24:0.022262180555421562,25:-0.0485632586340187,26:-0.0
> 5380823388650383,27:0.09299
>
> 533887785207,28:0.0019344239524856396,29:0.002936116541403362,30:-0.07
> 249587007236825,31:0.0016026176038041033,32:-0.0711115256224166,33:0.0
> 6603931206284432,34:0.01922
>
> 6806201249697,35:0.13972781245330326,36:0.0787696939450401,37:0.070653
> 56340476747,38:0.08437107545490818,39:0.06381670380272558,40:0.0464059
> 64753673735,41:0.0601332388
>
> 594578,42:-0.12996454299711707,43:0.10779361589915878,44:-0.0652470275
> 4474347,45:0.014785171162887613,46:-0.036630574690084586,47:-0.1506665
> 6149902793,48:0.16190482591
> 405958,49:-0.00869851116149916}
>
> As you said that propagation of keys to keys or/and names to names
> should happen . Any idea what's going wrong or if there is a mistake from my side ?
>
> Thanks
> Stuti Awasthi
>
> -----Original Message-----
> From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
> Sent: Friday, August 02, 2013 11:39 PM
> To: user@mahout.apache.org
> Subject: Re: How to SSVD output to generate Clusters
>
> by eyeballing the code, i think i don' t see a problem. if rows of A
> are named values, then row of U (or U*Sigma or U*Sigma^1/2) would also
> retain names from values of rows of A. Output would not contain
> NamedVector values for rows that were not NamedVector values
> themselves in the input. Does seq2sparse output create NamedVectors as values ?
>
> Note that if what you want is to have *keys* from from seq2sparse
> (such as
> 'file_1') propagated to name of named vector value in U, that much is
> not happening. The algorithm propagates keys to keys or/and names to
> names (but not any other combinations of those).
>
> -d
>
>
> On Fri, Aug 2, 2013 at 10:42 AM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
> > it should. i worked on the issue and last time it was checked it was
> > still working with name propagation. if not, then it is a bug
> >
> >
> > On Fri, Aug 2, 2013 at 3:33 AM, Stuti Awasthi <stutiawasthi@hcl.com
> >wrote:
> >
> >> Hey Ted,
> >>
> >> As suggested, I tried SSVD with Mahout 0.8. I think the issue of
> >> NamedVector not propagating to the output U ,still persists.
> >> Here is what I have done :
> >>
> >> 1. Created featureVector using "seq2sparse" with -nv option.
> >> Checked the output, named vector created.
> >> 2. Provided this featureVector to "ssvd" with params " -k 100 -U
> >> true -V true". After execution, 3 output got generated namely :
> >> sigma, U, V 3. I dumped "U" to check the output if it contained namedVectors:
> >>
> >> mahout-distribution-0.7/bin/mahout seqdumper -i
> >> /stuti/SSVD/Output/U
> >> | more
> >> Output:
> >> Key: /File_1: Value:
> >> {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.
> >> ....}
> >>
> >> I did not see the NamedVector getting created in the output of ssvd.
> >> Please point out if I have missed any step in between.
> >>
> >> As I wanted to perform the Clustering, I took the output of "U" and
> >> generated the NamedVector with custom code. The output looks like this :
> >> Key: /File_1: Value:
> >> /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.03343115
> >> 00
> >> 858222,.....}
> >>
> >> Then I fed this namedvector file to KMeans to generate 10 Clusters.
> >> In this I have used Random Centroid selection with KMeans.
> >> Finally I dumped the ClusterOutput as :
> >> <ClusterId>,<DocumentID1>,<DocumentId2>.....
> >>
> >> Please let me know if I have performed any mistake in the end to
> >> end execution as well Im not sure Why SSVD output is not generating
> >> the named vectors as the issue id fixed..
> >>
> >> Please suggest
> >>
> >> Regards
> >> Stuti Awasthi
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Ted Dunning [mailto:ted.dunning@gmail.com]
> >> Sent: Thursday, August 01, 2013 8:37 PM
> >> To: user@mahout.apache.org
> >> Subject: Re: How to SSVD output to generate Clusters
> >>
> >> On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi
> >> <st...@hcl.com>
> >> wrote:
> >>
> >> > I think there is a problem because of NamedVector as after some
> >> > search I get this Jira.
> >> > https://issues.apache.org/jira/browse/MAHOUT-1067
> >> >
> >>
> >> Note also that this bug is fixed in 0.8
> >>
> >>
> >> ::DISCLAIMER::
> >>
> >> -------------------------------------------------------------------
> >> --
> >> -------------------------------------------------------------------
> >> --
> >> ----------
> >>
> >> The contents of this e-mail and any attachment(s) are confidential
> >> and intended for the named recipient(s) only.
> >> E-mail transmission is not guaranteed to be secure or error-free as
> >> information could be intercepted, corrupted, lost, destroyed,
> >> arrive late or incomplete, or may contain viruses in transmission.
> >> The e mail and its contents (with or without referred errors) shall
> >> therefore not attach any liability on the originator or HCL or its
> >> affiliates.
> >> Views or opinions, if any, presented in this email are solely those
> >> of the author and may not necessarily reflect the views or opinions
> >> of HCL or its affiliates. Any form of reproduction, dissemination,
> >> copying, disclosure, modification, distribution and / or
> >> publication of this message without the prior written consent of
> >> authorized representative of HCL is strictly prohibited. If you
> >> have received this email in error please delete it and notify the
> >> sender immediately.
> >> Before opening any email and/or attachments, please check them for
> >> viruses and other defects.
> >>
> >>
> >> -------------------------------------------------------------------
> >> --
> >> -------------------------------------------------------------------
> >> --
> >> ----------
> >>
> >
> >
>
RE: How to SSVD output to generate Clusters
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
Thanks Stuti. Yes it looks like it is not there. Let me run a test. One
question . Did you use -q 0 or 1 or sometni g elw se?
On Aug 7, 2013 12:18 AM, "Stuti Awasthi" <st...@hcl.com> wrote:
> Hey Dmitriy,
>
> Sorry for replying late.
> I have re-run the steps to generate U. Here are the output:
>
> 1. "seq2sparse" Output: Contains Named Vector
> Key: /Description_10: Value:
> /Description_10:{554:1.0,54:1.0,514:1.0,322:1.0,247:1.0,91:1.0,127:1.0,456:1.0,480:1.0,713:1.0,674:1.0,117:1.0,461:1.0,595:2.0,446:1.0,296:2.0}
>
> 2. "ssvd" Output of U: Not Contained NamedVector
> Key: /Description_10: Value:
> {0:0.010564205396743468,1:-0.01989403962804719,2:0.015640314765729225,3:0.04031717183780774,4:-0.03325995251075869,5:0.0294201152018514,6:
>
> 0.03834130611889856,7:-0.008686421005328312,8:-0.06164515883823538,9:0.03752875772953153,10:0.04739786931946798,11:-0.07912744917669134,12:0.020078421275704143,13:-0.0
>
> 4017504785907734,14:0.012539132502559502,15:0.0733073647645918,16:-0.02111033727307056,17:0.0799478317610193,18:-0.08481960414593219,19:-0.06987848875856222,20:0.03693
>
> 2920091059446,21:-0.06949180571421532,22:-0.03447267994522256,23:-0.07104196347181493,24:0.022262180555421562,25:-0.0485632586340187,26:-0.05380823388650383,27:0.09299
>
> 533887785207,28:0.0019344239524856396,29:0.002936116541403362,30:-0.07249587007236825,31:0.0016026176038041033,32:-0.0711115256224166,33:0.06603931206284432,34:0.01922
>
> 6806201249697,35:0.13972781245330326,36:0.0787696939450401,37:0.07065356340476747,38:0.08437107545490818,39:0.06381670380272558,40:0.046405964753673735,41:0.0601332388
>
> 594578,42:-0.12996454299711707,43:0.10779361589915878,44:-0.06524702754474347,45:0.014785171162887613,46:-0.036630574690084586,47:-0.15066656149902793,48:0.16190482591
> 405958,49:-0.00869851116149916}
>
> As you said that propagation of keys to keys or/and names to names should
> happen . Any idea what's going wrong or if there is a mistake from my side ?
>
> Thanks
> Stuti Awasthi
>
> -----Original Message-----
> From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
> Sent: Friday, August 02, 2013 11:39 PM
> To: user@mahout.apache.org
> Subject: Re: How to SSVD output to generate Clusters
>
> by eyeballing the code, i think i don' t see a problem. if rows of A are
> named values, then row of U (or U*Sigma or U*Sigma^1/2) would also retain
> names from values of rows of A. Output would not contain NamedVector values
> for rows that were not NamedVector values themselves in the input. Does
> seq2sparse output create NamedVectors as values ?
>
> Note that if what you want is to have *keys* from from seq2sparse (such as
> 'file_1') propagated to name of named vector value in U, that much is not
> happening. The algorithm propagates keys to keys or/and names to names (but
> not any other combinations of those).
>
> -d
>
>
> On Fri, Aug 2, 2013 at 10:42 AM, Dmitriy Lyubimov <dl...@gmail.com>
> wrote:
>
> > it should. i worked on the issue and last time it was checked it was
> > still working with name propagation. if not, then it is a bug
> >
> >
> > On Fri, Aug 2, 2013 at 3:33 AM, Stuti Awasthi <stutiawasthi@hcl.com
> >wrote:
> >
> >> Hey Ted,
> >>
> >> As suggested, I tried SSVD with Mahout 0.8. I think the issue of
> >> NamedVector not propagating to the output U ,still persists.
> >> Here is what I have done :
> >>
> >> 1. Created featureVector using "seq2sparse" with -nv option. Checked
> >> the output, named vector created.
> >> 2. Provided this featureVector to "ssvd" with params " -k 100 -U true
> >> -V true". After execution, 3 output got generated namely : sigma, U,
> >> V 3. I dumped "U" to check the output if it contained namedVectors:
> >>
> >> mahout-distribution-0.7/bin/mahout seqdumper -i /stuti/SSVD/Output/U
> >> | more
> >> Output:
> >> Key: /File_1: Value:
> >> {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.
> >> ....}
> >>
> >> I did not see the NamedVector getting created in the output of ssvd.
> >> Please point out if I have missed any step in between.
> >>
> >> As I wanted to perform the Clustering, I took the output of "U" and
> >> generated the NamedVector with custom code. The output looks like this :
> >> Key: /File_1: Value:
> >> /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500
> >> 858222,.....}
> >>
> >> Then I fed this namedvector file to KMeans to generate 10 Clusters.
> >> In this I have used Random Centroid selection with KMeans.
> >> Finally I dumped the ClusterOutput as :
> >> <ClusterId>,<DocumentID1>,<DocumentId2>.....
> >>
> >> Please let me know if I have performed any mistake in the end to end
> >> execution as well Im not sure Why SSVD output is not generating the
> >> named vectors as the issue id fixed..
> >>
> >> Please suggest
> >>
> >> Regards
> >> Stuti Awasthi
> >>
> >>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Ted Dunning [mailto:ted.dunning@gmail.com]
> >> Sent: Thursday, August 01, 2013 8:37 PM
> >> To: user@mahout.apache.org
> >> Subject: Re: How to SSVD output to generate Clusters
> >>
> >> On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi <st...@hcl.com>
> >> wrote:
> >>
> >> > I think there is a problem because of NamedVector as after some
> >> > search I get this Jira.
> >> > https://issues.apache.org/jira/browse/MAHOUT-1067
> >> >
> >>
> >> Note also that this bug is fixed in 0.8
> >>
> >>
> >> ::DISCLAIMER::
> >>
> >> ---------------------------------------------------------------------
> >> ---------------------------------------------------------------------
> >> ----------
> >>
> >> The contents of this e-mail and any attachment(s) are confidential
> >> and intended for the named recipient(s) only.
> >> E-mail transmission is not guaranteed to be secure or error-free as
> >> information could be intercepted, corrupted, lost, destroyed, arrive
> >> late or incomplete, or may contain viruses in transmission. The e
> >> mail and its contents (with or without referred errors) shall
> >> therefore not attach any liability on the originator or HCL or its
> >> affiliates.
> >> Views or opinions, if any, presented in this email are solely those
> >> of the author and may not necessarily reflect the views or opinions
> >> of HCL or its affiliates. Any form of reproduction, dissemination,
> >> copying, disclosure, modification, distribution and / or publication
> >> of this message without the prior written consent of authorized
> >> representative of HCL is strictly prohibited. If you have received
> >> this email in error please delete it and notify the sender
> >> immediately.
> >> Before opening any email and/or attachments, please check them for
> >> viruses and other defects.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> ---------------------------------------------------------------------
> >> ----------
> >>
> >
> >
>
RE: How to SSVD output to generate Clusters
Posted by Stuti Awasthi <st...@hcl.com>.
Hey Dmitriy,
Sorry for replying late.
I have re-run the steps to generate U. Here are the output:
1. "seq2sparse" Output: Contains Named Vector
Key: /Description_10: Value: /Description_10:{554:1.0,54:1.0,514:1.0,322:1.0,247:1.0,91:1.0,127:1.0,456:1.0,480:1.0,713:1.0,674:1.0,117:1.0,461:1.0,595:2.0,446:1.0,296:2.0}
2. "ssvd" Output of U: Not Contained NamedVector
Key: /Description_10: Value: {0:0.010564205396743468,1:-0.01989403962804719,2:0.015640314765729225,3:0.04031717183780774,4:-0.03325995251075869,5:0.0294201152018514,6:
0.03834130611889856,7:-0.008686421005328312,8:-0.06164515883823538,9:0.03752875772953153,10:0.04739786931946798,11:-0.07912744917669134,12:0.020078421275704143,13:-0.0
4017504785907734,14:0.012539132502559502,15:0.0733073647645918,16:-0.02111033727307056,17:0.0799478317610193,18:-0.08481960414593219,19:-0.06987848875856222,20:0.03693
2920091059446,21:-0.06949180571421532,22:-0.03447267994522256,23:-0.07104196347181493,24:0.022262180555421562,25:-0.0485632586340187,26:-0.05380823388650383,27:0.09299
533887785207,28:0.0019344239524856396,29:0.002936116541403362,30:-0.07249587007236825,31:0.0016026176038041033,32:-0.0711115256224166,33:0.06603931206284432,34:0.01922
6806201249697,35:0.13972781245330326,36:0.0787696939450401,37:0.07065356340476747,38:0.08437107545490818,39:0.06381670380272558,40:0.046405964753673735,41:0.0601332388
594578,42:-0.12996454299711707,43:0.10779361589915878,44:-0.06524702754474347,45:0.014785171162887613,46:-0.036630574690084586,47:-0.15066656149902793,48:0.16190482591
405958,49:-0.00869851116149916}
As you said that propagation of keys to keys or/and names to names should happen . Any idea what's going wrong or if there is a mistake from my side ?
Thanks
Stuti Awasthi
-----Original Message-----
From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
Sent: Friday, August 02, 2013 11:39 PM
To: user@mahout.apache.org
Subject: Re: How to SSVD output to generate Clusters
by eyeballing the code, i think i don' t see a problem. if rows of A are named values, then row of U (or U*Sigma or U*Sigma^1/2) would also retain names from values of rows of A. Output would not contain NamedVector values for rows that were not NamedVector values themselves in the input. Does seq2sparse output create NamedVectors as values ?
Note that if what you want is to have *keys* from from seq2sparse (such as
'file_1') propagated to name of named vector value in U, that much is not happening. The algorithm propagates keys to keys or/and names to names (but not any other combinations of those).
-d
On Fri, Aug 2, 2013 at 10:42 AM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> it should. i worked on the issue and last time it was checked it was
> still working with name propagation. if not, then it is a bug
>
>
> On Fri, Aug 2, 2013 at 3:33 AM, Stuti Awasthi <st...@hcl.com>wrote:
>
>> Hey Ted,
>>
>> As suggested, I tried SSVD with Mahout 0.8. I think the issue of
>> NamedVector not propagating to the output U ,still persists.
>> Here is what I have done :
>>
>> 1. Created featureVector using "seq2sparse" with -nv option. Checked
>> the output, named vector created.
>> 2. Provided this featureVector to "ssvd" with params " -k 100 -U true
>> -V true". After execution, 3 output got generated namely : sigma, U,
>> V 3. I dumped "U" to check the output if it contained namedVectors:
>>
>> mahout-distribution-0.7/bin/mahout seqdumper -i /stuti/SSVD/Output/U
>> | more
>> Output:
>> Key: /File_1: Value:
>> {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.
>> ....}
>>
>> I did not see the NamedVector getting created in the output of ssvd.
>> Please point out if I have missed any step in between.
>>
>> As I wanted to perform the Clustering, I took the output of "U" and
>> generated the NamedVector with custom code. The output looks like this :
>> Key: /File_1: Value:
>> /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500
>> 858222,.....}
>>
>> Then I fed this namedvector file to KMeans to generate 10 Clusters.
>> In this I have used Random Centroid selection with KMeans.
>> Finally I dumped the ClusterOutput as :
>> <ClusterId>,<DocumentID1>,<DocumentId2>.....
>>
>> Please let me know if I have performed any mistake in the end to end
>> execution as well Im not sure Why SSVD output is not generating the
>> named vectors as the issue id fixed..
>>
>> Please suggest
>>
>> Regards
>> Stuti Awasthi
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Ted Dunning [mailto:ted.dunning@gmail.com]
>> Sent: Thursday, August 01, 2013 8:37 PM
>> To: user@mahout.apache.org
>> Subject: Re: How to SSVD output to generate Clusters
>>
>> On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi <st...@hcl.com>
>> wrote:
>>
>> > I think there is a problem because of NamedVector as after some
>> > search I get this Jira.
>> > https://issues.apache.org/jira/browse/MAHOUT-1067
>> >
>>
>> Note also that this bug is fixed in 0.8
>>
>>
>> ::DISCLAIMER::
>>
>> ---------------------------------------------------------------------
>> ---------------------------------------------------------------------
>> ----------
>>
>> The contents of this e-mail and any attachment(s) are confidential
>> and intended for the named recipient(s) only.
>> E-mail transmission is not guaranteed to be secure or error-free as
>> information could be intercepted, corrupted, lost, destroyed, arrive
>> late or incomplete, or may contain viruses in transmission. The e
>> mail and its contents (with or without referred errors) shall
>> therefore not attach any liability on the originator or HCL or its
>> affiliates.
>> Views or opinions, if any, presented in this email are solely those
>> of the author and may not necessarily reflect the views or opinions
>> of HCL or its affiliates. Any form of reproduction, dissemination,
>> copying, disclosure, modification, distribution and / or publication
>> of this message without the prior written consent of authorized
>> representative of HCL is strictly prohibited. If you have received
>> this email in error please delete it and notify the sender
>> immediately.
>> Before opening any email and/or attachments, please check them for
>> viruses and other defects.
>>
>>
>> ---------------------------------------------------------------------
>> ---------------------------------------------------------------------
>> ----------
>>
>
>
Re: How to SSVD output to generate Clusters
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
by eyeballing the code, i think i don' t see a problem. if rows of A are
named values, then row of U (or U*Sigma or U*Sigma^1/2) would also retain
names from values of rows of A. Output would not contain NamedVector values
for rows that were not NamedVector values themselves in the input. Does
seq2sparse output create NamedVectors as values ?
Note that if what you want is to have *keys* from from seq2sparse (such as
'file_1') propagated to name of named vector value in U, that much is not
happening. The algorithm propagates keys to keys or/and names to names (but
not any other combinations of those).
-d
On Fri, Aug 2, 2013 at 10:42 AM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> it should. i worked on the issue and last time it was checked it was still
> working with name propagation. if not, then it is a bug
>
>
> On Fri, Aug 2, 2013 at 3:33 AM, Stuti Awasthi <st...@hcl.com>wrote:
>
>> Hey Ted,
>>
>> As suggested, I tried SSVD with Mahout 0.8. I think the issue of
>> NamedVector not propagating to the output U ,still persists.
>> Here is what I have done :
>>
>> 1. Created featureVector using "seq2sparse" with -nv option. Checked the
>> output, named vector created.
>> 2. Provided this featureVector to "ssvd" with params " -k 100 -U true -V
>> true". After execution, 3 output got generated namely : sigma, U, V
>> 3. I dumped "U" to check the output if it contained namedVectors:
>>
>> mahout-distribution-0.7/bin/mahout seqdumper -i /stuti/SSVD/Output/U |
>> more
>> Output:
>> Key: /File_1: Value:
>> {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.....}
>>
>> I did not see the NamedVector getting created in the output of ssvd.
>> Please point out if I have missed any step in between.
>>
>> As I wanted to perform the Clustering, I took the output of "U" and
>> generated the NamedVector with custom code. The output looks like this :
>> Key: /File_1: Value:
>> /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.....}
>>
>> Then I fed this namedvector file to KMeans to generate 10 Clusters. In
>> this I have used Random Centroid selection with KMeans.
>> Finally I dumped the ClusterOutput as :
>> <ClusterId>,<DocumentID1>,<DocumentId2>.....
>>
>> Please let me know if I have performed any mistake in the end to end
>> execution as well Im not sure Why SSVD output is not generating the named
>> vectors as the issue id fixed..
>>
>> Please suggest
>>
>> Regards
>> Stuti Awasthi
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Ted Dunning [mailto:ted.dunning@gmail.com]
>> Sent: Thursday, August 01, 2013 8:37 PM
>> To: user@mahout.apache.org
>> Subject: Re: How to SSVD output to generate Clusters
>>
>> On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi <st...@hcl.com>
>> wrote:
>>
>> > I think there is a problem because of NamedVector as after some search
>> > I get this Jira. https://issues.apache.org/jira/browse/MAHOUT-1067
>> >
>>
>> Note also that this bug is fixed in 0.8
>>
>>
>> ::DISCLAIMER::
>>
>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>
>> The contents of this e-mail and any attachment(s) are confidential and
>> intended for the named recipient(s) only.
>> E-mail transmission is not guaranteed to be secure or error-free as
>> information could be intercepted, corrupted,
>> lost, destroyed, arrive late or incomplete, or may contain viruses in
>> transmission. The e mail and its contents
>> (with or without referred errors) shall therefore not attach any
>> liability on the originator or HCL or its affiliates.
>> Views or opinions, if any, presented in this email are solely those of
>> the author and may not necessarily reflect the
>> views or opinions of HCL or its affiliates. Any form of reproduction,
>> dissemination, copying, disclosure, modification,
>> distribution and / or publication of this message without the prior
>> written consent of authorized representative of
>> HCL is strictly prohibited. If you have received this email in error
>> please delete it and notify the sender immediately.
>> Before opening any email and/or attachments, please check them for
>> viruses and other defects.
>>
>>
>> ----------------------------------------------------------------------------------------------------------------------------------------------------
>>
>
>
Re: How to SSVD output to generate Clusters
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
it should. i worked on the issue and last time it was checked it was still
working with name propagation. if not, then it is a bug
On Fri, Aug 2, 2013 at 3:33 AM, Stuti Awasthi <st...@hcl.com> wrote:
> Hey Ted,
>
> As suggested, I tried SSVD with Mahout 0.8. I think the issue of
> NamedVector not propagating to the output U ,still persists.
> Here is what I have done :
>
> 1. Created featureVector using "seq2sparse" with -nv option. Checked the
> output, named vector created.
> 2. Provided this featureVector to "ssvd" with params " -k 100 -U true -V
> true". After execution, 3 output got generated namely : sigma, U, V
> 3. I dumped "U" to check the output if it contained namedVectors:
>
> mahout-distribution-0.7/bin/mahout seqdumper -i /stuti/SSVD/Output/U | more
> Output:
> Key: /File_1: Value:
> {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.....}
>
> I did not see the NamedVector getting created in the output of ssvd.
> Please point out if I have missed any step in between.
>
> As I wanted to perform the Clustering, I took the output of "U" and
> generated the NamedVector with custom code. The output looks like this :
> Key: /File_1: Value:
> /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.....}
>
> Then I fed this namedvector file to KMeans to generate 10 Clusters. In
> this I have used Random Centroid selection with KMeans.
> Finally I dumped the ClusterOutput as :
> <ClusterId>,<DocumentID1>,<DocumentId2>.....
>
> Please let me know if I have performed any mistake in the end to end
> execution as well Im not sure Why SSVD output is not generating the named
> vectors as the issue id fixed..
>
> Please suggest
>
> Regards
> Stuti Awasthi
>
>
>
>
>
> -----Original Message-----
> From: Ted Dunning [mailto:ted.dunning@gmail.com]
> Sent: Thursday, August 01, 2013 8:37 PM
> To: user@mahout.apache.org
> Subject: Re: How to SSVD output to generate Clusters
>
> On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> > I think there is a problem because of NamedVector as after some search
> > I get this Jira. https://issues.apache.org/jira/browse/MAHOUT-1067
> >
>
> Note also that this bug is fixed in 0.8
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
RE: How to SSVD output to generate Clusters
Posted by Stuti Awasthi <st...@hcl.com>.
Hey Ted,
As suggested, I tried SSVD with Mahout 0.8. I think the issue of NamedVector not propagating to the output U ,still persists.
Here is what I have done :
1. Created featureVector using "seq2sparse" with -nv option. Checked the output, named vector created.
2. Provided this featureVector to "ssvd" with params " -k 100 -U true -V true". After execution, 3 output got generated namely : sigma, U, V
3. I dumped "U" to check the output if it contained namedVectors:
mahout-distribution-0.7/bin/mahout seqdumper -i /stuti/SSVD/Output/U | more
Output:
Key: /File_1: Value: {0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.....}
I did not see the NamedVector getting created in the output of ssvd. Please point out if I have missed any step in between.
As I wanted to perform the Clustering, I took the output of "U" and generated the NamedVector with custom code. The output looks like this :
Key: /File_1: Value: /File_1:{0:0.027019746696983288,1:0.006124424321845726,2:0.0334311500858222,.....}
Then I fed this namedvector file to KMeans to generate 10 Clusters. In this I have used Random Centroid selection with KMeans.
Finally I dumped the ClusterOutput as :
<ClusterId>,<DocumentID1>,<DocumentId2>.....
Please let me know if I have performed any mistake in the end to end execution as well Im not sure Why SSVD output is not generating the named vectors as the issue id fixed..
Please suggest
Regards
Stuti Awasthi
-----Original Message-----
From: Ted Dunning [mailto:ted.dunning@gmail.com]
Sent: Thursday, August 01, 2013 8:37 PM
To: user@mahout.apache.org
Subject: Re: How to SSVD output to generate Clusters
On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi <st...@hcl.com> wrote:
> I think there is a problem because of NamedVector as after some search
> I get this Jira. https://issues.apache.org/jira/browse/MAHOUT-1067
>
Note also that this bug is fixed in 0.8
::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------
The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only.
E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,
lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents
(with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates.
Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the
views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification,
distribution and / or publication of this message without the prior written consent of authorized representative of
HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.
Before opening any email and/or attachments, please check them for viruses and other defects.
----------------------------------------------------------------------------------------------------------------------------------------------------
Re: How to SSVD output to generate Clusters
Posted by Ted Dunning <te...@gmail.com>.
On Thu, Aug 1, 2013 at 5:49 AM, Stuti Awasthi <st...@hcl.com> wrote:
> I think there is a problem because of NamedVector as after some search I
> get this Jira. https://issues.apache.org/jira/browse/MAHOUT-1067
>
Note also that this bug is fixed in 0.8
Re: How to SSVD output to generate Clusters
Posted by Ted Dunning <te...@gmail.com>.
The original motivation of spectral clustering talks about graphs.
But the idea of clustering the reduced dimension form of a matrix simply
depends on the fact[1] that the metric is approximately preserved by the
reduced form and is thus applicable to any matrix.
[1] Johnson-Lindenstrauss yet again.
On Thu, Aug 1, 2013 at 6:22 AM, Chirag Lakhani <cl...@zaloni.com> wrote:
> Maybe someone can clarify this issue but the spectral clustering
> implementation assumes an affinity graph, am I correct? Are there direct
> ways of going from a list of feature vectors to an affinity matrix in order
> to then implement spectral clustering?
>
>
> On Thu, Aug 1, 2013 at 8:49 AM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> > Thanks Ted, Dmitriy
> >
> > Il check the Spectral Clustering as well PCA option but first with normal
> > approach I want to execute it once.
> >
> > Here is what I am doing with Mahout 0.7:
> > 1. seqdirectory :
> > ~/mahout-distribution-0.7/bin/mahout seqdirectory -i
> > /stuti/SSVD/ClusteringInput -o /stuti/SSVD/data-seq
> >
> > 2.seq2sparse
> > ~/mahout-distribution-0.7/bin/mahout seq2sparse -i /stuti/SSVD/data-seq
> -o
> > /stuti/SSVD/data-vectors -s 5 -ml 50 -nv -ng 3 -n 2 -x 70
> >
> > 3. ssvd
> > ~/mahout-distribution-0.7/bin/mahout ssvd -i
> > /stuti/SSVD/data-vectors/tf-vectors -o /stuti/SSVD/Output -k 10 -U true
> -V
> > true --reduceTasks 1
> >
> > 4.kmeans: with U as input
> > ~/mahout-distribution-0.7/bin/mahout kmeans -i /stuti/SSVD/Output/U -c
> > /stuti/intial-centroids -o /stuti/SSVD/Cluster/kmeans-clusters -dm
> > org.apache.mahout.common.distance.CosineDistanceMeasure -cd 0.1 -x 20 -cl
> > -k 10
> >
> > 5. Clusterdump
> > ~/mahout-distribution-0.7/bin/mahout clusterdump -dt sequencefile -i
> > /stuti/SSVD/Cluster/kmeans-clusters/clusters-*-final -d
> > /stuti/SSVD/data-vectors/dictionary.file-* -o
> > ~/ClusterOutput/SSVD/KMeans_10 -p
> > /stuti/SSVD/Cluster/kmeans-clusters/clusteredPoints -n 10 -b 200 -of CSV
> >
> > Output :
> > Normally if I use Clusterdump with CSV option, the I receive the
> ClusterId
> > and associated documents names but this time Im getting the output like :
> >
> >
> 120,_0_-0.06453357851086772_1_-0.11705342976172932_2_0.04432960668756471_3_0.10046604725589514_4_-0.06602768838676538_5_-0.16253383395031692_6_-0.0042184763959784155_7_0.03321981657725734_8_-0.04904708660966478_9_0.015635264416337353_,
> > .......
> >
> > I think there is a problem because of NamedVector as after some search I
> > get this Jira. https://issues.apache.org/jira/browse/MAHOUT-1067
> >
> > My Queries :
> > 1. Is the process which Im doing is correct ? should U be directly fed as
> > input to Clustering Algorithm
> >
> > 2. The Output issue is because of NamedVector ?? If yes , then if I use
> > Mahout 0.8 will the issue be resolved ?
> >
> > 3. Im confused between parameter "-k" in SSVD and "-k" in
> > Clustering(KMeans). How these are different ? As -k in Clustering means
> > Number of cluster to be created . What is the purpose of -k(rank) in SSVD
> > (My apologies, but I am having some problem in grasping the SSVD
> > algorithm. The concept of Rank is not clear to me)
> >
> > 4. If I generate -k =100 in SSVD, will I still be able to create say 10
> > Clusters using the clustering with this data.
> >
> > Thanks
> > Stuti Awasthi
> >
> > -----Original Message-----
> > From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
> > Sent: Wednesday, July 31, 2013 11:15 PM
> > To: user@mahout.apache.org
> > Subject: Re: How to SSVD output to generate Clusters
> >
> > many people also use PCA options workflow with SSVD and then try
> > clusterize the output U*Sigma which is dimensionally reduced
> representation
> > of original row-wise dataset. To enable PCA and U*Sigma output, use
> >
> > ssvd -pca true -us true -u false -v false -k=... -q=1 ...
> >
> > -q=1 recommended for accuracy.
> >
> >
> >
> > On Wed, Jul 31, 2013 at 5:09 AM, Stuti Awasthi <st...@hcl.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I wanted to group the documents with same context but which belongs to
> > > one single domain together. I have tried KMeans and LDA provided in
> > > Mahout to perform the clustering but the groups which are generated
> > > are not very good. Hence I thought to use LSA to indentify the context
> > > related to the word and then perform the Clustering.
> > >
> > > I am able to run SSVD of Mahout and generated 3 files : Sigma,U,V as
> > > output of SSVD.
> > > I am not sure how to use the output of SSVD to fed to the Clustering
> > > Algorithm so that we can generate the clusters of the documents which
> > > might be talking about same context.
> > >
> > > Any pointers how can I achieve this ?
> > >
> > > Regards
> > > Stuti Awasthi
> > >
> > >
> > > ::DISCLAIMER::
> > >
> > > ----------------------------------------------------------------------
> > > ----------------------------------------------------------------------
> > > --------
> > >
> > > The contents of this e-mail and any attachment(s) are confidential and
> > > intended for the named recipient(s) only.
> > > E-mail transmission is not guaranteed to be secure or error-free as
> > > information could be intercepted, corrupted, lost, destroyed, arrive
> > > late or incomplete, or may contain viruses in transmission. The e mail
> > > and its contents (with or without referred errors) shall therefore not
> > > attach any liability on the originator or HCL or its affiliates.
> > > Views or opinions, if any, presented in this email are solely those of
> > > the author and may not necessarily reflect the views or opinions of
> > > HCL or its affiliates. Any form of reproduction, dissemination,
> > > copying, disclosure, modification, distribution and / or publication
> > > of this message without the prior written consent of authorized
> > > representative of HCL is strictly prohibited. If you have received
> > > this email in error please delete it and notify the sender
> > > immediately.
> > > Before opening any email and/or attachments, please check them for
> > > viruses and other defects.
> > >
> > >
> > > ----------------------------------------------------------------------
> > >
> >
> ------------------------------------------------------------------------------
> > >
> >
>
>
>
> --
>
> *Chirag Lakhani*
>
> Data Scientist
>
> Zaloni, Inc. | www.zaloni.com
>
> 633 Davis Dr., Suite 200
>
> Durham, NC 27713
> e: clakhani@zaloni.com
> p: 919.602.4965 x7020
>
Re: How to SSVD output to generate Clusters
Posted by Chirag Lakhani <cl...@zaloni.com>.
Maybe someone can clarify this issue but the spectral clustering
implementation assumes an affinity graph, am I correct? Are there direct
ways of going from a list of feature vectors to an affinity matrix in order
to then implement spectral clustering?
On Thu, Aug 1, 2013 at 8:49 AM, Stuti Awasthi <st...@hcl.com> wrote:
> Thanks Ted, Dmitriy
>
> Il check the Spectral Clustering as well PCA option but first with normal
> approach I want to execute it once.
>
> Here is what I am doing with Mahout 0.7:
> 1. seqdirectory :
> ~/mahout-distribution-0.7/bin/mahout seqdirectory -i
> /stuti/SSVD/ClusteringInput -o /stuti/SSVD/data-seq
>
> 2.seq2sparse
> ~/mahout-distribution-0.7/bin/mahout seq2sparse -i /stuti/SSVD/data-seq -o
> /stuti/SSVD/data-vectors -s 5 -ml 50 -nv -ng 3 -n 2 -x 70
>
> 3. ssvd
> ~/mahout-distribution-0.7/bin/mahout ssvd -i
> /stuti/SSVD/data-vectors/tf-vectors -o /stuti/SSVD/Output -k 10 -U true -V
> true --reduceTasks 1
>
> 4.kmeans: with U as input
> ~/mahout-distribution-0.7/bin/mahout kmeans -i /stuti/SSVD/Output/U -c
> /stuti/intial-centroids -o /stuti/SSVD/Cluster/kmeans-clusters -dm
> org.apache.mahout.common.distance.CosineDistanceMeasure -cd 0.1 -x 20 -cl
> -k 10
>
> 5. Clusterdump
> ~/mahout-distribution-0.7/bin/mahout clusterdump -dt sequencefile -i
> /stuti/SSVD/Cluster/kmeans-clusters/clusters-*-final -d
> /stuti/SSVD/data-vectors/dictionary.file-* -o
> ~/ClusterOutput/SSVD/KMeans_10 -p
> /stuti/SSVD/Cluster/kmeans-clusters/clusteredPoints -n 10 -b 200 -of CSV
>
> Output :
> Normally if I use Clusterdump with CSV option, the I receive the ClusterId
> and associated documents names but this time Im getting the output like :
>
> 120,_0_-0.06453357851086772_1_-0.11705342976172932_2_0.04432960668756471_3_0.10046604725589514_4_-0.06602768838676538_5_-0.16253383395031692_6_-0.0042184763959784155_7_0.03321981657725734_8_-0.04904708660966478_9_0.015635264416337353_,
> .......
>
> I think there is a problem because of NamedVector as after some search I
> get this Jira. https://issues.apache.org/jira/browse/MAHOUT-1067
>
> My Queries :
> 1. Is the process which Im doing is correct ? should U be directly fed as
> input to Clustering Algorithm
>
> 2. The Output issue is because of NamedVector ?? If yes , then if I use
> Mahout 0.8 will the issue be resolved ?
>
> 3. Im confused between parameter "-k" in SSVD and "-k" in
> Clustering(KMeans). How these are different ? As -k in Clustering means
> Number of cluster to be created . What is the purpose of -k(rank) in SSVD
> (My apologies, but I am having some problem in grasping the SSVD
> algorithm. The concept of Rank is not clear to me)
>
> 4. If I generate -k =100 in SSVD, will I still be able to create say 10
> Clusters using the clustering with this data.
>
> Thanks
> Stuti Awasthi
>
> -----Original Message-----
> From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
> Sent: Wednesday, July 31, 2013 11:15 PM
> To: user@mahout.apache.org
> Subject: Re: How to SSVD output to generate Clusters
>
> many people also use PCA options workflow with SSVD and then try
> clusterize the output U*Sigma which is dimensionally reduced representation
> of original row-wise dataset. To enable PCA and U*Sigma output, use
>
> ssvd -pca true -us true -u false -v false -k=... -q=1 ...
>
> -q=1 recommended for accuracy.
>
>
>
> On Wed, Jul 31, 2013 at 5:09 AM, Stuti Awasthi <st...@hcl.com>
> wrote:
>
> > Hi All,
> >
> > I wanted to group the documents with same context but which belongs to
> > one single domain together. I have tried KMeans and LDA provided in
> > Mahout to perform the clustering but the groups which are generated
> > are not very good. Hence I thought to use LSA to indentify the context
> > related to the word and then perform the Clustering.
> >
> > I am able to run SSVD of Mahout and generated 3 files : Sigma,U,V as
> > output of SSVD.
> > I am not sure how to use the output of SSVD to fed to the Clustering
> > Algorithm so that we can generate the clusters of the documents which
> > might be talking about same context.
> >
> > Any pointers how can I achieve this ?
> >
> > Regards
> > Stuti Awasthi
> >
> >
> > ::DISCLAIMER::
> >
> > ----------------------------------------------------------------------
> > ----------------------------------------------------------------------
> > --------
> >
> > The contents of this e-mail and any attachment(s) are confidential and
> > intended for the named recipient(s) only.
> > E-mail transmission is not guaranteed to be secure or error-free as
> > information could be intercepted, corrupted, lost, destroyed, arrive
> > late or incomplete, or may contain viruses in transmission. The e mail
> > and its contents (with or without referred errors) shall therefore not
> > attach any liability on the originator or HCL or its affiliates.
> > Views or opinions, if any, presented in this email are solely those of
> > the author and may not necessarily reflect the views or opinions of
> > HCL or its affiliates. Any form of reproduction, dissemination,
> > copying, disclosure, modification, distribution and / or publication
> > of this message without the prior written consent of authorized
> > representative of HCL is strictly prohibited. If you have received
> > this email in error please delete it and notify the sender
> > immediately.
> > Before opening any email and/or attachments, please check them for
> > viruses and other defects.
> >
> >
> > ----------------------------------------------------------------------
> >
> ------------------------------------------------------------------------------
> >
>
--
*Chirag Lakhani*
Data Scientist
Zaloni, Inc. | www.zaloni.com
633 Davis Dr., Suite 200
Durham, NC 27713
e: clakhani@zaloni.com
p: 919.602.4965 x7020
RE: How to SSVD output to generate Clusters
Posted by Stuti Awasthi <st...@hcl.com>.
Thanks Ted, Dmitriy
Il check the Spectral Clustering as well PCA option but first with normal approach I want to execute it once.
Here is what I am doing with Mahout 0.7:
1. seqdirectory :
~/mahout-distribution-0.7/bin/mahout seqdirectory -i /stuti/SSVD/ClusteringInput -o /stuti/SSVD/data-seq
2.seq2sparse
~/mahout-distribution-0.7/bin/mahout seq2sparse -i /stuti/SSVD/data-seq -o /stuti/SSVD/data-vectors -s 5 -ml 50 -nv -ng 3 -n 2 -x 70
3. ssvd
~/mahout-distribution-0.7/bin/mahout ssvd -i /stuti/SSVD/data-vectors/tf-vectors -o /stuti/SSVD/Output -k 10 -U true -V true --reduceTasks 1
4.kmeans: with U as input
~/mahout-distribution-0.7/bin/mahout kmeans -i /stuti/SSVD/Output/U -c /stuti/intial-centroids -o /stuti/SSVD/Cluster/kmeans-clusters -dm org.apache.mahout.common.distance.CosineDistanceMeasure -cd 0.1 -x 20 -cl -k 10
5. Clusterdump
~/mahout-distribution-0.7/bin/mahout clusterdump -dt sequencefile -i /stuti/SSVD/Cluster/kmeans-clusters/clusters-*-final -d /stuti/SSVD/data-vectors/dictionary.file-* -o ~/ClusterOutput/SSVD/KMeans_10 -p /stuti/SSVD/Cluster/kmeans-clusters/clusteredPoints -n 10 -b 200 -of CSV
Output :
Normally if I use Clusterdump with CSV option, the I receive the ClusterId and associated documents names but this time Im getting the output like :
120,_0_-0.06453357851086772_1_-0.11705342976172932_2_0.04432960668756471_3_0.10046604725589514_4_-0.06602768838676538_5_-0.16253383395031692_6_-0.0042184763959784155_7_0.03321981657725734_8_-0.04904708660966478_9_0.015635264416337353_, .......
I think there is a problem because of NamedVector as after some search I get this Jira. https://issues.apache.org/jira/browse/MAHOUT-1067
My Queries :
1. Is the process which Im doing is correct ? should U be directly fed as input to Clustering Algorithm
2. The Output issue is because of NamedVector ?? If yes , then if I use Mahout 0.8 will the issue be resolved ?
3. Im confused between parameter "-k" in SSVD and "-k" in Clustering(KMeans). How these are different ? As -k in Clustering means Number of cluster to be created . What is the purpose of -k(rank) in SSVD
(My apologies, but I am having some problem in grasping the SSVD algorithm. The concept of Rank is not clear to me)
4. If I generate -k =100 in SSVD, will I still be able to create say 10 Clusters using the clustering with this data.
Thanks
Stuti Awasthi
-----Original Message-----
From: Dmitriy Lyubimov [mailto:dlieu.7@gmail.com]
Sent: Wednesday, July 31, 2013 11:15 PM
To: user@mahout.apache.org
Subject: Re: How to SSVD output to generate Clusters
many people also use PCA options workflow with SSVD and then try clusterize the output U*Sigma which is dimensionally reduced representation of original row-wise dataset. To enable PCA and U*Sigma output, use
ssvd -pca true -us true -u false -v false -k=... -q=1 ...
-q=1 recommended for accuracy.
On Wed, Jul 31, 2013 at 5:09 AM, Stuti Awasthi <st...@hcl.com> wrote:
> Hi All,
>
> I wanted to group the documents with same context but which belongs to
> one single domain together. I have tried KMeans and LDA provided in
> Mahout to perform the clustering but the groups which are generated
> are not very good. Hence I thought to use LSA to indentify the context
> related to the word and then perform the Clustering.
>
> I am able to run SSVD of Mahout and generated 3 files : Sigma,U,V as
> output of SSVD.
> I am not sure how to use the output of SSVD to fed to the Clustering
> Algorithm so that we can generate the clusters of the documents which
> might be talking about same context.
>
> Any pointers how can I achieve this ?
>
> Regards
> Stuti Awasthi
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------
> ----------------------------------------------------------------------
> --------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted, lost, destroyed, arrive
> late or incomplete, or may contain viruses in transmission. The e mail
> and its contents (with or without referred errors) shall therefore not
> attach any liability on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of
> the author and may not necessarily reflect the views or opinions of
> HCL or its affiliates. Any form of reproduction, dissemination,
> copying, disclosure, modification, distribution and / or publication
> of this message without the prior written consent of authorized
> representative of HCL is strictly prohibited. If you have received
> this email in error please delete it and notify the sender
> immediately.
> Before opening any email and/or attachments, please check them for
> viruses and other defects.
>
>
> ----------------------------------------------------------------------
> ------------------------------------------------------------------------------
>
Re: How to SSVD output to generate Clusters
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
many people also use PCA options workflow with SSVD and then try clusterize
the output U*Sigma which is dimensionally reduced representation of
original row-wise dataset. To enable PCA and U*Sigma output, use
ssvd -pca true -us true -u false -v false -k=... -q=1 ...
-q=1 recommended for accuracy.
On Wed, Jul 31, 2013 at 5:09 AM, Stuti Awasthi <st...@hcl.com> wrote:
> Hi All,
>
> I wanted to group the documents with same context but which belongs to one
> single domain together. I have tried KMeans and LDA provided in Mahout to
> perform the clustering but the groups which are generated are not very
> good. Hence I thought to use LSA to indentify the context related to the
> word and then perform the Clustering.
>
> I am able to run SSVD of Mahout and generated 3 files : Sigma,U,V as
> output of SSVD.
> I am not sure how to use the output of SSVD to fed to the Clustering
> Algorithm so that we can generate the clusters of the documents which might
> be talking about same context.
>
> Any pointers how can I achieve this ?
>
> Regards
> Stuti Awasthi
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> E-mail transmission is not guaranteed to be secure or error-free as
> information could be intercepted, corrupted,
> lost, destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents
> (with or without referred errors) shall therefore not attach any liability
> on the originator or HCL or its affiliates.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the
> views or opinions of HCL or its affiliates. Any form of reproduction,
> dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior
> written consent of authorized representative of
> HCL is strictly prohibited. If you have received this email in error
> please delete it and notify the sender immediately.
> Before opening any email and/or attachments, please check them for viruses
> and other defects.
>
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>