You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Charly Lizarralde <ch...@gmail.com> on 2013/08/23 17:18:04 UTC
lda + vector dump
Hi everyone, I am experimenting with cvb algorithm and I have a few
questions....
a) Is there any updated documentation? I have been collecting info from
mail lists, blogs, etc. I have been writing a small beginers tutorial, if
you like I'll send it.
b) Should I remove "stop-words" before building the feature vectors ? I am
having some trouble "reading" the results....
c) Vectordump is not sorting well...is this a reported bug ? ( I am
building mahout from trunk now )
d) Any considerations on performance? It took 10 hours on a 5 node cluster
and I've set 20 iterations on less than 10.000 docs and it took
Thanks!
Charly
Re: lda + vector dump
Posted by Jake Mannix <ja...@gmail.com>.
On Fri, Aug 23, 2013 at 10:49 AM, Corey Hyllested <corey.hyllested@gmail.com
> wrote:
> Charly,
>
> The documents I've used for CVB is quite noisy, removing stopwords is key.
>
>
> To remove the stopwords. seq2sparse provides a few options for the analyzer
> to choose what the stopwords are:
> --minDF
> --maxDFPercent
>
> You can write your own sequenceFile parser which will give you more
> control, but that's a lot more work.
>
> I believe removing extra "stopwords" has performance implications in terms
> of memory utilization of the CVB algorithm. It can mean a drastic
> reduction in memory allocated.
>
Actually, the thing which reduces memory overhead is the reverse
of the stopwords: the tail of the language model, those terms which occur
only 1-5 times. "--minDF 5" will chop off those rare terms.
"--maxDFPercent 90"
will chop out those pesky stop words which occur in 90+% of your documents.
> As for using vectordump. Some of the output files are more easily read
> with 'mahout seqdumper'
>
Use vectordump for vector sequence files, as it knows how to join together
your numeric vectors with your textual dictionary. But seqdumper is yes
more general purpose and will print out the string form of any sequencefile
you have handy.
>
>
>
> - Corey
>
>
> On Fri, Aug 23, 2013 at 10:24 AM, Suneel Marthi <suneel_marthi@yahoo.com
> >wrote:
>
> > Charly,
> >
> > If the documentation isn't clear, I would like at the CVB example in
> > examples/bin/cluster_reuters.sh for the correct sequence of steps and the
> > various parameters.
> >
> >
> >
> >
> > ________________________________
> > From: Charly Lizarralde <ch...@gmail.com>
> > To: user@mahout.apache.org
> > Sent: Friday, August 23, 2013 11:45 AM
> > Subject: Re: lda + vector dump
> >
> >
> > I think I am doing it on the cvb output ( 1 record per topic ) so
> > dictionary is used to output the topic most relevant terms....but I'll
> > check!
> >
> >
> > On Fri, Aug 23, 2013 at 12:37 PM, Liz Merkhofer <
> > lmerkhofer@bericotechnologies.com> wrote:
> >
> > > Hi Charly,
> > >
> > > I've been playing around with cvb, too. I have a few thoughts on b,
> > > vectordump:
> > >
> > > What are you doing vectordump on? If you're doing it on your cvb
> output,
> > > you're getting something like a dictionary per topic, with
> > > <input-word-key>:<probability-it's-in-this-cluster>. If you're doing it
> > on
> > > cvb-topics output, for each document, you're getting the likelihood
> that
> > it
> > > belongs to each of your topics.
> > >
> > > I wonder if your problem is that you read the same book I did, "Hadoop
> > > MapReduce Cookbook," that advised to use vectordump with the dictionary
> > > flag as your dictionary from s2s. Don't do that - that translates your
> > > document or topic keys as if they were your vocab keys, and it's just
> > > completely nonsensical.
> > >
> > > Best,
> > > Liz Merkhofer
> > >
> > >
> > >
> > > On Fri, Aug 23, 2013 at 11:18 AM, Charly Lizarralde <
> > > charly.lizarralde@gmail.com> wrote:
> > >
> > > > Hi everyone, I am experimenting with cvb algorithm and I have a few
> > > > questions....
> > > >
> > > > a) Is there any updated documentation? I have been collecting info
> from
> > > > mail lists, blogs, etc. I have been writing a small beginers
> tutorial,
> > if
> > > > you like I'll send it.
> > > >
> > > > b) Should I remove "stop-words" before building the feature vectors
> ? I
> > > am
> > > > having some trouble "reading" the results....
> > > >
> > > > c) Vectordump is not sorting well...is this a reported bug ? ( I am
> > > > building mahout from trunk now )
> > > >
> > > > d) Any considerations on performance? It took 10 hours on a 5 node
> > > cluster
> > > > and I've set 20 iterations on less than 10.000 docs and it took
> > > >
> > > > Thanks!
> > > > Charly
> > > >
> > >
> >
>
--
-jake
Re: lda + vector dump
Posted by Corey Hyllested <co...@gmail.com>.
Charly,
The documents I've used for CVB is quite noisy, removing stopwords is key.
To remove the stopwords. seq2sparse provides a few options for the analyzer
to choose what the stopwords are:
--minDF
--maxDFPercent
You can write your own sequenceFile parser which will give you more
control, but that's a lot more work.
I believe removing extra "stopwords" has performance implications in terms
of memory utilization of the CVB algorithm. It can mean a drastic
reduction in memory allocated.
As for using vectordump. Some of the output files are more easily read
with 'mahout seqdumper'
- Corey
On Fri, Aug 23, 2013 at 10:24 AM, Suneel Marthi <su...@yahoo.com>wrote:
> Charly,
>
> If the documentation isn't clear, I would like at the CVB example in
> examples/bin/cluster_reuters.sh for the correct sequence of steps and the
> various parameters.
>
>
>
>
> ________________________________
> From: Charly Lizarralde <ch...@gmail.com>
> To: user@mahout.apache.org
> Sent: Friday, August 23, 2013 11:45 AM
> Subject: Re: lda + vector dump
>
>
> I think I am doing it on the cvb output ( 1 record per topic ) so
> dictionary is used to output the topic most relevant terms....but I'll
> check!
>
>
> On Fri, Aug 23, 2013 at 12:37 PM, Liz Merkhofer <
> lmerkhofer@bericotechnologies.com> wrote:
>
> > Hi Charly,
> >
> > I've been playing around with cvb, too. I have a few thoughts on b,
> > vectordump:
> >
> > What are you doing vectordump on? If you're doing it on your cvb output,
> > you're getting something like a dictionary per topic, with
> > <input-word-key>:<probability-it's-in-this-cluster>. If you're doing it
> on
> > cvb-topics output, for each document, you're getting the likelihood that
> it
> > belongs to each of your topics.
> >
> > I wonder if your problem is that you read the same book I did, "Hadoop
> > MapReduce Cookbook," that advised to use vectordump with the dictionary
> > flag as your dictionary from s2s. Don't do that - that translates your
> > document or topic keys as if they were your vocab keys, and it's just
> > completely nonsensical.
> >
> > Best,
> > Liz Merkhofer
> >
> >
> >
> > On Fri, Aug 23, 2013 at 11:18 AM, Charly Lizarralde <
> > charly.lizarralde@gmail.com> wrote:
> >
> > > Hi everyone, I am experimenting with cvb algorithm and I have a few
> > > questions....
> > >
> > > a) Is there any updated documentation? I have been collecting info from
> > > mail lists, blogs, etc. I have been writing a small beginers tutorial,
> if
> > > you like I'll send it.
> > >
> > > b) Should I remove "stop-words" before building the feature vectors ? I
> > am
> > > having some trouble "reading" the results....
> > >
> > > c) Vectordump is not sorting well...is this a reported bug ? ( I am
> > > building mahout from trunk now )
> > >
> > > d) Any considerations on performance? It took 10 hours on a 5 node
> > cluster
> > > and I've set 20 iterations on less than 10.000 docs and it took
> > >
> > > Thanks!
> > > Charly
> > >
> >
>
Re: lda + vector dump
Posted by Suneel Marthi <su...@yahoo.com>.
Charly,
If the documentation isn't clear, I would like at the CVB example in examples/bin/cluster_reuters.sh for the correct sequence of steps and the various parameters.
________________________________
From: Charly Lizarralde <ch...@gmail.com>
To: user@mahout.apache.org
Sent: Friday, August 23, 2013 11:45 AM
Subject: Re: lda + vector dump
I think I am doing it on the cvb output ( 1 record per topic ) so
dictionary is used to output the topic most relevant terms....but I'll
check!
On Fri, Aug 23, 2013 at 12:37 PM, Liz Merkhofer <
lmerkhofer@bericotechnologies.com> wrote:
> Hi Charly,
>
> I've been playing around with cvb, too. I have a few thoughts on b,
> vectordump:
>
> What are you doing vectordump on? If you're doing it on your cvb output,
> you're getting something like a dictionary per topic, with
> <input-word-key>:<probability-it's-in-this-cluster>. If you're doing it on
> cvb-topics output, for each document, you're getting the likelihood that it
> belongs to each of your topics.
>
> I wonder if your problem is that you read the same book I did, "Hadoop
> MapReduce Cookbook," that advised to use vectordump with the dictionary
> flag as your dictionary from s2s. Don't do that - that translates your
> document or topic keys as if they were your vocab keys, and it's just
> completely nonsensical.
>
> Best,
> Liz Merkhofer
>
>
>
> On Fri, Aug 23, 2013 at 11:18 AM, Charly Lizarralde <
> charly.lizarralde@gmail.com> wrote:
>
> > Hi everyone, I am experimenting with cvb algorithm and I have a few
> > questions....
> >
> > a) Is there any updated documentation? I have been collecting info from
> > mail lists, blogs, etc. I have been writing a small beginers tutorial, if
> > you like I'll send it.
> >
> > b) Should I remove "stop-words" before building the feature vectors ? I
> am
> > having some trouble "reading" the results....
> >
> > c) Vectordump is not sorting well...is this a reported bug ? ( I am
> > building mahout from trunk now )
> >
> > d) Any considerations on performance? It took 10 hours on a 5 node
> cluster
> > and I've set 20 iterations on less than 10.000 docs and it took
> >
> > Thanks!
> > Charly
> >
>
Re: lda + vector dump
Posted by Charly Lizarralde <ch...@gmail.com>.
I think I am doing it on the cvb output ( 1 record per topic ) so
dictionary is used to output the topic most relevant terms....but I'll
check!
On Fri, Aug 23, 2013 at 12:37 PM, Liz Merkhofer <
lmerkhofer@bericotechnologies.com> wrote:
> Hi Charly,
>
> I've been playing around with cvb, too. I have a few thoughts on b,
> vectordump:
>
> What are you doing vectordump on? If you're doing it on your cvb output,
> you're getting something like a dictionary per topic, with
> <input-word-key>:<probability-it's-in-this-cluster>. If you're doing it on
> cvb-topics output, for each document, you're getting the likelihood that it
> belongs to each of your topics.
>
> I wonder if your problem is that you read the same book I did, "Hadoop
> MapReduce Cookbook," that advised to use vectordump with the dictionary
> flag as your dictionary from s2s. Don't do that - that translates your
> document or topic keys as if they were your vocab keys, and it's just
> completely nonsensical.
>
> Best,
> Liz Merkhofer
>
>
>
> On Fri, Aug 23, 2013 at 11:18 AM, Charly Lizarralde <
> charly.lizarralde@gmail.com> wrote:
>
> > Hi everyone, I am experimenting with cvb algorithm and I have a few
> > questions....
> >
> > a) Is there any updated documentation? I have been collecting info from
> > mail lists, blogs, etc. I have been writing a small beginers tutorial, if
> > you like I'll send it.
> >
> > b) Should I remove "stop-words" before building the feature vectors ? I
> am
> > having some trouble "reading" the results....
> >
> > c) Vectordump is not sorting well...is this a reported bug ? ( I am
> > building mahout from trunk now )
> >
> > d) Any considerations on performance? It took 10 hours on a 5 node
> cluster
> > and I've set 20 iterations on less than 10.000 docs and it took
> >
> > Thanks!
> > Charly
> >
>
Re: lda + vector dump
Posted by Liz Merkhofer <lm...@bericotechnologies.com>.
Hi Charly,
I've been playing around with cvb, too. I have a few thoughts on b,
vectordump:
What are you doing vectordump on? If you're doing it on your cvb output,
you're getting something like a dictionary per topic, with
<input-word-key>:<probability-it's-in-this-cluster>. If you're doing it on
cvb-topics output, for each document, you're getting the likelihood that it
belongs to each of your topics.
I wonder if your problem is that you read the same book I did, "Hadoop
MapReduce Cookbook," that advised to use vectordump with the dictionary
flag as your dictionary from s2s. Don't do that - that translates your
document or topic keys as if they were your vocab keys, and it's just
completely nonsensical.
Best,
Liz Merkhofer
On Fri, Aug 23, 2013 at 11:18 AM, Charly Lizarralde <
charly.lizarralde@gmail.com> wrote:
> Hi everyone, I am experimenting with cvb algorithm and I have a few
> questions....
>
> a) Is there any updated documentation? I have been collecting info from
> mail lists, blogs, etc. I have been writing a small beginers tutorial, if
> you like I'll send it.
>
> b) Should I remove "stop-words" before building the feature vectors ? I am
> having some trouble "reading" the results....
>
> c) Vectordump is not sorting well...is this a reported bug ? ( I am
> building mahout from trunk now )
>
> d) Any considerations on performance? It took 10 hours on a 5 node cluster
> and I've set 20 iterations on less than 10.000 docs and it took
>
> Thanks!
> Charly
>
Re: lda + vector dump
Posted by Ritwik Kumar <li...@gmail.com>.
with Mahout 0.8, I'd try:
mahout vectordump --seqFile
*-i* tn/topics/part-m-00000 --dictionary tn/vectors/dictionary.file-0
--dictionaryType sequencefile --output topicdump.txt -sort *true*--vectorSize 1
On Fri, Aug 23, 2013 at 8:36 AM, Charly Lizarralde <
charly.lizarralde@gmail.com> wrote:
> Thanks! Docs are in spanish, so, maybe I should provide the spanish list
> then...
>
> The vector dump comand is: mahout vectordump --seqFile
> tn/topics/part-m-00000 --dictionary tn/vectors/dictionary.file-0
> --dictionaryType sequencefile --output topicdump.txt -sort --vectorSize 10
>
> And the output ( topicdump.txt) is:
>
>
> {yo:0.05347025391826375,pero:0.01621850129739,hay:0.01256010577070346,como:0.015645488997146385,apellido:0.0138762425391
>
> 95612,quiero:0.02141736852909945,mis:0.011207260571060144,mi:0.10256988732241464,me:0.13765803016629644,zi:0.03454083289
> 2116805}
>
> As you can see, the topicterms are not sorted.
>
Re: lda + vector dump
Posted by Charly Lizarralde <ch...@gmail.com>.
Thanks! Docs are in spanish, so, maybe I should provide the spanish list
then...
The vector dump comand is: mahout vectordump --seqFile
tn/topics/part-m-00000 --dictionary tn/vectors/dictionary.file-0
--dictionaryType sequencefile --output topicdump.txt -sort --vectorSize 10
And the output ( topicdump.txt) is:
{yo:0.05347025391826375,pero:0.01621850129739,hay:0.01256010577070346,como:0.015645488997146385,apellido:0.0138762425391
95612,quiero:0.02141736852909945,mis:0.011207260571060144,mi:0.10256988732241464,me:0.13765803016629644,zi:0.03454083289
2116805}
As you can see, the topicterms are not sorted.
Re: lda + vector dump
Posted by Suneel Marthi <su...@yahoo.com>.
________________________________
From: Charly Lizarralde <ch...@gmail.com>
To: user@mahout.apache.org
Sent: Friday, August 23, 2013 11:18 AM
Subject: lda + vector dump
Hi everyone, I am experimenting with cvb algorithm and I have a few
questions....
a) Is there any updated documentation? I have been collecting info from
mail lists, blogs, etc. I have been writing a small beginers tutorial, if
you like I'll send it.
>> Not sure if the documentation's been updated for this, what were u looking at?
Please send across what you have.
b) Should I remove "stop-words" before building the feature vectors ? I am
having some trouble "reading" the results....
>> If you r running 'seq2sparse' for building the feature vectors and are using the Lucene StandardAnalyzer (which is the default),
the English stopwords should be removed automatically.
c) Vectordump is not sorting well...is this a reported bug ? ( I am
building mahout from trunk now )
>> Please post more details on this.
d) Any considerations on performance? It took 10 hours on a 5 node cluster
and I've set 20 iterations on less than 10.000 docs and it took
Thanks!
Charly