You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@mahout.apache.org by Ben West <bw...@yahoo.com> on 2011/09/05 17:38:05 UTC
LDA question
Hey all,
I'm trying the Latent Dirichlet Allocation operator. I made my term vectors as specified here: https://cwiki.apache.org/MAHOUT/creating-vectors-from-text.html with these commands:
~/Scripts/Mahout/trunk/bin/mahout seqdirectory --input /home/ben/Scripts/eipi/files --output /home/ben/Scripts/eipi/mahout_out -chunk 1
~/Scripts/Mahout/trunk/bin/mahout seq2sparse -i /home/ben/Scripts/eipi/mahout_out -o /home/ben/Scripts/eipi/termvecs -wt tf -seq
Then I run this, trying to follow these instructions: https://cwiki.apache.org/MAHOUT/latent-dirichlet-allocation.html
~/Scripts/Mahout/trunk/bin/mahout lda -i /home/ben/Scripts/eipi/termvecs -o /home/ben/Scripts/eipi/lda_working -k 2 -v 100
And I get:
MAHOUT-JOB: /home/ben/Scripts/Mahout/trunk/examples/target/mahout-examples-0.6-SNAPSHOT-job.jar 11/09/04 16:28:59 INFO common.AbstractJob: Command line arguments:
{--endPhase=2147483647, --input=/home/ben/Scripts/eipi/termvecs,
--maxIter=-1, --numTopics=2, --numWords=100,
--output=/home/ben/Scripts/eipi/lda_working, --startPhase=0,
--tempDir=temp, --topicSmoothing=-1.0} 11/09/04 16:29:00 INFO lda.LDADriver: LDA Iteration 1 11/09/04 16:29:01 INFO input.FileInputFormat: Total input paths to
process : 4 11/09/04 16:29:01 INFO mapred.JobClient: Cleaning up the staging area
file:/tmp/hadoop-ben/mapred/staging/ben692167368/.staging/job_local_0001 Exception in thread "main" java.io.FileNotFoundException: File
file:/home/ben/Scripts/eipi/termvecs/tokenized-documents/data does not
exist. at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371) at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245) at
org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:63) at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:252) at
...
Does anyone know what I'm doing wrong?
Re: LDA question
Posted by Ben West <bw...@yahoo.com>.
Never mind, I guess the -v parameter is a limit not on the number of words you would like to use, but the number of words which exist in the dictionary.
----- Original Message -----
From: Ben West <bw...@yahoo.com>
To: "user@mahout.apache.org" <us...@mahout.apache.org>
Cc:
Sent: Monday, September 5, 2011 12:34 PM
Subject: Re: LDA question
Thanks Jake! I changed my command to:
$MAHOUT lda -i $BASE_DIR/termvecs/tf-vectors -o $BASE_DIR/lda_working -k 2 -v 10000
And now I get:
11/09/05 12:06:25 WARN mapred.LocalJobRunner: job_local_0001
org.apache.mahout.math.IndexException: Index 10007 is outside allowable range of [0,10000)
at org.apache.mahout.math.AbstractMatrix.get(AbstractMatrix.java:412)
at org.apache.mahout.clustering.lda.LDAState.logProbWordGivenTopic(LDAState.java:45)
at org.apache.mahout.clustering.lda.LDAInference.eStepForWord(LDAInference.java:225)
at org.apache.mahout.clustering.lda.LDAInference.infer(LDAInference.java:110)
at org.apache.mahout.clustering.lda.LDAWordTopicMapper.map(LDAWordTopicMapper.java:48)
at org.apache.mahout.clustering.lda.LDAWordTopicMapper.map(LDAWordTopicMapper.java:36)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
11/09/05 12:06:26 INFO mapred.JobClient: map 0% reduce 0%
11/09/05 12:06:26 INFO mapred.JobClient: Job complete: job_local_0001
11/09/05 12:06:26 INFO mapred.JobClient: Counters: 0
Exception in thread "main" java.lang.InterruptedException: LDA Iteration failed processing /home/ben/Scripts/eipi/lda_working/state-0
at org.apache.mahout.clustering.lda.LDADriver.runIteration(LDADriver.java:427)
at org.apache.mahout.clustering.lda.LDADriver.run(LDADriver.java:226)
at org.apache.mahout.clustering.lda.LDADriver.run(LDADriver.java:174)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.mahout.clustering.lda.LDADriver.main(LDADriver.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:188)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
I reran the seqdirectory and seq2sparse commands and they seemed to work fine, but I keep getting this error. Any idea what I'm doing wrong?
Thanks,
-Ben
----- Original Message -----
From: Jake Mannix <ja...@gmail.com>
To: user@mahout.apache.org; Ben West <bw...@yahoo.com>
Cc:
Sent: Monday, September 5, 2011 11:30 AM
Subject: Re: LDA question
Hi Ben,
On Mon, Sep 5, 2011 at 8:38 AM, Ben West <bw...@yahoo.com> wrote:
>
>
> ~/Scripts/Mahout/trunk/bin/mahout seqdirectory --input
> /home/ben/Scripts/eipi/files --output /home/ben/Scripts/eipi/mahout_out
> -chunk 1
> ~/Scripts/Mahout/trunk/bin/mahout seq2sparse -i
> /home/ben/Scripts/eipi/mahout_out -o /home/ben/Scripts/eipi/termvecs -wt tf
> -seq
>
>
The "output" directory (/home/ben/Scripts/eipi/termvecs) has a bunch of
subdirectories, only one of which actually contains your Vectors. In this
case, you've done tf-normalization, so they're
in /home/ben/Scripts/eipi/termvecs/tf-vectors. This is the directory you
want to give to LDA as input.
-jake
Re: LDA question
Posted by Ben West <bw...@yahoo.com>.
Thanks Jake! I changed my command to:
$MAHOUT lda -i $BASE_DIR/termvecs/tf-vectors -o $BASE_DIR/lda_working -k 2 -v 10000
And now I get:
11/09/05 12:06:25 WARN mapred.LocalJobRunner: job_local_0001
org.apache.mahout.math.IndexException: Index 10007 is outside allowable range of [0,10000)
at org.apache.mahout.math.AbstractMatrix.get(AbstractMatrix.java:412)
at org.apache.mahout.clustering.lda.LDAState.logProbWordGivenTopic(LDAState.java:45)
at org.apache.mahout.clustering.lda.LDAInference.eStepForWord(LDAInference.java:225)
at org.apache.mahout.clustering.lda.LDAInference.infer(LDAInference.java:110)
at org.apache.mahout.clustering.lda.LDAWordTopicMapper.map(LDAWordTopicMapper.java:48)
at org.apache.mahout.clustering.lda.LDAWordTopicMapper.map(LDAWordTopicMapper.java:36)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
11/09/05 12:06:26 INFO mapred.JobClient: map 0% reduce 0%
11/09/05 12:06:26 INFO mapred.JobClient: Job complete: job_local_0001
11/09/05 12:06:26 INFO mapred.JobClient: Counters: 0
Exception in thread "main" java.lang.InterruptedException: LDA Iteration failed processing /home/ben/Scripts/eipi/lda_working/state-0
at org.apache.mahout.clustering.lda.LDADriver.runIteration(LDADriver.java:427)
at org.apache.mahout.clustering.lda.LDADriver.run(LDADriver.java:226)
at org.apache.mahout.clustering.lda.LDADriver.run(LDADriver.java:174)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.mahout.clustering.lda.LDADriver.main(LDADriver.java:90)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:188)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
I reran the seqdirectory and seq2sparse commands and they seemed to work fine, but I keep getting this error. Any idea what I'm doing wrong?
Thanks,
-Ben
----- Original Message -----
From: Jake Mannix <ja...@gmail.com>
To: user@mahout.apache.org; Ben West <bw...@yahoo.com>
Cc:
Sent: Monday, September 5, 2011 11:30 AM
Subject: Re: LDA question
Hi Ben,
On Mon, Sep 5, 2011 at 8:38 AM, Ben West <bw...@yahoo.com> wrote:
>
>
> ~/Scripts/Mahout/trunk/bin/mahout seqdirectory --input
> /home/ben/Scripts/eipi/files --output /home/ben/Scripts/eipi/mahout_out
> -chunk 1
> ~/Scripts/Mahout/trunk/bin/mahout seq2sparse -i
> /home/ben/Scripts/eipi/mahout_out -o /home/ben/Scripts/eipi/termvecs -wt tf
> -seq
>
>
The "output" directory (/home/ben/Scripts/eipi/termvecs) has a bunch of
subdirectories, only one of which actually contains your Vectors. In this
case, you've done tf-normalization, so they're
in /home/ben/Scripts/eipi/termvecs/tf-vectors. This is the directory you
want to give to LDA as input.
-jake
Re: LDA question
Posted by Jake Mannix <ja...@gmail.com>.
Hi Ben,
On Mon, Sep 5, 2011 at 8:38 AM, Ben West <bw...@yahoo.com> wrote:
>
>
> ~/Scripts/Mahout/trunk/bin/mahout seqdirectory --input
> /home/ben/Scripts/eipi/files --output /home/ben/Scripts/eipi/mahout_out
> -chunk 1
> ~/Scripts/Mahout/trunk/bin/mahout seq2sparse -i
> /home/ben/Scripts/eipi/mahout_out -o /home/ben/Scripts/eipi/termvecs -wt tf
> -seq
>
>
The "output" directory (/home/ben/Scripts/eipi/termvecs) has a bunch of
subdirectories, only one of which actually contains your Vectors. In this
case, you've done tf-normalization, so they're
in /home/ben/Scripts/eipi/termvecs/tf-vectors. This is the directory you
want to give to LDA as input.
-jake