[jira] Updated: (MAHOUT-397) SparseVectorsFromSequenceFiles only outputs a single vector file

["Jeff Eastman (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/MAHOUT-397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jeff Eastman updated MAHOUT-397:
--------------------------------

    Attachment: MAHOUT-397.patch

This patch seems to resolve the issue by propagating the number of reducers argument through to the back-end processing steps where the actual output vectors are produced. It also includes a slight modification to SequenceFilesFromDirectory to remove chunk-size upsizing to 64mb which allows Reuters data to be split into 3 smaller files to improve processing. All unit tests run.

Files modified:
M       core/src/main/java/org/apache/mahout/clustering/lda/LDADriver.java
M       utils/src/test/java/org/apache/mahout/utils/vectors/text/DictionaryVectorizerTest.java
M       utils/src/main/java/org/apache/mahout/utils/vectors/text/DictionaryVectorizer.java
M       utils/src/main/java/org/apache/mahout/utils/vectors/common/PartialVectorMerger.java
M       utils/src/main/java/org/apache/mahout/utils/vectors/tfidf/TFIDFConverter.java
M       utils/src/main/java/org/apache/mahout/text/SparseVectorsFromSequenceFiles.java
M       examples/src/main/java/org/apache/mahout/text/SequenceFilesFromDirectory.java
M       examples/bin/build-reuters.sh

The attached build-reuters.sh runs LDA iterations in about 1.5 min vs. 5.5 min with a single vector file on a 3-node cluster using 3 mappers and 2-3 reducers for the vectorization. I will commit it in a day or so but want some more eyeballs on it since this is new code for me.

> SparseVectorsFromSequenceFiles only outputs a single vector file
> ----------------------------------------------------------------
>
>                 Key: MAHOUT-397
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-397
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Utils
>    Affects Versions: 0.3
>            Reporter: Jeff Eastman
>            Assignee: Jeff Eastman
>             Fix For: 0.4
>
>         Attachments: MAHOUT-397.patch
>
>
> When running LDA via build-reuters.sh on a 3-node Hadoop cluster, I've noticed that there is only a single vector file produced by the utility preprocessing steps. This means LDA (and other clustering too) can only use a single mapper no matter how large the cluster is. Investigating, it seems that the program argument (-nr) for setting the number of reducers - and hence the number of output files - is not propagated to the final stages where the output vectors are created.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.