Fwd: Mahout dictionary file

[James Ferretti <jx...@cs.bham.ac.uk>]

Hi,
I am trying to perform TFIDF on a set of documents (as text files)
using mahout to do to calculations, following this
guide:
http://www.philippeadjiman.com/blog/2010/12/30/how-to-easily-build-and-observe-tf-idf-weight-vectors-with-lucene-and-mahout/

I
have successfully created the dictionary and vector weights, and am now
attempting to access the output. In the guide it says you "can for
instance
easily load the content of the generated dictionary file into a Map with
token index as keys and the tokens as values."

I am not sure how to go
about loading in this file to a map as he suggests, does anybody know how
it is done?

I created my vectors from a directory of text files, one issue
I encountered when running "./mahout seq2sparse..." was the -a flag that
controls the analyser - which should be lucene's StandardAnalyzer. When
trying to run with this flag I received a ClassNotFoundException, but
removing the flag solved the problem and I think the default analyser is
also this one anyway, therefore the output should be the same as the
example.

If anybody knows how to load this dictionary into a map I will be
eternally grateful!

James

Re: Fwd: Mahout dictionary file

[Ioan Eugen Stan <st...@gmail.com>]

Pe 14.03.2012 19:40, James Ferretti a scris:
> Hi,
> I am trying to perform TFIDF on a set of documents (as text files)
> using mahout to do to calculations, following this
> guide:
> http://www.philippeadjiman.com/blog/2010/12/30/how-to-easily-build-and-observe-tf-idf-weight-vectors-with-lucene-and-mahout/
>
> I
> have successfully created the dictionary and vector weights, and am now
> attempting to access the output. In the guide it says you "can for
> instance
> easily load the content of the generated dictionary file into a Map with
> token index as keys and the tokens as values."
>
> I am not sure how to go
> about loading in this file to a map as he suggests, does anybody know how
> it is done?

Try VectorHelper class:

   public static String[] loadTermDictionary(Configuration conf, String 
filePattern) {
     OpenObjectIntHashMap<String> dict = new OpenObjectIntHashMap<String>();
     for (Pair<Text,IntWritable> record :
          new SequenceFileDirIterable<Text,IntWritable>(new 
Path(filePattern), PathType.GLOB,
              null, null, true, conf)) {
       dict.put(record.getFirst().toString(), record.getSecond().get());
     }
     String[] dictionary = new String[dict.size()];
     for (String feature : dict.keys()) {
       dictionary[dict.get(feature)] = feature;
     }
     return dictionary;
   }

>
> I created my vectors from a directory of text files, one issue
> I encountered when running "./mahout seq2sparse..." was the -a flag that
> controls the analyser - which should be lucene's StandardAnalyzer. When
> trying to run with this flag I received a ClassNotFoundException, but
> removing the flag solved the problem and I think the default analyser is
> also this one anyway, therefore the output should be the same as the
> example.

You will have to add the jar that contains the class to the classpath or 
specify it's fully qualified domain name.

>
> If anybody knows how to load this dictionary into a map I will be
> eternally grateful!
>

Thank you, gratefulness is always appreciated.

> James

Cheers,

-- 
Ioan Eugen Stan
http://ieugen.blogspot.com