[CONF] Apache Mahout > Wikipedia Bayes Example

[co...@apache.org]

Space: Apache Mahout (https://cwiki.apache.org/confluence/display/MAHOUT)
Page: Wikipedia Bayes Example (https://cwiki.apache.org/confluence/display/MAHOUT/Wikipedia+Bayes+Example)

Change Comment:
---------------------------------------------------------------------
modified for revised instructions based on 0.4 with mahout command line util

Edited by Joe Prasanna Kumar:
---------------------------------------------------------------------
h1. Intro

The Mahout Examples source comes with tools for classifying a Wikipedia data dump using either the Naive Bayes or Complementary Naive Bayes implementations in Mahout.  The example (described below) gets a Wikipedia dump and then splits it up into chunks.  These chunks are then further split by country.  From these splits, a classifier is trained to predict what country an unseen article should be categorized into.


h1. Running the example

# download the wikipedia data set [here | http://download.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2]
# unzip the bz2 file to get the enwiki-latest-pages-articles.xml. 
# Create directory $MAHOUT_HOME/examples/temp and copy the xml file into this directory
# Chunk the Data into pieces: {code}$MAHOUT_HOME/bin/mahout wikipediaXMLSplitter -d $MAHOUT_HOME/examples/temp/enwiki-latest-pages-articles10.xml -o wikipedia/chunks -c 64{code} {quote}*We strongly suggest you backup the results to some other place so that you don't have to do this step again in case it gets accidentally erased*{quote}
# This would have created the chunks in HDFS. Verify the same by executing {code}hadoop fs -ls wikipedia/chunks{code} and it'll list all the xml files as chunk-0001.xml and so on.
# Create the countries based Split of wikipedia dataset. {code}$MAHOUT_HOME/bin/mahout  wikipediaDataSetCreator  -i wikipedia/chunks -o wikipediainput -c $MAHOUT_HOME/examples/src/test/resources/country.txt
{code}
# Verify the creation of input data set by executing {code} hadoop fs -ls wikipediainput {code} and you'll be able to see part-r-00000 file inside wikipediainput directory
# Train the classifier: {code}$MAHOUT_HOME/bin/mahout trainclassifier -i wikipediainput -o wikipediamodel{code}. The model file will be available in the wikipediamodel folder in HDFS.
# Test the classifier: {code}$MAHOUT_HOME/bin/mahout testclassifier -m wikipediamodel -d wikipediainput{code}


Change your notification preferences: https://cwiki.apache.org/confluence/users/viewnotifications.action