You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mahout.apache.org by is...@apache.org on 2013/11/03 22:36:27 UTC
svn commit: r1538467 [20/20] - in /mahout/site/mahout_cms: ./ cgi-bin/
content/ content/css/ content/developers/ content/general/ content/images/
content/js/ content/users/ content/users/basics/
content/users/classification/ content/users/clustering/ c...
Added: mahout/site/mahout_cms/content/users/stuff/hidden-markov-models.mdtext
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/content/users/stuff/hidden-markov-models.mdtext?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/content/users/stuff/hidden-markov-models.mdtext (added)
+++ mahout/site/mahout_cms/content/users/stuff/hidden-markov-models.mdtext Sun Nov 3 21:36:23 2013
@@ -0,0 +1,53 @@
+Title: Hidden Markov Models
+<a name="HiddenMarkovModels-IntroductionandUsage"></a>
+### Introduction and Usage
+
+Hidden Markov Models are used in multiple areas of Machine Learning, such
+as speech recognition, handwritten letter recognition or natural language
+processing.
+
+<a name="HiddenMarkovModels-FormalDefinition"></a>
+### Formal Definition
+
+A Hidden Markov Model (HMM) is a statistical model of a process consisting
+of two (in our case discrete) random variables O and Y, which change their
+state sequentially. The variable Y with states \{y_1, ... , y_n\} is called
+the "hidden variable", since its state is not directly observable. The
+state of Y changes sequentially with a so called - in our case first-order
+- Markov Property. This means, that the state change probability of Y only
+depends on its current state and does not change in time. Formally we
+write: P(Y(t+1)=y_i|Y(0)...Y(t)) = P(Y(t+1)=y_i|Y(t)) = P(Y(2)=y_i|Y(1)).
+The variable O with states \{o_1, ... , o_m\} is called the "observable
+variable", since its state can be directly observed. O does not have a
+Markov Property, but its state probability depends statically on the
+current state of Y.
+
+Formally, an HMM is defined as a tuple M=(n,m,P,A,B), where n is the number of hidden states, m is the number of observable states, P is an n-dimensional vector containing initial hidden state probabilities, A is the nxn-dimensional "transition matrix" containing the transition probabilities such that A\[i,j\](i,j\.html)
+=P(Y(t)=y_i|Y(t-1)=y_j) and B is the mxn-dimensional "emission matrix"
+containing the observation probabilities such that B\[i,j\]=
+P(O=o_i|Y=y_j).
+
+<a name="HiddenMarkovModels-Problems"></a>
+### Problems
+
+Rabiner \[1\](1\.html)
+ defined three main problems for HMM models:
+
+ 1. Evaluation: Given a sequence O of observations and a model M, what is
+the probability P(O|M) that sequence O was generated by model M. The
+Evaluation problem can be efficiently solved using the Forward algorithm
+ 2. Decoding: Given a sequence O of observations and a model M, what is
+the most likely sequence Y*=argmax(Y) P(O|M,Y) of hidden variables to
+generate this sequence. The Decoding problem can be efficiently solved
+using the Viterbi algorithm.
+ 3. Learning: Given a sequence O of observations, what is the most likely
+model M*=argmax(M)P(O|M) to generate this sequence. The Learning problem
+can be efficiently solved using the Baum-Welch algorithm.
+
+<a name="HiddenMarkovModels-Resources"></a>
+### Resources
+
+\[1\](1\.html)
+ Lawrence R. Rabiner (February 1989). "A tutorial on Hidden Markov Models
+and selected applications in speech recognition". Proceedings of the IEEE
+77 (2): 257-286. doi:10.1109/5.18626.
Added: mahout/site/mahout_cms/content/users/stuff/mr---map-reduce.mdtext
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/content/users/stuff/mr---map-reduce.mdtext?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/content/users/stuff/mr---map-reduce.mdtext (added)
+++ mahout/site/mahout_cms/content/users/stuff/mr---map-reduce.mdtext Sun Nov 3 21:36:23 2013
@@ -0,0 +1,13 @@
+Title: MR - Map Reduce
+{excerpt}MapReduce is a framework for processing huge datasets on certain
+kinds of distributable problems using a large number of computers (nodes),
+collectively referred to as a cluster.{excerpt} Computational processing
+can occur on data stored either in a filesystem (unstructured) or within a
+database (structured).
+
+ Also written M/R
+
+
+ See Also
+* [http://wiki.apache.org/hadoop/HadoopMapReduce](http://wiki.apache.org/hadoop/HadoopMapReduce)
+* [http://en.wikipedia.org/wiki/MapReduce](http://en.wikipedia.org/wiki/MapReduce)
Added: mahout/site/mahout_cms/content/users/stuff/parallel-frequent-pattern-mining.mdtext
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/content/users/stuff/parallel-frequent-pattern-mining.mdtext?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/content/users/stuff/parallel-frequent-pattern-mining.mdtext (added)
+++ mahout/site/mahout_cms/content/users/stuff/parallel-frequent-pattern-mining.mdtext Sun Nov 3 21:36:23 2013
@@ -0,0 +1,180 @@
+Title: Parallel Frequent Pattern Mining
+Mahout has a Top K Parallel FPGrowth Implementation. Its based on the paper [http://infolab.stanford.edu/~echang/recsys08-69.pdf](http://infolab.stanford.edu/~echang/recsys08-69.pdf)
+ with some optimisations in mining the data.
+
+Given a huge transaction list, the algorithm finds all unique features(sets
+of field values) and eliminates those features whose frequency in the whole
+dataset is less that minSupport. Using these remaining features N, we find
+the top K closed patterns for each of them, generating a total of NxK
+patterns. FPGrowth Algorithm is a generic implementation, we can use any
+Object type to denote a feature. Current implementation requires you to use
+a String as the object type. You may implement a version for any object by
+creating Iterators, Convertors and TopKPatternWritable for that particular
+object. For more information please refer the package
+org.apache.mahout.fpm.pfpgrowth.convertors.string
+
+ e.g:
+ FPGrowth<String> fp = new FPGrowth<String>();
+ Set<String> features = new HashSet<String>();
+ fp.generateTopKStringFrequentPatterns(
+ new StringRecordIterator(new FileLineIterable(new File(input),
+encoding, false), pattern),
+ fp.generateFList(
+ new StringRecordIterator(new FileLineIterable(new File(input),
+encoding, false), pattern), minSupport),
+ minSupport,
+ maxHeapSize,
+ features,
+ new StringOutputConvertor(new SequenceFileOutputCollector<Text,
+TopKStringPatterns>(writer))
+ );
+
+* The first argument is the iterator of transaction in this case its
+Iterator<List<String>>
+* The second argument is the output of generateFList function, which
+returns the frequent items and their frequencies from the given database
+transaction iterator
+* The third argument is the minimum Support of the pattern to be generated
+* The fourth argument is the maximum number of patterns to be mined for
+each feature
+* The fifth argument is the set of features for which the frequent patterns
+has to be mined
+* The last argument is an output collector which takes \[key, value\](key,-value\.html)
+ of Feature and TopK Patterns of the format \[String,
+List<Pair<List<String>, Long>>\] and writes them to the appropriate writer
+class which takes care of storing the object, in this case in a Sequence
+File Output format
+
+<a name="ParallelFrequentPatternMining-RunningFrequentPatternGrowthviacommandline"></a>
+## Running Frequent Pattern Growth via command line
+
+The command line launcher for string transaction data
+org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver has other features including
+specifying the regex pattern for spitting a string line of a transaction
+into the constituent features.
+
+Input files have to be in the following format.
+
+<optional document id>TAB<TOKEN1>SPACE<TOKEN2>SPACE....
+
+instead of tab you could use , or \| as the default tokenization is done using a java Regex pattern {code}[,\t](,\t.html)
+*[,|\t][ ,\t]*{code}
+You can override this parameter to parse your log files or transaction
+files (each line is a transaction.) The FPGrowth algorithm mines the top K
+frequently occurring sets of items and their counts from the given input
+data
+
+$MAHOUT_HOME/core/src/test/resources/retail.dat is a sample dataset in this
+format.
+Other sample files are accident.dat.gz from [http://fimi.cs.helsinki.fi/data/](http://fimi.cs.helsinki.fi/data/)
+. As a quick test, try this:
+
+
+ bin/mahout fpg \
+ -i core/src/test/resources/retail.dat \
+ -o patterns \
+ -k 50 \
+ -method sequential \
+ -regex '[\ ]
+' \
+ -s 2
+
+
+The minimumSupport parameter \-s is the minimum number of times a pattern
+or a feature needs to occur in the dataset so that it is included in the
+patterns generated. You can speed up the process by having a large value of
+s. There are cases where you will have less than k patterns for a
+particular feature as the rest don't for qualify the minimum support
+criteria
+
+Note that the input to the algorithm, could be uncompressed or compressed
+gz file or even a directory containing any number of such files.
+We modified the regex to use space to split the token. Note that input
+regex string is escaped.
+
+<a name="ParallelFrequentPatternMining-RunningParallelFPGrowth"></a>
+## Running Parallel FPGrowth
+
+Running parallel FPGrowth is as easy as adding changing the flag \-method
+mapreduce and adding the number of groups parameter e.g. \-g 20 for 20
+groups. First, let's run the above sample test in map-reduce mode:
+
+ bin/mahout fpg \
+ -i core/src/test/resources/retail.dat \
+ -o patterns \
+ -k 50 \
+ -method mapreduce \
+ -regex '[\ ]
+' \
+ -s 2
+
+The above test took 102 seconds on dual-core laptop, v.s. 609 seconds in
+the sequential mode, (with 5 gigs of ram allocated). In a separate test,
+the first 1000 lines of retail.dat took 20 seconds in map/reduce v.s. 30
+seconds in sequential mode.
+
+Here is another dataset which, while several times larger, requires much
+less time to find frequent patterns, as there are very few. Get
+accidents.dat.gz from [http://fimi.cs.helsinki.fi/data/](http://fimi.cs.helsinki.fi/data/)
+ and place it on your hdfs in a folder named accidents. Then, run the
+hadoop version of the FPGrowth job:
+
+ bin/mahout fpg \
+ -i accidents \
+ -o patterns \
+ -k 50 \
+ -method mapreduce \
+ -regex '[\ ]
+' \
+ -s 2
+
+
+OR to run a dataset of this size in sequential mode on a single machine
+let's give Mahout a lot more memory and only keep features with more than
+300 members:
+
+ export MAHOUT_HEAPSIZE=-Xmx5000m
+ bin/mahout fpg \
+ -i accidents \
+ -o patterns \
+ -k 50 \
+ -method sequential \
+ -regex '[\ ]
+' \
+ -s 2
+
+
+
+The numGroups parameter \-g in FPGrowthJob specifies the number of groups
+into which transactions have to be decomposed. The default of 1000 works
+very well on a single-machine cluster; this may be very different on large
+clusters.
+
+Note that accidents.dat has 340 unique features. So we chose \-g 10 to
+split the transactions across 10 shards where 34 patterns are mined from
+each shard. (Note: g doesnt need to be exactly divisible.) The Algorithm
+takes care of calculating the split. For better performance in large
+datasets and clusters, try not to mine for more than 20-25 features per
+shard. Stick to the defaults on a small machine.
+
+The numTreeCacheEntries parameter \-tc specifies the number of generated
+conditional FP-Trees to be kept in memory so that subsequent operations do
+not to regenerate them. Increasing this number increases the memory
+consumption but might improve speed until a certain point. This depends
+entirely on the dataset in question. A value of 5-10 is recommended for
+mining up to top 100 patterns for each feature.
+
+<a name="ParallelFrequentPatternMining-Viewingtheresults"></a>
+## Viewing the results
+The output will be dumped to a SequenceFile in the frequentpatterns
+directory in Text=>TopKStringPatterns format. Run this command to see a few
+of the Frequent Patterns:
+
+ bin/mahout seqdumper \
+ -i patterns/frequentpatterns/part-?-00000 \
+ -n 4
+
+or replace -n 4 with -c for the count of patterns.
+
+Open questions: how does one experiment and monitor with these various
+parameters?
Added: mahout/site/mahout_cms/content/users/stuff/partial-implementation.mdtext
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/content/users/stuff/partial-implementation.mdtext?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/content/users/stuff/partial-implementation.mdtext (added)
+++ mahout/site/mahout_cms/content/users/stuff/partial-implementation.mdtext Sun Nov 3 21:36:23 2013
@@ -0,0 +1,136 @@
+Title: Partial Implementation
+<a name="PartialImplementation-Introduction"></a>
+# Introduction
+
+This quick start page shows how to build a decision forest using the
+partial implementation. This tutorial also explains how to use the decision
+forest to classify new data.
+Partial Decision Forests is a mapreduce implementation where each mapper
+builds a subset of the forest using only the data available in its
+partition. This allows building forests using large datasets as long as
+each partition can be loaded in-memory.
+
+<a name="PartialImplementation-Steps"></a>
+# Steps
+<a name="PartialImplementation-Downloadthedata"></a>
+## Download the data
+* The current implementation is compatible with the UCI repository file
+format. In this example we'll use the NSL-KDD dataset because its large
+enough to show the performances of the partial implementation.
+You can download the dataset here http://nsl.cs.unb.ca/NSL-KDD/
+You can either download the full training set "KDDTrain+.ARFF", or a 20%
+subset "KDDTrain+_20Percent.ARFF" (we'll use the full dataset in this
+tutorial) and the test set "KDDTest+.ARFF".
+* Open the train and test files and remove all the lines that begin with
+'@'. All those lines are at the top of the files. Actually you can keep
+those lines somewhere, because they'll help us describe the dataset to
+Mahout
+* Put the data in HDFS: {code}
+$HADOOP_HOME/bin/hadoop fs -mkdir testdata
+$HADOOP_HOME/bin/hadoop fs -put <PATH TO DATA> testdata{code}
+
+<a name="PartialImplementation-BuildtheJobfiles"></a>
+## Build the Job files
+* In $MAHOUT_HOME/ run: {code}mvn clean install -DskipTests{code}
+
+<a name="PartialImplementation-Generateafiledescriptorforthedataset:"></a>
+## Generate a file descriptor for the dataset:
+run the following command:
+
+ $HADOOP_HOME/bin/hadoop jar
+$MAHOUT_HOME/core/target/mahout-core-<VERSION>-job.jar
+org.apache.mahout.classifier.df.tools.Describe -p testdata/KDDTrain+.arff
+-f testdata/KDDTrain+.info -d N 3 C 2 N C 4 N C 8 N 2 C 19 N L
+
+The "N 3 C 2 N C 4 N C 8 N 2 C 19 N L" string describes all the attributes
+of the data. In this cases, it means 1 numerical(N) attribute, followed by
+3 Categorical(C) attributes, ...L indicates the label. You can also use 'I'
+to ignore some attributes
+
+<a name="PartialImplementation-Runtheexample"></a>
+## Run the example
+
+
+ $HADOOP_HOME/bin/hadoop jar
+$MAHOUT_HOME/examples/target/mahout-examples-<version>-job.jar
+org.apache.mahout.classifier.df.mapreduce.BuildForest
+-Dmapred.max.split.size=1874231 -d testdata/KDDTrain+.arff -ds
+testdata/KDDTrain+.info -sl 5 -p -t 100 -o nsl-forest
+
+which builds 100 trees (-t argument) using the partial implementation (-p).
+Each tree is built using 5 random selected attribute per node (-sl
+argument) and the example outputs the decision tree in the "nsl-forest"
+directory (-o).
+The number of partitions is controlled by the -Dmapred.max.split.size
+argument that indicates to Hadoop the max. size of each partition, in this
+case 1/10 of the size of the dataset. Thus 10 partitions will be used.
+IMPORTANT: using less partitions should give better classification results,
+but needs a lot of memory. So if the Jobs are failing, try increasing the
+number of partitions.
+* The example outputs the Build Time and the oob error estimation
+
+
+ 10/03/13 17:57:29 INFO mapreduce.BuildForest: Build Time: 0h 7m 43s 582
+ 10/03/13 17:57:33 INFO mapreduce.BuildForest: oob error estimate :
+0.002325895231517865
+ 10/03/13 17:57:33 INFO mapreduce.BuildForest: Storing the forest in:
+nsl-forest/forest.seq
+
+
+<a name="PartialImplementation-UsingtheDecisionForesttoClassifynewdata"></a>
+## Using the Decision Forest to Classify new data
+run the following command:
+
+ $HADOOP_HOME/bin/hadoop jar
+$MAHOUT_HOME/examples/target/mahout-examples-<version>-job.jar
+org.apache.mahout.classifier.df.mapreduce.TestForest -i
+nsl-kdd/KDDTest+.arff -ds nsl-kdd/KDDTrain+.info -m nsl-forest -a -mr -o
+predictions
+
+This will compute the predictions of "KDDTest+.arff" dataset (-i argument)
+using the same data descriptor generated for the training dataset (-ds) and
+the decision forest built previously (-m). Optionally (if the test dataset
+contains the labels of the tuples) run the analyzer to compute the
+confusion matrix (-a), and you can also store the predictions in a text
+file or a directory of text files(-o). Passing the (-mr) parameter will use
+Hadoop to distribute the classification.
+
+* The example should output the classification time and the confusion
+matrix
+
+
+ 10/03/13 18:08:56 INFO mapreduce.TestForest: Classification Time: 0h 0m 6s
+355
+ 10/03/13 18:08:56 INFO mapreduce.TestForest:
+=======================================================
+ Summary
+ -------------------------------------------------------
+ Correctly Classified Instances : 17657 78.3224%
+ Incorrectly Classified Instances : 4887 21.6776%
+ Total Classified Instances : 22544
+
+ =======================================================
+ Confusion Matrix
+ -------------------------------------------------------
+ a b <--Classified as
+ 9459 252 | 9711 a = normal
+ 4635 8198 | 12833 b = anomaly
+ Default Category: unknown: 2
+
+
+If the input is a single file then the output will be a single text file,
+in the above example 'predictions' would be one single file. If the input
+if a directory containing for example two files 'a.data' and 'b.data', then
+the output will be a directory 'predictions' containing two files
+'a.data.out' and 'b.data.out'
+
+<a name="PartialImplementation-KnownIssuesandlimitations"></a>
+## Known Issues and limitations
+The "Decision Forest" code is still "a work in progress", many features are
+still missing. Here is a list of some known issues:
+* For now, the training does not support multiple input files. The input
+dataset must be one single file. Classifying new data does support multiple
+input files.
+* The tree building is done when each mapper.close() method is called.
+Because the mappers don't refresh their state, the job can fail when the
+dataset is big and you try to build a large number of trees.
Added: mahout/site/mahout_cms/content/users/stuff/perceptron-and-winnow.mdtext
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/content/users/stuff/perceptron-and-winnow.mdtext?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/content/users/stuff/perceptron-and-winnow.mdtext (added)
+++ mahout/site/mahout_cms/content/users/stuff/perceptron-and-winnow.mdtext Sun Nov 3 21:36:23 2013
@@ -0,0 +1,36 @@
+Title: Perceptron and Winnow
+<a name="PerceptronandWinnow-ClassificationwithPerceptronorWinnow"></a>
+# Classification with Perceptron or Winnow
+
+Both algorithms are comparably simple linear classifiers. Given training
+data in some n-dimensional vector space that is annotated with binary
+labels the algorithms are guaranteed to find a linear separating hyperplane
+if one exists. In contrast to the Perceptron, Winnow works only for binary
+feature vectors.
+
+For more information on the Perceptron see for instance:
+http://en.wikipedia.org/wiki/Perceptron
+
+Concise course notes on both algorithms:
+http://pages.cs.wisc.edu/~shuchi/courses/787-F07/scribe-notes/lecture24.pdf
+
+Although the algorithms are comparably simple they still work pretty well
+for text classification and are fast to train even for huge example sets.
+In contrast to Naive Bayes they are not based on the assumption that all
+features (in the domain of text classification: all terms in a document)
+are independent.
+
+<a name="PerceptronandWinnow-Strategyforparallelisation"></a>
+## Strategy for parallelisation
+
+Currently the strategy for parallelisation is simple: Given there is enough
+training data, split the training data. Train the classifier on each split.
+The resulting hyperplanes are then averaged.
+
+<a name="PerceptronandWinnow-Roadmap"></a>
+## Roadmap
+
+Currently the patch only contains the code for the classifier itself. It is
+planned to provide unit tests and at least one example based on the WebKB
+dataset by the end of November for the serial version. After that the
+parallelisation will be added.
Added: mahout/site/mahout_cms/content/users/stuff/testing.mdtext
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/content/users/stuff/testing.mdtext?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/content/users/stuff/testing.mdtext (added)
+++ mahout/site/mahout_cms/content/users/stuff/testing.mdtext Sun Nov 3 21:36:23 2013
@@ -0,0 +1,41 @@
+Title: Testing
+<a name="Testing-Intro"></a>
+# Intro
+
+As Mahout matures, solid testing procedures are needed. This page and its
+children capture test plans along with ideas for improving our testing.
+
+<a name="Testing-TestPlans"></a>
+# Test Plans
+
+* [0.6](0.6.html)
+ - Test Plans for the 0.6 release
+There are no special plans except for unit tests, and user testing of the
+Hadoop jobs.
+
+<a name="Testing-TestIdeas"></a>
+# Test Ideas
+
+<a name="Testing-Regressions/Benchmarks/Integrations"></a>
+## Regressions/Benchmarks/Integrations
+* Algorithmic quality and speed are not tested, except in a few instances.
+Such tests often require much longer run times (minutes to hours), a
+running Hadoop cluster, and downloads of large datasets (in the megabytes).
+* Standardized speed tests are difficult on different hardware.
+* Unit tests of external integrations require access to externals: HDFS,
+S3, JDBC, Cassandra, etc.
+
+Apache Jenkins is not able to support these environments. Commercial
+donations would help.
+
+<a name="Testing-UnitTests"></a>
+## Unit Tests
+Mahout's current tests are almost entirely unit tests. Algorithm tests
+generally supply a few numbers to code paths and verify that expected
+numbers come out. 'mvn test' runs these tests. There is "positive" coverage
+of a great many utilities and algorithms. A much smaller percent include
+"negative" coverage (bogus setups, inputs, combinations).
+
+<a name="Testing-Other"></a>
+## Other
+
Added: mahout/site/mahout_cms/content/users/stuff/traveling-salesman.mdtext
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/content/users/stuff/traveling-salesman.mdtext?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/content/users/stuff/traveling-salesman.mdtext (added)
+++ mahout/site/mahout_cms/content/users/stuff/traveling-salesman.mdtext Sun Nov 3 21:36:23 2013
@@ -0,0 +1,24 @@
+Title: Traveling Salesman
+<a name="TravelingSalesman-Intro"></a>
+# Intro
+
+The Traveling Salesman Problem (TSP) is a classic computer science question
+classified as NP-Hard. See
+http://en.wikipedia.org/wiki/Travelling_salesman_problem for background
+information.
+
+<a name="TravelingSalesman-EvolutionaryExample"></a>
+# Evolutionary Example
+
+As an example of evolutionary programming, Mahout has an example
+implementation that attempts to solve TSP. It should be noted that the
+implementation of evolutionary programming in Mahout has not been
+maintained for some time and may be removed in the future due to lack of
+interest.
+
+To run the example, do:
+
+1. cd <MAHOUT_HOME>
+1. mvn install
+1. Run the Job: {code}./bin/mahout
+org.apache.mahout.ga.watchmaker.travellingsalesman.TravellingSalesman{code}
Added: mahout/site/mahout_cms/content/users/stuff/using-mahout-with-python-via-jpype.mdtext
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/content/users/stuff/using-mahout-with-python-via-jpype.mdtext?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/content/users/stuff/using-mahout-with-python-via-jpype.mdtext (added)
+++ mahout/site/mahout_cms/content/users/stuff/using-mahout-with-python-via-jpype.mdtext Sun Nov 3 21:36:23 2013
@@ -0,0 +1,228 @@
+Title: Using Mahout with Python via JPype
+<a name="UsingMahoutwithPythonviaJPype-overview"></a>
+# overview
+This tutorial provides some sample code illustrating how we can read and
+write sequence files containing Mahout vectors from Python using JPype.
+This tutorial is intended for people who want to use Python for analyzing
+and plotting Mahout data. Using Mahout from Python turns out to be quite
+easy.
+
+This tutorial concerns the use of cPython (cython) as opposed to Jython.
+JPython wasn't an option for me, because (to the best of my knowledge)
+JPython doesn't work with Python extensions numpy, matplotlib, or h5py
+which I rely on heavily.
+
+The instructions below explain how to setup a python script to read and
+write the output of Mahout clustering.
+
+You will first need to download and install the JPype package for python.
+
+The first step to setting up JPype is determining the path to the dynamic
+library for the jvm ; on linux this will be a .so file on and on windows it
+will be a .dll.
+
+In your python script, create a global variable with the path to this dll
+
+
+
+Next we need to figure out how we need to set the classpath for mahout. The
+easiest way to do this is to edit the script in "bin/mahout" to print out
+the classpath. Add the line "echo $CLASSPATH" to the script somewhere after
+the comment "run it" (this is line 195 or so). Execute the script to print
+out the classpath. Copy this output and paste it into a variable in your
+python script. The result for me looks like the following
+
+
+
+
+Now we can create a function to start the jvm in python using jype
+
+ jvm=None
+ def start_jpype():
+ global jvm
+ if (jvm is None):
+ cpopt="-Djava.class.path={cp}".format(cp=classpath)
+ startJVM(jvmlib,"-ea",cpopt)
+ jvm="started"
+
+
+
+<a name="UsingMahoutwithPythonviaJPype-WritingNamedVectorstoSequenceFilesfromPython"></a>
+# Writing Named Vectors to Sequence Files from Python
+We can now use JPype to create sequence files which will contain vectors to
+be used by Mahout for kmeans. The example below is a function which creates
+vectors from two Gaussian distributions with unit variance.
+
+
+ def create_inputs(ifile,*args,**param):
+ """Create a sequence file containing some normally distributed
+ ifile - path to the sequence file to create
+ """
+
+ #matrix of the cluster means
+ cmeans=np.array([[1,1]
+,[-1,-1]],np.int)
+
+ nperc=30 #number of points per cluster
+
+ vecs=[]
+
+ vnames=[]
+ for cind in range(cmeans.shape[0]
+):
+ pts=np.random.randn(nperc,2)
+ pts=pts+cmeans[cind,:]
+.reshape([1,cmeans.shape[1]])
+ vecs.append(pts)
+
+ #names for the vectors
+ #names are just the points with an index
+ #we do this so we can validate by cross-refencing the name with the
+vector
+ vn=np.empty(nperc,dtype=(np.str,30))
+ for row in range(nperc):
+ vn[row]
+="c"+str(cind)+"_"+pts[row,0].astype((np.str,4))+"_"+pts[row,1].astype((np.str,4))
+ vnames.append(vn)
+
+ vecs=np.vstack(vecs)
+ vnames=np.hstack(vnames)
+
+
+ #start the jvm
+ start_jpype()
+
+ #create the sequence file that we will write to
+ io=JPackage("org").apache.hadoop.io
+ FileSystemCls=JPackage("org").apache.hadoop.fs.FileSystem
+
+ PathCls=JPackage("org").apache.hadoop.fs.Path
+ path=PathCls(ifile)
+
+ ConfCls=JPackage("org").apache.hadoop.conf.Configuration
+ conf=ConfCls()
+
+ fs=FileSystemCls.get(conf)
+
+ #vector classes
+ VectorWritableCls=JPackage("org").apache.mahout.math.VectorWritable
+ DenseVectorCls=JPackage("org").apache.mahout.math.DenseVector
+ NamedVectorCls=JPackage("org").apache.mahout.math.NamedVector
+ writer=io.SequenceFile.createWriter(fs, conf, path,
+io.Text,VectorWritableCls)
+
+
+ vecwritable=VectorWritableCls()
+ for row in range(vecs.shape[0]
+):
+ nvector=NamedVectorCls(DenseVectorCls(JArray(JDouble,1)(vecs[row,:]
+)),vnames[row])
+ #need to wrap key and value because of overloading
+ wrapkey=JObject(io.Text("key "+str(row)),io.Writable)
+ wrapval=JObject(vecwritable,io.Writable)
+
+ vecwritable.set(nvector)
+ writer.append(wrapkey,wrapval)
+
+ writer.close()
+
+
+<a name="UsingMahoutwithPythonviaJPype-ReadingtheKMeansClusteredPointsfromPython"></a>
+# Reading the KMeans Clustered Points from Python
+Similarly we can use JPype to easily read the clustered points outputted by
+mahout.
+
+ def read_clustered_pts(ifile,*args,**param):
+ """Read the clustered points
+ ifile - path to the sequence file containing the clustered points
+ """
+
+ #start the jvm
+ start_jpype()
+
+ #create the sequence file that we will write to
+ io=JPackage("org").apache.hadoop.io
+ FileSystemCls=JPackage("org").apache.hadoop.fs.FileSystem
+
+ PathCls=JPackage("org").apache.hadoop.fs.Path
+ path=PathCls(ifile)
+
+ ConfCls=JPackage("org").apache.hadoop.conf.Configuration
+ conf=ConfCls()
+
+ fs=FileSystemCls.get(conf)
+
+ #vector classes
+ VectorWritableCls=JPackage("org").apache.mahout.math.VectorWritable
+ NamedVectorCls=JPackage("org").apache.mahout.math.NamedVector
+
+
+ ReaderCls=io.__getattribute__("SequenceFile$Reader")
+ reader=ReaderCls(fs, path,conf)
+
+
+ key=reader.getKeyClass()()
+
+
+ valcls=reader.getValueClass()
+ vecwritable=valcls()
+ while (reader.next(key,vecwritable)):
+ weight=vecwritable.getWeight()
+ nvec=vecwritable.getVector()
+
+ cname=nvec.__class__.__name__
+ if (cname.rsplit('.',1)[1]
+=="NamedVector"):
+ print "cluster={key} Name={name} x={x}
+y={y}".format(key=key.toString(),name=nvec.getName(),x=nvec.get(0),y=nvec.get(1))
+ else:
+ raise NotImplementedError("Vector isn't a NamedVector. Need to
+modify/test the code to handle this case.")
+
+
+<a name="UsingMahoutwithPythonviaJPype-ReadingtheKMeansCentroids"></a>
+# Reading the KMeans Centroids
+Finally we can create a function to print out the actual cluster centers
+found by mahout,
+
+ def getClusters(ifile,*args,**param):
+ """Read the centroids from the clusters outputted by kmenas
+ ifile - Path to the sequence file containing the centroids
+ """
+
+ #start the jvm
+ start_jpype()
+
+ #create the sequence file that we will write to
+ io=JPackage("org").apache.hadoop.io
+ FileSystemCls=JPackage("org").apache.hadoop.fs.FileSystem
+
+ PathCls=JPackage("org").apache.hadoop.fs.Path
+ path=PathCls(ifile)
+
+ ConfCls=JPackage("org").apache.hadoop.conf.Configuration
+ conf=ConfCls()
+
+ fs=FileSystemCls.get(conf)
+
+ #vector classes
+ VectorWritableCls=JPackage("org").apache.mahout.math.VectorWritable
+ NamedVectorCls=JPackage("org").apache.mahout.math.NamedVector
+ ReaderCls=io.__getattribute__("SequenceFile$Reader")
+ reader=ReaderCls(fs, path,conf)
+
+
+ key=io.Text()
+
+
+ valcls=reader.getValueClass()
+
+ vecwritable=valcls()
+
+ while (reader.next(key,vecwritable)):
+ center=vecwritable.getCenter()
+
+ print "id={cid}
+center={center}".format(cid=vecwritable.getId(),center=center.values)
+ pass
+
Added: mahout/site/mahout_cms/templates/standard.html
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/templates/standard.html?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/templates/standard.html (added)
+++ mahout/site/mahout_cms/templates/standard.html Sun Nov 3 21:36:23 2013
@@ -0,0 +1,414 @@
+<!DOCTYPE html>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
+ <title>Apache Mahout: Scalable machine learning and data mining</title>
+ <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
+ <meta name="Distribution" content="Global">
+ <meta name="Robots" content="index,follow">
+ <meta name="keywords" content="apache, apache hadoop, apache lucene,
+ business data mining, cluster analysis,
+ collaborative filtering, data extraction, data filtering, data framework, data integration,
+ data matching, data mining, data mining algorithms, data mining analysis, data mining data,
+ data mining introduction, data mining software,
+ data mining techniques, data representation, data set, datamining,
+ feature extraction, fuzzy k means, genetic algorithm, hadoop,
+ hierarchical clustering, high dimensional, introduction to data mining, kmeans,
+ knowledge discovery, learning approach, learning approaches, learning methods,
+ learning techniques, lucene, machine learning, machine translation, mahout apache,
+ mahout taste, map reduce hadoop, mining data, mining methods, naive bayes,
+ natural language processing,
+ supervised, text mining, time series data, unsupervised, web data mining">
+ <link rel="shortcut icon" type="image/x-icon" href="http://mahout.apache.org/images/favicon.ico">
+ <script id="twitter-wjs" src="/~isabel/content/js/widgets.js"></script>
+ <script type="text/javascript" src="/~isabel/content/js/prototype.js"></script>
+ <script type="text/javascript" src="/~isabel/content/js/effects.js"></script>
+ <script type="text/javascript" src="/~isabel/content/js/search.js"></script>
+ <script type="text/javascript" src="/~isabel/content/js/slides.js"></script>
+
+ <link href="/~isabel/content/css/bootstrap.min.css" rel="stylesheet" media="screen">
+ <link href="/~isabel/content/css/bootstrap-responsive.css" rel="stylesheet">
+ <link rel="stylesheet" href="/~isabel/content/css/global.css" type="text/css">
+</head>
+
+<body id="home" data-twttr-rendered="true">
+ <div id="wrap">
+ <div id="header">
+ <div id="logo"><a href="/~isabel/content/overview.html"></a></div>
+ <div id="search">
+ <script type="text/javascript">
+ function getBlank(input, stdValue) {
+ if (input.value == stdValue) {
+ input.value = '';
+ }
+ return true;
+ }
+
+ function selectProvider(form) {
+ provider = form.elements['searchProvider'].value;
+ if (provider == "any") {
+ if (Math.random() > 0.5) {
+ provider = "lucid";
+ } else {
+ provider = "sl";
+ }
+ }
+
+ if (provider == "lucid") {
+ form.action = "http://search.lucidimagination.com/p:mahout";
+ } else if (provider == "sl") {
+ form.action = "http://search-lucene.com/mahout";
+ }
+
+ days = 90; // cookie will be valid for 90 days
+ date = new Date();
+ date.setTime(date.getTime() + (days * 24 * 60 * 60 * 1000));
+ expires = "; expires=" + date.toGMTString();
+ document.cookie = "searchProvider=" + provider + expires + "; path=/";
+ return true;
+ }
+ </script>
+ <form id="quick-search" method="GET" onsubmit="return selectProvider(this)" action="http://search-lucene.com/mahout" name="searchform">
+ <fieldset>
+ <input type="search" name="q" value="Search with Apache Solr..." class="class1 class2 hint" accesskey="q" onfocus="getBlank(this, 'Search with Apache Solr...')">
+ <span style="color:white">@</span>
+ <select name="searchProvider" id="searchProvider">
+ <option value="any">select provider</option>
+ <option value="lucid">Lucid Find</option>
+ <option value="sl">Search-Lucene</option>
+ </select>
+ </fieldset>
+ <script type="text/javascript">
+ if (document.cookie.length>0) {
+ cStart=document.cookie.indexOf("searchProvider=");
+ if (cStart!=-1) {
+ cStart=cStart + "searchProvider=".length;
+ cEnd=document.cookie.indexOf(";", cStart);
+ if (cEnd==-1) {
+ cEnd=document.cookie.length;
+ }
+ provider = unescape(document.cookie.substring(cStart,cEnd));
+ document.forms['searchform'].elements['searchProvider'].value = provider;
+ }
+ }
+ </script>
+ </form>
+ </div>
+
+ <div class="navbar navbar-inverse" style="position:absolute;padding-right:5px;padding-left:5px;top:84px;right:0">
+ <div class="navbar-inner">
+ <div class="container">
+ <button type="button" class="btn btn-navbar" data-toggle="collapse" data-target=".nav-collapse">
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ <span class="icon-bar"></span>
+ </button>
+ <!-- <a class="brand" href="#">Apache Community Development Project</a> -->
+ <div class="nav-collapse collapse">
+ <ul class="nav">
+ <li><a href="/">Home</a></li>
+ <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">General<b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/~isabel/content/general/downloads.html">Downloads</a>
+ <li><a href="/~isabel/content/general/who-we-are.html">Who we are</a>
+ <li><a href="/~isabel/content/general/mailing-lists,-irc-and-archives.html">Contact</a>
+ <li><a href="/~isabel/content/general/books-tutorials-and-talks.html">Books, Tutorials, Talks</a></li>
+ <li><a href="/~isabel/content/general/powered-by-mahout.html">Powered By Mahout</a>
+ <li><a href="/~isabel/content/general/professional-support.html">Professional Support</a>
+ <li><a href="/~isabel/content/general/mahout-wiki.html">Mahout Wiki</a>
+ <li class="divider"></li>
+ <li class="nav-header">Resources</li>
+ <li><a href="/~isabel/content/general/mahout-benchmarks.html">Benchmarks</a>
+ <li><a href="/~isabel/content/general/reference-reading.html">Reference Reading</a>
+ <li><a href="/~isabel/content/general/faq.html">FAQ</a>
+ <li><a href="/~isabel/content/general/glossary.html">Glossary</a></li>
+ <li class="divider"></li>
+ <li class="nav-header">Legal</li>
+ <li><a href="http://www.apache.org/licenses/">License</a></li>
+ <li><a href="http://www.apache.org/security/">Security</a></li>
+ <li><a href="/~isabel/content/general/privacy-policy.html">Privacy Policy</a>
+ </ul>
+ </li>
+ <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Developers<b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/~isabel/content/developers/developer-resources.html">Developer resources</a></li>
+ <li><a href="/~isabel/content/developers/version-control.html">Version control</a></li>
+ <li><a href="/~isabel/content/developers/buildingmahout.html">Build from source</a></li>
+ <li><a href="/~isabel/content/developers/issue-tracker.html">Issue tracker</a></li>
+ <li><a href="https://builds.apache.org/hudson/job/Mahout-Quality/clover/" target="_blank">Code quality reports</a></li>
+ <li class="divider"></li>
+ <li class="nav-header">Contributions</li>
+ <li><a href="/~isabel/content/developers/how-to-contribute.html">How to contribute</a></li>
+ <li><a href="/~isabel/content/developers/how-to-become-a-committer.html">How to become a committer</a></li>
+ <li><a href="/~isabel/content/developers/gsoc.html">GSoC</a></li>
+ <li class="divider"></li>
+ <li class="nav-header">For committers</li>
+ <li><a href="/~isabel/content/developers/how-to-update-the-website.html">How to update the website</a></li>
+ <li><a href="/~isabel/content/developers/patch-check-list.html">Patch check list</a></li>
+ <li><a href="/~isabel/content/developers/how-to-release.html">How to release</a></li>
+ <li><a href="/~isabel/content/developers/thirdparty-dependencies.html">Third party dependencies</a></li>
+ </ul>
+ </li>
+ <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Basics<b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/~isabel/content/users/basics/algorithms.html">List of algorithms</a>
+ <li><a href="/~isabel/content/users/basics/quickstart.html">Quickstart</a>
+ <li><a href="/~isabel/content/users/basics/system-requirements.html">System requirements</a>
+ <li><a href="/~isabel/content/users/basics/mahoutintegration.html">Integration</a>
+ <li class="divider"></li>
+ <li class="nav-header">Collections</li>
+
+ <li><a href="/~isabel/content/users/basics/mahout-collections.html">Mahout collections</a>
+ <li><a href="/~isabel/content/users/basics/collections.html">Mahout collections</a>
+ <li><a href="/~isabel/content/users/basics/matrix-and-vector-needs.html">Matrix/Vector needs</a>
+ <li class="divider"></li>
+ <li class="nav-header">Creating vectors</li>
+ <li><a href="/~isabel/content/users/basics/creating-vectors.html">General vector creation</a>
+ <li><a href="/~isabel/content/users/basics/creating-vectors-from-text.html">Creating vectors from text</a>
+ <li><a href="/~isabel/content/users/basics/tf-idf---term-frequency-inverse-document-frequency.html">TF-IDF</a>
+ <li class="divider"></li>
+ <li class="nav-header">Other</li>
+ <li><a href="/~isabel/content/users/basics/collocations.html">Co-Locations</a>
+ <li><a href="/~isabel/content/users/basics/dimensional-reduction.html">Dimensional reduction</a>
+ <li><a href="/~isabel/content/users/basics/gaussian-discriminative-analysis.html">Gaussian discriminative analysis</a>
+ <li><a href="/~isabel/content/users/basics/independent-component-analysis.html">Independent component analysis</a>
+ <li><a href="/~isabel/content/users/basics/mahout.ga.tutorial.html">GA tutorial</a>
+ <li><a href="/~isabel/content/users/basics/principal-components-analysis.html">Principal components analysis</a>
+ <li><a href="/~isabel/content/users/basics/svd---singular-value-decomposition.html">SVD</a>
+ </ul>
+ </li>
+ <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Classification<b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/~isabel/content/users/classification/classifyingyourdata.html">Classifying data</a></li>
+ <li><a href="/~isabel/content/users/classification/naivebayes.html">Design naive bayes</a></li>
+ <li><a href="/~isabel/content/users/classification/complementary-naive-bayes.html">Design complimentary bayes</a></li>
+ <li><a href="/~isabel/content/users/classification/bayesian-commandline.html">Bayes command line</a></li>
+ <li><a href="/~isabel/content/users/classification/bayesian.html">Bayes design doc</a></li>
+ <li><a href="/~isabel/content/users/classification/wikipedia-bayes-example.html">Wikipedia Example</a></li>
+ <li><a href="/~isabel/content/users/clustering/20newsgroups.html">20 newsgroups example</a></li>
+ <li><a href="/~isabel/content/users/clustering/twenty-newsgroups.html">20 newsgroups example</a></li>
+ <li><a href="/~isabel/content/users/classification/breiman-example.html">Breiman example</a></li>
+ <li><a href="/~isabel/content/users/classification/class-discovery.html">Class discovery</a></li>
+ <li><a href="/~isabel/content/users/classification/logistic-regression.html">Design logisitic regression</a></li>
+ <li class="divider"></li>
+ <li class="nav-header">Open designs</li>
+ <li><a href="/~isabel/content/users/classification/locally-weighted-linear-regression.html">Design locally weighted regression</a></li>
+ <li><a href="/~isabel/content/users/classification/neural-network.html">Design neural networks</a></li>
+ <li><a href="/~isabel/content/users/classification/random-forests.html">Design random forests</a></li>
+ <li><a href="/~isabel/content/users/classification/restricted-boltzmann-machines.html">Design restricted boltzmann machines</a></li>
+ <li><a href="/~isabel/content/users/classification/support-vector-machines.html">Design SVM</a></li>
+ </ul></li>
+ <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Clustering<b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/~isabel/content/users/clustering/clusteringyourdata.html">Overview</a></li>
+ <li><a href="/~isabel/content/users/clustering/cluster-dumper.html">Cluster dumper tool</a></li>
+ <li class="divider"></li>
+ <li class="nav-header">Algorithms</li>
+ <li><a href="/~isabel/content/users/clustering/k-means-clustering.html">Design k-Means clustering</a></li>
+ <li><a href="/~isabel/content/users/clustering/k-means-commandline.html">Commandline k-Means</a></li>
+ <li><a href="/~isabel/content/users/clustering/canopy-clustering.html">Design Canopy clustering</a></li>
+ <li><a href="/~isabel/content/users/clustering/canopy-commandline.html">Commandline Canopy clustering</a></li>
+ <li><a href="/~isabel/content/users/clustering/dirichlet-process-clustering.html">Design Dirichlet clustering</a></li>
+ <li><a href="/~isabel/content/users/clustering/dirichlet-commandline.html">Commandline Dirichlet clustering</a></li>
+ <li><a href="/~isabel/content/users/clustering/fuzzy-k-means.html">Design Fuzzy k-Means</a></li>
+ <li><a href="/~isabel/content/users/clustering/fuzzy-k-means-commandline.html">Commandline Fuzzy k-Means</a></li>
+ <li><a href="/~isabel/content/users/clustering/latent-dirichlet-allocation.html">Design LDA</a></li>
+ <li><a href="/~isabel/content/users/clustering/lda-commandline.html">Commandline LDA</a></li>
+ <li><a href="/~isabel/content/users/clustering/mean-shift-clustering.html">Design Meanshift</a></li>
+ <li><a href="/~isabel/content/users/clustering/mean-shift-commandline.html">Commandline Meanshift</a></li>
+ <li class="divider"></li>
+ <li class="nav-header">Examples</li>
+ <li><a href="/~isabel/content/users/clustering/clustering-of-synthetic-control-data.html">Example: Synthetic data</a></li>
+ <li><a href="/~isabel/content/users/clustering/clustering-seinfeld-episodes.html">Example: Seinfeld</a></li>
+ <li class="divider"></li>
+ <li class="nav-header">Post processing</li>
+ <li><a href="/~isabel/content/users/clustering/viewing-result.html">Viewing results, part a</a></li>
+ <li><a href="/~isabel/content/users/clustering/viewing-results.html">Viewing results, part b</a></li>
+ <li><a href="/~isabel/content/users/clustering/visualizing-sample-clusters.html">Cluster visualisation</a></li>
+ <li class="divider"></li>
+ <li class="nav-header">Other designs</li>
+ <li><a href="/~isabel/content/users/clustering/expectation-maximization.html">Design EM</a></li>
+ <li><a href="/~isabel/content/users/clustering/hierarchical-clustering.html">Design hierarchical</a></li>
+ <li><a href="/~isabel/content/users/clustering/llr---log-likelihood-ratio.html">Design LLR</a></li>
+ <li><a href="/~isabel/content/users/clustering/spectral-clustering.html">Design Spectral</a></li>
+ </ul></li>
+ <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Recommendations<b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/~isabel/content/users/recommender/recommender-documentation.html">General</a></li>
+ <li><a href="/~isabel/content/users/recommender/recommendationexamples.html">Example</a></li>
+ <li><a href="/~isabel/content/users/recommender/tastecommandline.html">Taste Commandline</a></li>
+ <li><a href="/~isabel/content/users/recommender/recommender-first-timer-faq.html">First Timer FAQ</a></li>
+ <li><a href="/~isabel/content/users/recommender/itembased-collaborative-filtering.html">Item based collaborative filtering</a></li>
+ <li><a href="/~isabel/content/users/recommender/pearsoncorrelation.html">Pearson Correlation</a></li>
+ <li class="divider"></li>
+ <li class="nav-header">Recommenders on EMR</li>
+ <li><a href="/~isabel/content/users/emr/mahout-on-amazon-ec2.html">Mahout on EC2</a></li>
+ <li><a href="/~isabel/content/users/emr/mahout-on-elastic-mapreduce.html">Mahout on EMR</a></li>
+ <li><a href="/~isabel/content/users/emr/use-an-existing-hadoop-ami.html">Mahout on existing Hadoop AMI</a></li>
+ </ul></li>
+ <li class="dropdown"> <a href="#" class="dropdown-toggle" data-toggle="dropdown">Stuff user docs<b class="caret"></b></a>
+ <ul class="dropdown-menu">
+ <li><a href="/~isabel/content/users/stuff/hidden-markov-models.html">HMM</a></li>
+ <li><a href="/~isabel/content/users/stuff/mr---map-reduce.html">Map Reduce</a></li>
+ <li><a href="/~isabel/content/users/stuff/parallel-frequent-pattern-mining.html">Frequent pattern mining</a></li>
+ <li><a href="/~isabel/content/users/stuff/partial-implementation.html">Partial Impl</a></li>
+ <li><a href="/~isabel/content/users/stuff/perceptron-and-winnow.html">Perceptron/Winnow</a></li>
+ <li><a href="/~isabel/content/users/stuff/testing.html">Testing</a></li>
+ <li><a href="/~isabel/content/users/stuff/traveling-salesman.html">Traveling Salesman</a></li>
+ <li><a href="/~isabel/content/users/stuff/using-mahout-with-python-via-jpype.html">Mahout from Jython</a></li>
+ </ul></li>
+ </ul>
+ </div><!--/.nav-collapse -->
+ </div>
+ </div>
+ </div>
+ <div id="slides">
+ <div class="slideshow" id="mantleSlides">
+ <div id="slide1" style="display: none;">
+ <div class="callout">
+ Scalable machine learning<br>
+ and data mining
+ </div>
+ <div class="description">
+ Apache Mahout has implementations of a wide range of machine learning and data mining algorithms: clustering,
+ classification, collaborative filtering and frequent pattern mining
+ </div>
+ <div class="mantle">
+ <img alt="Apache Mahout" src="/~isabel/content/images/mantle-mahout.png">
+ </div>
+ </div>
+ <div id="slide2" style="display: none;">
+ <div class="callout">
+ Leveraging the power of Map/Reduce<br>
+ with Apache Hadoop
+ </div>
+ <div class="description">
+ Our core algorithms for clustering, classification and batch based collaborative filtering are implemented on
+ top of Apache Hadoop using the map/reduce paradigm
+ </div>
+ <div class="mantle">
+ <img alt="Hadoop" src="/~isabel/content/images/mantle-hadoop.png">
+ </div>
+ </div>
+ <div id="slide3" style="">
+ <div class="callout">
+ Scalable and vibrant community
+ </div>
+ <div class="description">
+ The goal of Apache Mahout is to build a vibrant, responsive, diverse community to facilitate discussions not only on
+ the project itself but also on potential use cases
+ </div>
+ <div class="mantle">
+ <img alt="Apache Mahout community" src="/~isabel/content/images/mantle-community.png">
+ </div>
+ </div>
+ <div id="slide4" style="opacity: 0.5; display: none;">
+ <div class="callout">
+ Apache 2.0 licensed
+ </div>
+ <div class="description">
+ Apache Mahout is distributed under a commercially friendly Apache Software license
+ </div>
+ <div class="mantle">
+ <img alt="Apache Software Foundation" src="/~isabel/content/images/mantle-asf.png">
+ </div>
+ </div>
+ </div>
+ <div class="controls">
+ <a class="next" id="next" title="Next" href="http://mahout.apache.org/#">Next</a>
+ <a class="previous" id="previous" title="Previous" href="http://mahout.apache.org/#">Previous</a>
+ <a class="start" id="start" title="Start" href="http://mahout.apache.org/#" style="display: none;">Start</a>
+ <a class="stop" id="stop" title="Stop" href="http://mahout.apache.org/#">Stop</a>
+ </div>
+ </div>
+
+ <script type="text/javascript">
+ function StartSlides() {
+ new Slides('mantleSlides');
+ }
+ document.observe ('dom:loaded', StartSlides);
+ </script>
+</div>
+
+ <div id="sidebar">
+ <div class="downloadNow">
+ <a href="http://cwiki.apache.org/confluence/display/MAHOUT/Downloads">
+ <img alt="Download Apache Mahout" src="/~isabel/content/images/download.png"></a>
+ </div>
+ <div id="sidebar-wrap">
+ <ul class="sidemenu">
+ <li>Latest release version: Mahout 0.8</li>
+ <li>Latest development version: Mahout 0.9-SNAPSHOT</li>
+ </ul>
+ <h2>Twitter</h2>
+ <ul class="sidemenu">
+ <li data-twttr-id="twttr-sandbox-0">
+ <iframe id="twitter-widget-0" scrolling="no" frameborder="0" allowtransparency="true" class="twitter-timeline twitter-timeline-rendered" title="Twitter Timeline Widget" style="border: none; max-width: 100%; min-width: 180px;" width="360" height="600"></iframe>
+ <script>!function(d,s,id){var js,fjs=d.getElementsByTagName(s)[0],p=/^http:/.test(d.location)?'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+"://platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs");</script>
+ </li>
+ </ul>
+ <h2>Apache Software Foundation</h2>
+ <ul class="sidemenu">
+ <li><a href="http://www.apache.org/foundation/how-it-works.html">How the ASF works</a></li>
+ <li><a href="http://www.apache.org/foundation/getinvolved.html">Get Involved</a></li>
+ <li><a href="http://www.apache.org/dev/">Developer Resources</a></li>
+ <li><a href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a></li>
+ <li><a href="http://www.apache.org/foundation/thanks.html">Thanks</a></li>
+ </ul>
+ <h2>Related Projects</h2>
+ <ul class="sidemenu">
+ <li><a href="http://lucene.apache.org/">Lucene</a></li>
+ <li><a href="http://hadoop.apache.org/">Hadoop</a></li>
+ </ul>
+ </div>
+</div>
+
+ <div id="content-wrap" class="clearfix">
+ <div id="main">
+ {% block content %}{% endblock %}
+ </div>
+ </div>
+</div>
+ <footer class="footer" align="center">
+ <div class="container">
+ <p>
+ Copyright © 2013 The Apache Software Foundation, Licensed under
+ the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.
+ <br />
+ Apache and the Apache feather logos are trademarks of The Apache Software Foundation.
+ </p>
+ </div>
+ </footer>
+
+ <script src="/~isabel/content/js/jquery-1.9.1.min.js"></script>
+ <script src="/~isabel/content/js/bootstrap.min.js"></script>
+ <script>
+ (function() {
+ var cx = '012254517474945470291:vhsfv7eokdc';
+ var gcse = document.createElement('script');
+ gcse.type = 'text/javascript';
+ gcse.async = true;
+ gcse.src = (document.location.protocol == 'https:' ? 'https:' : 'http:') +
+ '//www.google.com/cse/cse.js?cx=' + cx;
+ var s = document.getElementsByTagName('script')[0];
+ s.parentNode.insertBefore(gcse, s);
+ })();
+ </script>
+</body>
+</html>
Added: mahout/site/mahout_cms/templates/standard_markdown.html
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/templates/standard_markdown.html?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/templates/standard_markdown.html (added)
+++ mahout/site/mahout_cms/templates/standard_markdown.html Sun Nov 3 21:36:23 2013
@@ -0,0 +1,3 @@
+{% extends "standard.html" %}
+{% block title %}{{ headers.title }}{% endblock %}
+{% block content %}{{ content|markdown }}{% endblock %}
Added: mahout/site/mahout_cms/wiki-conversion/convert_cwiki_markup.pl
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/wiki-conversion/convert_cwiki_markup.pl?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/wiki-conversion/convert_cwiki_markup.pl (added)
+++ mahout/site/mahout_cms/wiki-conversion/convert_cwiki_markup.pl Sun Nov 3 21:36:23 2013
@@ -0,0 +1,203 @@
+#!/usr/bin/perl
+# Converts a CWiki markup to MarkDown syntax
+use strict;
+use warnings;
+
+use Text::Wrap qw/wrap $huge/;
+$huge = "overflow";
+
+my $pageName = shift;
+my $compressedPageName = $pageName;
+$compressedPageName =~ s/\s//g;
+
+my $source = shift;
+unless($source && -f $source) {
+ print "Use:\n";
+ print " $0 <page name> <cwiki template> [export.html]\n";
+ exit 1;
+}
+my $dest = shift;
+unless($dest) {
+ $dest = $source;
+ $dest .= ".html";
+}
+
+sub convertURL {
+ my $url = shift;
+ if($url =~ /^http/) {
+ return $url;
+ }
+ $url =~ s/\s/-/g;
+ $url = lc($url).".html";
+ return $url;
+}
+
+open(INP, "<$source");
+open(OUT, ">$dest");
+
+print OUT "Title: $pageName\n";
+
+# Load the file into a temporary array, and do some line wrapping on
+# the paragraphs
+my @contents;
+while(my $line = <INP>) {
+ $line =~ s/\r//;
+ unless($line =~ /^\s*$/) {
+ my @parts = split(/( \[)/, $line);
+ $line = "";
+ foreach my $p (@parts) {
+ if($p =~ /^(.*?\])(.*)$/) {
+ $p = $1."\n".wrap("","",$2);
+ } else {
+ $p = wrap("","",$p);
+ }
+ $line .= $p;
+ }
+ unless($line =~ /\n$/s) { $line .= "\n"; }
+ }
+ push(@contents, $line);
+}
+
+# Convert it
+my $in = "";
+foreach my $line (@contents) {
+ if($in eq "noformat") {
+ if($line =~ /^\s*^{noformat}/) {
+ print OUT "\n";
+ $in = "";
+ } else {
+ print OUT " ".$line;
+ }
+ next;
+ } elsif($in eq "code:xml" || $in eq "code:text") {
+ if($line =~ /^\s*^{code}/) {
+ print OUT "\n";
+ $in = "";
+ } else {
+ print OUT " ".$line;
+ }
+ next;
+ } elsif($in eq "table") {
+ if($line =~ /^\s*$/) {
+ print OUT "</table>\n";
+ $in = "";
+ }
+ } elsif($in) {
+ warn("Unexpected block '$in' for $line");
+ $in = "";
+ }
+
+ # Numbered lists
+ if($line =~ /^\s*#/) {
+ $line =~ s/^\s*#\s*/1. /;
+ }
+
+ # Headings
+ if($line =~ /^h(\d)\.\s?(.*)$/) {
+ # Build the heading entry
+ $line = ("#"x$1)." ".$2."\n";
+ # Now replicate the a name
+ my $name = $2;
+ $name =~ s/\s//g;
+ $line = "<a name=\"$compressedPageName-$name\"></a>\n".$line;
+ }
+
+ # Links
+ if($line =~ /(\[(.*?)\])/) {
+ my ($all,$text,$link) = ($1,$2,"");
+ if($text =~ /^(.*?)\|(.*)/) {
+ ($text,$link) = ($1,$2);
+ }
+
+ my $newlink;
+ if($link) {
+ $newlink = "[$text](".convertURL($link).")";
+ } else {
+ $newlink = "[$text](".convertURL($text).")";
+ }
+ $line =~ s/\Q$all\E/$newlink/;
+ }
+
+ # Old-style bold / italic
+ $line =~ s/\{\{(.*?)\}\}/*$1*/g;
+ $line =~ s/\{\{\{(.*?)\}\}\}/**$1**/g;
+
+ # No-Format
+ if($line =~ /^\s*^{noformat}/) {
+ $in = "noformat";
+ $line = "\n";
+ }
+
+ # Code blocks
+ if($line =~ /^\s*^{code:xml}/) {
+ $in = "code:xml";
+ $line = "\n";
+ }
+ if($line =~ /^\s*^{code:title=(.*?)}/) {
+ $in = "code:text";
+ $line = '<DIV class="code panel" style="border-style: solid;border-width: 1px;"><DIV class="codeHeader panelHeader" style="border-bottom-width: 1px;border-bottom-style: solid;"><B>'.$1.'</B></DIV><DIV class="codeContent panelContent">'."\n";
+ }
+ if($line =~ /^\s*^{code}/) {
+ $in = "code:text";
+ $line = "\n";
+ }
+
+ # Forced breaks
+ if($line =~ /\\\\/) {
+ $line = " \n \n";
+ }
+
+ # Tables
+ if($line =~ /^\|/) {
+ unless($in eq "table") {
+ print OUT "<table>\n";
+ $in = "table";
+ }
+ $line =~ s/\|\|/<\/th><th>/g;
+ $line =~ s/\|/<\/td><td>/g;
+ $line =~ s/<t[dh]>\s*$//gs;
+ $line =~ s/^\s*<\/t[dh]>//gs;
+ $line = "<tr>".$line."</tr>\n";
+ }
+
+ # Table of contents
+ if($line =~ /\s*^{toc}/) {
+ my @headings = grep(/^h/, @contents);
+ $line = "";
+ foreach my $h (@headings) {
+ $h =~ /^\s*h(\d).\s+(.*?)\s*$/s;
+ my $hnum = $1;
+ my $text = $2;
+ my $name = $2;
+ $name =~ s/\s//g;
+
+ my $l = " "x($hnum-1);
+ $l .= "* [$text](#$compressedPageName-$name)\n";
+ $line .= $l;
+ }
+ }
+
+ print OUT $line;
+}
+
+close INP;
+close OUT;
+
+=head1 LICENSE
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
Propchange: mahout/site/mahout_cms/wiki-conversion/convert_cwiki_markup.pl
------------------------------------------------------------------------------
svn:executable = *
Added: mahout/site/mahout_cms/wiki-conversion/convert_export_template.pl
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/wiki-conversion/convert_export_template.pl?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/wiki-conversion/convert_export_template.pl (added)
+++ mahout/site/mahout_cms/wiki-conversion/convert_export_template.pl Sun Nov 3 21:36:23 2013
@@ -0,0 +1,96 @@
+#!/usr/bin/perl
+# Converts a CWiki export template to a DTL template
+use strict;
+use warnings;
+
+my $source = shift;
+unless($source && -f $source) {
+ print "Use:\n";
+ print " $0 <cwiki template.xml> [export.html] [export_markdown.html]\n";
+ exit 1;
+}
+
+my $dest = shift;
+unless($dest) {
+ $dest = $source;
+ $dest =~ s/\.xml/\.html/;
+}
+if(-f $dest) {
+ print "Destination file $dest already exists\n";
+ exit 1;
+}
+
+my $mdest = shift;
+unless($mdest) {
+ $mdest = $dest;
+ $mdest =~ s/\.html/_markdown.html/;
+}
+if(-f $mdest) {
+ print "Destination file $mdest already exists\n";
+ exit 1;
+}
+
+# Prepare to convert
+open(INP, "<$source");
+open(OUT, ">$dest");
+open(MOUT, ">$mdest");
+
+print MOUT "{% extends \"$dest\" %}\n";
+
+my $done_title = 0;
+my $done_body = 0;
+while(my $line = <INP>) {
+ if($line =~ /^#/) { next; }
+
+ if($line =~ /\$page.title/) {
+ $done_title++;
+ if($done_title > 1) {
+ warn("Found \$page.title multiple times, but only converting once\n");
+ $line =~ s/\$page.title//;
+ } else {
+ $line =~ s/\$page.title/{% block title %}{% endblock %}/;
+ print MOUT '{% block title %}{{ headers.title }}{% endblock %}'."\n";
+ }
+ }
+
+ if($line =~ /\$body/) {
+ $done_body++;
+ if($done_body > 1) {
+ warn("Found \$body multiple times, but only converting once\n");
+ $line =~ s/\$body//;
+ } else {
+ $line =~ s/\$body/{% block content %}{% endblock %}/;
+ print MOUT '{% block content %}{{ content|markdown }}{% endblock %}'."\n";
+ }
+ }
+
+ $line =~ s/\$autoexport.breadcrumbs\(\$page\)/{{ breadcrumbs|safe }}/;
+
+ print OUT $line;
+}
+
+close INP;
+close OUT;
+close MOUT;
+
+print "Generated $dest\n";
+print "Generated $mdest\n";
+
+=head1 LICENSE
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
Propchange: mahout/site/mahout_cms/wiki-conversion/convert_export_template.pl
------------------------------------------------------------------------------
svn:executable = *
Added: mahout/site/mahout_cms/wiki-conversion/export_site.pl
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/wiki-conversion/export_site.pl?rev=1538467&view=auto
==============================================================================
--- mahout/site/mahout_cms/wiki-conversion/export_site.pl (added)
+++ mahout/site/mahout_cms/wiki-conversion/export_site.pl Sun Nov 3 21:36:23 2013
@@ -0,0 +1,94 @@
+#!/usr/bin/perl
+# Spiders a site using the cwiki export, and generates markdown files
+# for each page in cwiki
+use strict;
+use warnings;
+
+use RPC::XML;
+use RPC::XML::Client;
+
+my $confluence = "https://cwiki.apache.org/confluence/";
+my $RPCURL = $confluence."rpc/xmlrpc";
+
+my $siteName = shift;
+my $username = shift;
+my $password = shift;
+unless($siteName && $username && $password) {
+ print "Use:\n";
+ print " $0 <cwiki site name> <username> <password>\n";
+ print "\n";
+ print " eg for https://cwiki.apache.org/confluence/display/COMDEVxSITE\n";
+ print " $0 COMDEVxSITE jim securePassword\n";
+ exit 1;
+}
+
+# Check we're in the right place
+if(-d "content" && -d "lib") {
+} else {
+ die("Must be run from root of site, containing /content/ and /lib/\n");
+}
+my ($binPath) = ($0 =~ /^(.*)\/(.*?)$/);
+
+# Connect to their icky API. (What's wrong with REST we ask...?)
+my $client = RPC::XML::Client->new($RPCURL);
+my $response = $client->send_request('confluence1.login', $username, $password);
+$response->is_fault and die "ERROR: could not login as $username: ", $response->value->{faultString};
+my $token = $response->value;
+
+# Get a list of all the pages
+$response = $client->send_request('confluence1.getPages', $token, $siteName);
+$response->is_fault and die "ERROR: could not get pages for $siteName: ", $response->value->{faultString};
+
+my @pages = @{$response->value};
+
+foreach my $page (@pages) {
+ my $title = $page->{title};
+ $title = $title . " ";
+
+ print "Fetching $title (".$page->{id}.")\n";
+ $response = $client->send_request('confluence1.getPage', $token, $siteName, $title);
+ $response->is_fault and die "ERROR: could not get page details: ", $response->value->{faultString};
+
+ my %details = %{$response->value};
+ my $content = $details{content};
+
+ print "Processing $title from ".$page->{url}."\n";
+
+ my $page = $title;
+ $page =~ s/\s/-/g;
+
+ my $cwikiFile = "content/".lc($page).".cwiki";
+ open(CWIKI, ">$cwikiFile");
+ print CWIKI $content;
+ close CWIKI;
+
+ my $mdFile = $cwikiFile;
+ $mdFile =~ s/\.cwiki/.mdtext/;
+
+ print " Generating markdown file\n";
+ `$binPath/convert_cwiki_markup.pl "$title" $cwikiFile $mdFile`;
+
+ print " Finished processing $title\n\n";
+}
+
+print "\n";
+print "Done!\n";
+
+=head1 LICENSE
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
Propchange: mahout/site/mahout_cms/wiki-conversion/export_site.pl
------------------------------------------------------------------------------
svn:executable = *