You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mahout.apache.org by pa...@apache.org on 2015/04/21 02:16:10 UTC
svn commit: r1675008 -
/mahout/site/mahout_cms/trunk/content/users/environment/how-to-build-an-app.mdtext
Author: pat
Date: Tue Apr 21 00:16:09 2015
New Revision: 1675008
URL: http://svn.apache.org/r1675008
Log:
CMS commit to mahout by pat
Modified:
mahout/site/mahout_cms/trunk/content/users/environment/how-to-build-an-app.mdtext
Modified: mahout/site/mahout_cms/trunk/content/users/environment/how-to-build-an-app.mdtext
URL: http://svn.apache.org/viewvc/mahout/site/mahout_cms/trunk/content/users/environment/how-to-build-an-app.mdtext?rev=1675008&r1=1675007&r2=1675008&view=diff
==============================================================================
--- mahout/site/mahout_cms/trunk/content/users/environment/how-to-build-an-app.mdtext (original)
+++ mahout/site/mahout_cms/trunk/content/users/environment/how-to-build-an-app.mdtext Tue Apr 21 00:16:09 2015
@@ -1,47 +1,168 @@
-#Multiple Indicator Creation
+#How to create and App using Mahout
-This is an example of how to create more that two indictors from more than two user interaction types with Mahout. We will use very simple hand created example data for one might see in an ecommerce application. The application records three interactions for item-purchase, item-detail-view, and category-preference (search for or click on a category).
+This is an example of how to create a simple app using Mahout as a Library. The source is available on Github in the [3-input-cooc project](https://github.com/pferrel/3-input-cooc) with more explanation about what it does. For this tutorial we'll concentrate on how to create an app.
-*spark-itemsimilarity* will handle two inputs but here we have three and rather than running *spark-itemsimilarity* twice we will create our own app to do it.
+This example is for reading three interactions types and creating indicators for them using cooccurrence and cross-cooccurrence. The indicators will be written to text files in a format ready for search engine indexing in recommender.
##Setup
In order to build and run the CooccurrenceDriver you need to install the following:
* Install the Java 7 JDK from Oracle. Mac users look here: [Java SE Development Kit 7u72](http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html).
-* Install sbt (simple build tool) 0.13.x for <a href="Installing-sbt-on-Mac.html">Mac</a>, <a href="Installing-sbt-on-Windows.html">Windows</a>,
-<a href="Installing-sbt-on-Linux.html">Linux</a>, or
-<a href="Manual-Installation.html">manual installation</a>.
+* Install sbt (simple build tool) 0.13.x for [Mac](http://www.scala-sbt.org/release/tutorial/Installing-sbt-on-Mac.html),[Linux](http://www.scala-sbt.org/release/tutorial/Installing-sbt-on-Linux.html) or [manual instalation](http://www.scala-sbt.org/release/tutorial/Manual-Installation.html).
* Install [Mahout](http://mahout.apache.org/general/downloads.html). Don't forget to setup MAHOUT_HOME and MAHOUT_LOCAL
-##Build
-Building the examples from project's root folder:
+##Application
+Using Mahout as a library in an application will require a little Scala code. We have an App trait in Scala so we'll create an object, which inherits from ```App```
- $ sbt pack
+```
+object CooccurrenceDriver extends App {
+}
+```
+This will look a little different than Java since ```App``` does delayed initialization, which causes the main body to be executed when the App is launched, just as in Java you would create a CooccurrenceDriver.main.
+
+Before we can execute something on Spark we'll need to create a context. We could use raw Spark calls here but default values are setup for a Ma // strip off names, which only takes and array of IndexedDatasets
+ val indicatorMatrices = SimilarityAnalysis.cooccurrencesIDSs(actions.map(a => a._2))
+hout context.
+
+```
+implicit val mc = mahoutSparkContext(masterUrl = "local", appName = "2-input-cooc")
+```
+We need to read in three files containing different interaction types. The files will each be read into a Mahout IndexedDataset. This allows us to preserve application-specific user and item IDs throughout the calculations.
+
+For example, here is data/purchase.csv:
+
+```
+u1,iphone
+u1,ipad
+u2,nexus
+u2,galaxy
+u3,surface
+u4,iphone
+u4,galaxy
+
+```
+Mahout has a helper function that reads the text delimited in SparkEngine.indexedDatasetDFSReadElements. The function reads single elements in a distributed way to create the IndexedDataset.
+
+Notice we read in all datasets before we adjust the number of rows in them to match the total number of users in the data. This is so the math works out even if some users took one action but not another.
+
+```
+/**
+ * Read files of element tuples and create IndexedDatasets one per action. These share a userID BiMap but have
+ * their own itemID BiMaps
+ */
+def readActions(actionInput: Array[(String, String)]): Array[(String, IndexedDataset)] = {
+ var actions = Array[(String, IndexedDataset)]()
+
+ val userDictionary: BiMap[String, Int] = HashBiMap.create()
+
+ // The first action named in the sequence is the "primary" action and
+ // begins to fill up the user dictionary
+ for ( actionDescription <- actionInput ) {// grab the path to actions
+ val action: IndexedDataset = SparkEngine.indexedDatasetDFSReadElements(
+ actionDescription._2,
+ schema = DefaultIndexedDatasetElementReadSchema,
+ existingRowIDs = userDictionary)
+ userDictionary.putAll(action.rowIDs)
+ // put the name in the tuple with the indexedDataset
+ actions = actions :+ (actionDescription._1, action)
+ }
+
+ // After all actions are read in the userDictonary will contain every user seen,
+ // even if they may not have taken all actions . Now we adjust the row rank of
+ // all IndxedDataset's to have this number of rows
+ // Note: this is very important or the cooccurrence calc may fail
+ val numUsers = userDictionary.size() // one more than the cardinality
+
+ val resizedNameActionPairs = actions.map { a =>
+ //resize the matrix by, in effect by adding empty rows
+ val resizedMatrix = a._2.create(a._2.matrix, userDictionary, a._2.columnIDs).newRowCardinality(numUsers)
+ (a._1, resizedMatrix) // return the Tuple of (name, IndexedDataset)
+ }
+ resizedNameActionPairs // return the array of Tuples
+}
+
+```
+
+Now that we have the data read in we can perform the cooccurrence calculation.
+
+```
+// strip off names, which only takes and array of IndexedDatasets
+val indicatorMatrices = SimilarityAnalysis.cooccurrencesIDSs(actions.map(a => a._2))
+
+```
+
+All we need to do now is write the indicators.
+
+```
+// zip a pair of arrays into an array of pairs, reattaching the action names
+val indicatorDescriptions = actions.map(a => a._1).zip(indicatorMatrices)
+writeIndicators(indicatorDescriptions)
+```
+
+The ```writeIndicators``` method uses the default write function ```dfsWrite```.
+
+```
+/**
+ * Write indicatorMatrices to the output dir in the default format
+ */
+def writeIndicators( indicators: Array[(String, IndexedDataset)]) = {
+ for (indicator <- indicators ) {
+ val indicatorDir = OutputPath + indicator._1
+ indicator._2.dfsWrite(
+ indicatorDir, // do we have to remove the last $ char?
+ // omit LLR strengths and format for search engine indexing
+ IndexedDatasetWriteBooleanSchema)
+ }
+}
+
+```
-This will automatically set up some launcher scripts for the driver. To run execute
+See the Github project for the full source. Now we create a build.sbt to build the example.
- $ target/pack/bin/cooc
-
-The driver will execute in Spark standalone mode one the provided sample data and output log information including various information about the input data. The output will be in /path/to/3-input-cooc/data/indicators/*indicator-type*
+```
+name := "cooccurrence-driver"
-##CooccurrenceDriver
+organization := "com.finderbots"
-This driver takes three actions in three separate input files. The input is in tuple form (user-id,item-id) one per line. It calculates all cooccurrence and cross-cooccurrence indicators. The sample actions are trivial hand made examples with somewhat intuitive data.
+version := "0.1"
-Actions:
+scalaVersion := "2.10.4"
- 1. **Purchase**: user purchases
- 2. **View**: user product details views
- 3. **Category**: user preference for category tags
-
-Indicators:
+val sparkVersion = "1.1.1"
+
+libraryDependencies ++= Seq(
+ "log4j" % "log4j" % "1.2.17",
+ // Mahout's Spark code
+ "commons-io" % "commons-io" % "2.4",
+ "org.apache.mahout" % "mahout-math-scala_2.10" % "0.10.0",
+ "org.apache.mahout" % "mahout-spark_2.10" % "0.10.0",
+ "org.apache.mahout" % "mahout-math" % "0.10.0",
+ "org.apache.mahout" % "mahout-hdfs" % "0.10.0",
+ // Google collections, AKA Guava
+ "com.google.guava" % "guava" % "16.0")
+
+resolvers += "typesafe repo" at " http://repo.typesafe.com/typesafe/releases/"
- 1. **Purchase cooccurrence**: may be interpretted as a list if similar items for each item. Similar in terms of which users purchased them.
- 2. **View cross-cooccurrence**: may be interpretted as a list of similar items in terms of which users viewed the item where the view led to a purchase.
- 3. **Category cross-cooccurrence**: may be interpretted as a list of similar categories in terms of which users preferred the category and this led to a purchase.
+resolvers += Resolver.mavenLocal
-##Data
-Mahout has reader traits that will read text delimited files. Input for *spark-itemsimilarity* and this CooccurrenceDriver are tuples of (user-id,item-id) with one line per tuple. The inputs for CooccurrenceDriver are files but in *spark-itemsimilarity* they may be directories of "part-xxxxx" files. These can be found in the ```data``` directory.
+packSettings
+
+packMain := Map(
+ "cooc" -> "CooccurrenceDriver"
+)
+
+```
+
+##Build
+Building the examples from project's root folder:
+
+ $ sbt pack
+
+This will automatically set up some launcher scripts for the driver. To run execute
+
+ $ target/pack/bin/cooc
+
+The driver will execute in Spark standalone mode and put the data in /path/to/3-input-cooc/data/indicators/*indicator-type*
##Using a Debugger
To build and run this example in a debugger like IntelliJ IDEA. Install from the IntelliJ site and add the Scala plugin.
@@ -58,3 +179,9 @@ Now choose "Application" in the left pan
After setting breakpoints you are now ready to debug the configuration. Go to the Run->Debug... menu and pick your configuration. This will execute using a local standalone instance of Spark.
+
+##The Mahout Shell
+
+For small script-like apps you may wish to use the Mahout shell. It is a Scala REPL type interactive shell built on the Spark shell with Mahout-Samsara extensions.
+
+For the shell you won't need the context, since it is created when the shell is launched. To control the configuration of Mahout and Spark we set environment variables.
\ No newline at end of file