You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@predictionio.apache.org by sh...@apache.org on 2017/07/10 04:51:23 UTC
[1/2] incubator-predictionio git commit: [PIO-97] Fixes docs under
the PredictionIO Official Templates menu.
Repository: incubator-predictionio
Updated Branches:
refs/heads/develop 76f340900 -> b6f168afb
[PIO-97] Fixes docs under the PredictionIO Official Templates menu.
Closes #403
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/75cfe461
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/75cfe461
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/75cfe461
Branch: refs/heads/develop
Commit: 75cfe461c7c7c0b70f2551a382ea7a0982daaabb
Parents: b4f2050
Author: shimamoto <sh...@apache.org>
Authored: Mon Jul 10 13:25:26 2017 +0900
Committer: shimamoto <sh...@apache.org>
Committed: Mon Jul 10 13:25:26 2017 +0900
----------------------------------------------------------------------
docs/manual/data/nav/main.yml | 18 +-
.../classification/add-algorithm.html.md | 8 +-
.../classification/quickstart.html.md.erb | 4 +-
.../reading-custom-properties.html.md | 21 +-
.../classification/transform-data.html.md | 16 --
.../quickstart.html.md.erb | 4 +-
.../adjust-score.html.md.erb | 186 ++++++++++++++++
.../ecommercerecommendation/how-to.html.md | 1 +
.../quickstart.html.md.erb | 4 +-
.../train-with-rate-event.html.md.erb | 89 ++++----
.../quickstart.html.md.erb | 4 +-
.../leadscoring/quickstart.html.md.erb | 4 +-
.../productranking/quickstart.html.md.erb | 4 +-
.../recommendation/blacklist-items.html.md | 40 ++--
.../recommendation/customize-data-prep.html.md | 12 +-
.../recommendation/customize-serving.html.md | 48 ++--
.../templates/recommendation/how-to.html.md | 2 +-
.../recommendation/quickstart.html.md.erb | 3 +-
.../reading-custom-events.html.md | 13 +-
.../training-with-implicit-preference.html.md | 55 ++---
.../templates/similarproduct/how-to.html.md | 6 +-
.../multi-events-multi-algos.html.md.erb | 36 ++-
.../similarproduct/quickstart.html.md.erb | 4 +-
.../similarproduct/recommended-user.html.md.erb | 188 ++++++++++++++++
.../return-item-properties.html.md.erb | 146 +++++++++++++
.../rid-user-set-event.html.md.erb | 134 ++++++++++++
.../train-with-rate-event.html.md.erb | 217 +++++++++++++++++++
27 files changed, 1076 insertions(+), 191 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/data/nav/main.yml
----------------------------------------------------------------------
diff --git a/docs/manual/data/nav/main.yml b/docs/manual/data/nav/main.yml
index abc1b33..245fbaf 100644
--- a/docs/manual/data/nav/main.yml
+++ b/docs/manual/data/nav/main.yml
@@ -214,6 +214,9 @@ root:
-
body: 'Train with Rate Event'
url: '/templates/ecommercerecommendation/train-with-rate-event/'
+ -
+ body: 'Adjust Score'
+ url: '/templates/ecommercerecommendation/adjust-score/'
-
body: 'Similar Product'
children:
@@ -229,9 +232,20 @@ root:
-
body: 'Multiple Events and Multiple Algorithms'
url: '/templates/similarproduct/multi-events-multi-algos/'
-
+ -
+ body: 'Returns Item Properties'
+ url: '/templates/similarproduct/return-item-properties/'
+ -
+ body: 'Train with Rate Event'
+ url: '/templates/similarproduct/train-with-rate-event/'
+ -
+ body: 'Get Rid of Events for Users'
+ url: '/templates/similarproduct/rid-user-set-event/'
+ -
+ body: 'Recommend Users'
+ url: '/templates/similarproduct/recommended-user/'
-
- body: 'Classfication'
+ body: 'Classification'
children:
-
body: 'Quick Start'
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/classification/add-algorithm.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/classification/add-algorithm.html.md b/docs/manual/source/templates/classification/add-algorithm.html.md
index 9f29adc..6d5ad3d 100644
--- a/docs/manual/source/templates/classification/add-algorithm.html.md
+++ b/docs/manual/source/templates/classification/add-algorithm.html.md
@@ -21,6 +21,8 @@ limitations under the License.
The classification template uses the Naive Bayes algorithm by default. You can easily add and use other MLlib classification algorithms. The following will demonstrate how to add the [MLlib Random Forests algorithm](https://spark.apache.org/docs/latest/mllib-ensembles.html) into the engine.
+You can find the complete modified source code [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-classification/add-algorithm).
+
## Create a new file RandomForestAlgorithm.scala
Locate `src/main/scala/NaiveBayesAlgorithm.scala` under your engine directory, which should be /MyClassification if you are following the [Classification QuickStart](/templates/classification/quickstart/). Copy `NaiveBayesAlgorithm.scala` and create a new file `RandomForestAlgorithm.scala`. You will modify this file and follow the instructions below to define a new RandomForestAlgorithm class.
@@ -98,14 +100,14 @@ class RandomForestAlgorithm(val ap: RandomForestAlgorithmParams) // CHANGED
query: Query): PredictedResult = {
val label = model.predict(Vectors.dense(
- query.attr0, query.attr1, query.attr2
+ Array(query.attr0, query.attr1, query.attr2)
))
new PredictedResult(label)
}
}
```
-Note that the MLlib Random Forest algorithm takes the same training data as the Navie Bayes algorithm (ie, RDD[LabeledPoint]) so you don't need to modify the `DataSource`, `TrainigData` and `PreparedData` classes. If the new algorithm to be added requires different types of training data, then you need to modify these classes accordingly to accomodate your new algorithm.
+Note that the MLlib Random Forest algorithm takes the same training data as the Naive Bayes algorithm (ie, RDD[LabeledPoint]) so you don't need to modify the `DataSource` and `PreparedData` classes. If the new algorithm to be added requires different types of training data, then you need to modify these classes accordingly to accommodate your new algorithm.
## Update Engine.scala
Modify the EngineFactory to add the new algorithm class `RandomForestAlgorithm` you just defined and give it a name `"randomforest"`. The name will be used in `engine.json` to specify which algorithm to use.
@@ -137,7 +139,7 @@ Update the engine.json to use **randomforest**:
{
"name": "randomforest",
"params": {
- "numClasses": 3,
+ "numClasses": 4,
"numTrees": 5,
"featureSubsetStrategy": "auto",
"impurity": "gini",
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/classification/quickstart.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/classification/quickstart.html.md.erb b/docs/manual/source/templates/classification/quickstart.html.md.erb
index 70a99e2..19e9c6f 100644
--- a/docs/manual/source/templates/classification/quickstart.html.md.erb
+++ b/docs/manual/source/templates/classification/quickstart.html.md.erb
@@ -365,11 +365,11 @@ Event Server using Python SDK. Please upgrade to the latest Python SDK.
<%= partial 'shared/quickstart/install_python_sdk' %>
-Make sure you are under the `MyClassification` directory. Replace the value of access_key parameter by your **Access Key** and run:
+Make sure you are under the `MyClassification` directory. Execute the following to import the data:
```
$ cd MyClassification
-$ python data/import_eventserver.py --access_key obbiTuSOiMzyFKsvjjkDnWk1vcaHjcjrv9oT3mtN3y6fOlpJoVH459O1bPmDzCdv
+$ python data/import_eventserver.py --access_key $ACCESS_KEY
```
You should see the following output:
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/classification/reading-custom-properties.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/classification/reading-custom-properties.html.md b/docs/manual/source/templates/classification/reading-custom-properties.html.md
index 4dc43fe..d3eb41a 100644
--- a/docs/manual/source/templates/classification/reading-custom-properties.html.md
+++ b/docs/manual/source/templates/classification/reading-custom-properties.html.md
@@ -19,13 +19,13 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-By default, the classification template reads 4 properties of a user entity: "attr0", "attr1", "attr2" and "plan". You can modify the [default DataSource](/templates/classification/dase/#data) to read to read your custom properties or different Entity Type.
+By default, the classification template reads 4 properties of a user entity: "attr0", "attr1", "attr2" and "plan". You can modify the [default DataSource](dase.html#data) to read your custom properties or different Entity Type.
-In this example, we modify DataSource to read properties "featureA", "featureB", "featureC", "featureD" and "label" for entity type "item".
+In this example, we modify DataSource to read properties "featureA", "featureB", "featureC", "featureD" and "label" for entity type "item". You can find the complete modified source code [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-classification/reading-custom-properties).
>> Note: you also need import events with these properties accordingly.
-Modify the `readTraining()` in DataSource.scala:
+Modify the `readTraining()` and `readEval()` in DataSource.scala:
- modify the `entityType` parameter
- modify the list of properties names in the `required` parameter
@@ -34,16 +34,17 @@ Modify the `readTraining()` in DataSource.scala:
```scala
def readTraining(sc: SparkContext): TrainingData = {
...
- val labeledPoints: RDD[LabeledPoint] = eventsDb.aggregateProperties(
- appId = dsp.appId,
- entityType = "item", // MODIFFIED HERE
- required = Some(List( // MODIFIED HERE
+ val labeledPoints: RDD[LabeledPoint] = PEventStore.aggregateProperties(
+ appName = dsp.appName,
+ entityType = "item", // MODIFIED
+ // only keep entities with these required properties defined
+ required = Some(List( // MODIFIED
"featureA", "featureB", "featureC", "featureD", "label")))(sc)
// aggregateProperties() returns RDD pair of
// entity ID and its aggregated properties
.map { case (entityId, properties) =>
try {
- // MODIFIED HERE
+ // MODIFIED
LabeledPoint(properties.get[Double]("label"),
Vectors.dense(Array(
properties.get[Double]("featureA"),
@@ -59,7 +60,7 @@ Modify the `readTraining()` in DataSource.scala:
throw e
}
}
- }
+ }.cache()
...
}
```
@@ -70,4 +71,4 @@ Lastly, redefine the Query class parameters to take in four double values: featu
$ curl -H "Content-Type: application/json" -d '{ "featureA":2, "featureB":0, "featureC":0, "featureD":0 }' http://localhost:8000/queries.json
```
-That's it! Now your classifcation engine is using different properties as training data.
+That's it! Now your classification engine is using different properties as training data.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/classification/transform-data.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/classification/transform-data.html.md b/docs/manual/source/templates/classification/transform-data.html.md
deleted file mode 100644
index e8d0e9c..0000000
--- a/docs/manual/source/templates/classification/transform-data.html.md
+++ /dev/null
@@ -1,16 +0,0 @@
-<!--
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/complementarypurchase/quickstart.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/complementarypurchase/quickstart.html.md.erb b/docs/manual/source/templates/complementarypurchase/quickstart.html.md.erb
index d6366c1..720f9b8 100644
--- a/docs/manual/source/templates/complementarypurchase/quickstart.html.md.erb
+++ b/docs/manual/source/templates/complementarypurchase/quickstart.html.md.erb
@@ -215,11 +215,11 @@ A Python import script `import_eventserver.py` is provided to import sample data
<%= partial 'shared/quickstart/install_python_sdk' %>
-Make sure you are under the `MyComplementaryPurchase` directory. Execute the following to import the data (Replace the value of access_key parameter with your **Access Key**):
+Make sure you are under the `MyComplementaryPurchase` directory. Execute the following to import the data:
```
$ cd MyComplementaryPurchase
-$ python data/import_eventserver.py --access_key 3mZWDzci2D5YsqAnqNnXH9SB6Rg3dsTBs8iHkK6X2i54IQsIZI1eEeQQyMfs7b3F
+$ python data/import_eventserver.py --access_key $ACCESS_KEY
```
You should see the following output:
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/ecommercerecommendation/adjust-score.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/ecommercerecommendation/adjust-score.html.md.erb b/docs/manual/source/templates/ecommercerecommendation/adjust-score.html.md.erb
new file mode 100644
index 0000000..40be547
--- /dev/null
+++ b/docs/manual/source/templates/ecommercerecommendation/adjust-score.html.md.erb
@@ -0,0 +1,186 @@
+---
+title: Adjust Score (E-Commerce Recommendation)
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+This examples demonstrates how to modify E-Commerce Recommendation template to further adjust score.
+
+By default, items have a weight of 1.0. Giving an item a weight greater than 1.0 will make them appear more often and can be useful for i.e. promoted products. An item can also be given a weight smaller than 1.0 (but bigger than 0), in which case it will be recommended less often than originally. Weight values smaller than 0.0 are invalid.
+
+You can find the complete modified source code [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-ecommercerecommendation/adjust-score).
+
+
+## Modification
+
+### ECommAlgorithm.scala
+
+Add a case class to represent each group items which are given the same weight.
+
+```scala
+// ADDED
+case class WeightGroup(
+ items: Set[String],
+ weight: Double
+)
+```
+
+In ECommAlgorithm, add `weightedItems` function to extract the sequence of `WeightGroup`.
+
+```scala
+ // ADDED
+ /** Get the latest constraint weightedItems */
+ def weightedItems: Seq[WeightGroup] = {
+ try {
+ val constr = LEventStore.findByEntity(
+ appName = ap.appName,
+ entityType = "constraint",
+ entityId = "weightedItems",
+ eventNames = Some(Seq("$set")),
+ limit = Some(1),
+ latest = true,
+ timeout = Duration(200, "millis")
+ )
+ if (constr.hasNext) {
+ constr.next.properties.get[Seq[WeightGroup]]("weights")
+ } else {
+ Nil
+ }
+ } catch {
+ case e: scala.concurrent.TimeoutException =>
+ logger.error(s"Timeout when read set weightedItems event." +
+ s" Empty list is used. ${e}")
+ Nil
+ case e: Exception =>
+ logger.error(s"Error when read set weightedItems event: ${e}")
+ throw e
+ }
+ }
+```
+
+Modify the `predictKnownUser()`, `predictDefault()` and `predictSimilar()`:
+
+- add the `weights: Map[Int, Double]` parameter
+- adjust score according to item weights
+
+```scala
+ def predictKnownUser(
+ userFeature: Array[Double],
+ productModels: Map[Int, ProductModel],
+ query: Query,
+ whiteList: Option[Set[Int]],
+ blackList: Set[Int],
+ weights: Map[Int, Double] // ADDED
+ ): Array[(Int, Double)] = {
+
+ ...
+ .map { case (i, pm) =>
+ // NOTE: features must be defined, so can call .get
+ val s = dotProduct(userFeature, pm.features.get)
+ // may customize here to further adjust score
+ // ADDED
+ val adjustedScore = s * weights(i)
+ (i, adjustedScore)
+ }
+ ...
+
+ }
+```
+
+Lastly, modify the `predict()` method. The sequence of `WeightGroup` transforms into a `Map[Int, Double]` that we can easily query to extract the weight given to an item, using its `Int` index.
+
+```scala
+ def predict(model: ECommModel, query: Query): PredictedResult = {
+
+ ...
+
+ // ADDED
+ val weights: Map[Int, Double] = (for {
+ group <- weightedItems
+ item <- group.items
+ index <- model.itemStringIntMap.get(item)
+ } yield (index, group.weight))
+ .toMap
+ .withDefaultValue(1.0)
+
+ ...
+
+ val topScores: Array[(Int, Double)] = if (userFeature.isDefined) {
+ // the user has feature vector
+ predictKnownUser(
+ userFeature = userFeature.get,
+ productModels = productModels,
+ query = query,
+ whiteList = whiteList,
+ blackList = finalBlackList,
+ weights = weights // ADDED
+ )
+ } else {
+ ...
+
+ if (recentFeatures.isEmpty) {
+ logger.info(s"No features vector for recent items ${recentItems}.")
+ predictDefault(
+ productModels = productModels,
+ query = query,
+ whiteList = whiteList,
+ blackList = finalBlackList,
+ weights = weights // ADDED
+ )
+ } else {
+ predictSimilar(
+ recentFeatures = recentFeatures,
+ productModels = productModels,
+ query = query,
+ whiteList = whiteList,
+ blackList = finalBlackList,
+ weights = weights // ADDED
+ )
+ }
+ }
+
+ ...
+ }
+```
+
+Now, to send an event to Event Server:
+
+```
+$ curl -i -X POST http://localhost:7070/events.json?accessKey=$ACCESS_KEY \
+-H "Content-Type: application/json" \
+-d '{
+ "event" : "$set",
+ "entityType" : "constraint",
+ "entityId" : "weightedItems",
+ "properties" : {
+ "weights": [
+ {
+ "items": ["i4", "i14"],
+ "weight": 1.2
+ },
+ {
+ "items": ["i11"],
+ "weight": 1.5
+ }
+ ]
+ },
+ "eventTime" : "2014-11-02T09:39:45.618-08:00"
+}'
+```
+
+That's it! Now your engine can predict with adjusted scores.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/ecommercerecommendation/how-to.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/ecommercerecommendation/how-to.html.md b/docs/manual/source/templates/ecommercerecommendation/how-to.html.md
index b5ac7d9..9aa4e5b 100644
--- a/docs/manual/source/templates/ecommercerecommendation/how-to.html.md
+++ b/docs/manual/source/templates/ecommercerecommendation/how-to.html.md
@@ -22,3 +22,4 @@ limitations under the License.
Here are the pages that show you how you can customize the E-Commerce Recommendation engine template.
- [Train with Rate Event](/templates/ecommercerecommendation/train-with-rate-event/)
+- [Adjust Score](/templates/ecommercerecommendation/adjust-score/)
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/ecommercerecommendation/quickstart.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/ecommercerecommendation/quickstart.html.md.erb b/docs/manual/source/templates/ecommercerecommendation/quickstart.html.md.erb
index c71ae77..5907a2d 100644
--- a/docs/manual/source/templates/ecommercerecommendation/quickstart.html.md.erb
+++ b/docs/manual/source/templates/ecommercerecommendation/quickstart.html.md.erb
@@ -410,11 +410,11 @@ A Python import script `import_eventserver.py` is provided to import sample data
<%= partial 'shared/quickstart/install_python_sdk' %>
-Make sure you are under the `MyECommerceRecommendation` directory. Execute the following to import the data (Replace the value of access_key parameter with your **Access Key**):
+Make sure you are under the `MyECommerceRecommendation` directory. Execute the following to import the data:
```
$ cd MyECommerceRecommendation
-$ python data/import_eventserver.py --access_key 3mZWDzci2D5YsqAnqNnXH9SB6Rg3dsTBs8iHkK6X2i54IQsIZI1eEeQQyMfs7b3F
+$ python data/import_eventserver.py --access_key $ACCESS_KEY
```
You should see the following output:
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/ecommercerecommendation/train-with-rate-event.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/ecommercerecommendation/train-with-rate-event.html.md.erb b/docs/manual/source/templates/ecommercerecommendation/train-with-rate-event.html.md.erb
index 7d83429..141a35d 100644
--- a/docs/manual/source/templates/ecommercerecommendation/train-with-rate-event.html.md.erb
+++ b/docs/manual/source/templates/ecommercerecommendation/train-with-rate-event.html.md.erb
@@ -25,7 +25,7 @@ However, recent "view" event is still used for recommendation for new user (to r
This template also supports that the user may rate same item multiple times and latest rating value will be used for training. The modification can be further simplified if the support of this case is not needed.
-You can find the complete modified source code [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-ecommercerecommendation/train-with-rate-event) and the modification is based on E-Commerce Recommendation template v0.1.1.
+You can find the complete modified source code [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-ecommercerecommendation/train-with-rate-event).
## Modification
@@ -53,66 +53,67 @@ Modify TrainingData class to use rateEvent
class TrainingData(
val users: RDD[(String, User)],
val items: RDD[(String, Item)],
- val rateEvents: RDD[RateEvent] // MODIFIED
+ val rateEvents: RDD[RateEvent], // MODIFIED
+ val buyEvents: RDD[BuyEvent]
) extends Serializable {
override def toString = {
s"users: [${users.count()} (${users.take(2).toList}...)]" +
s"items: [${items.count()} (${items.take(2).toList}...)]" +
// MODIFIED
- s"rateEvents: [${rateEvents.count()}] (${rateEvents.take(2).toList}...)"
+ s"rateEvents: [${rateEvents.count()}] (${rateEvents.take(2).toList}...)" +
+ s"buyEvents: [${buyEvents.count()}] (${buyEvents.take(2).toList}...)"
}
}
```
-Modify `readTraining()` function of `DataSource` to read "rate" events (commented with "// MODIFIED"). Replace all `ViewEvent` with `RateEvent`. Replace all `viewEvent` with `rateEvent`. Retrieve the rating value from the event properties:
+Modify `readTraining()` function of `DataSource` to read "rate" events (commented with "// MODIFIED"). Replace all `ViewEvent` with `RateEvent`. Replace all `viewEventsRDD` with `rateEventsRDD`. Retrieve the rating value from the event properties:
```scala
override
def readTraining(sc: SparkContext): TrainingData = {
- val eventsDb = Storage.getPEvents()
-
- ....
+ ...
- // get all "user" "rate" "item" events
- val rateEventsRDD: RDD[RateEvent] = eventsDb.find( // MODIFIED
- appId = dsp.appId,
+ val eventsRDD: RDD[Event] = PEventStore.find(
+ appName = dsp.appName,
entityType = Some("user"),
- eventNames = Some(List("rate")), // MODIFIED
+ eventNames = Some(List("rate", "buy")), // MODIFIED
// targetEntityType is optional field of an event.
targetEntityType = Some(Some("item")))(sc)
- // eventsDb.find() returns RDD[Event]
+ .cache()
+
+ val rateEventsRDD: RDD[RateEvent] = eventsRDD // MODIFIED
+ .filter { event => event.event == "rate" } // MODIFIED
.map { event =>
- val rateEvent = try {
- event.event match {
- case "rate" => RateEvent( // MODIFIED
- user = event.entityId,
- item = event.targetEntityId.get,
- rating = event.properties.get[Double]("rating"), // ADDED
- t = event.eventTime.getMillis)
- case _ => throw new Exception(s"Unexpected event ${event} is read.")
- }
+ try {
+ RateEvent( // MODIFIED
+ user = event.entityId,
+ item = event.targetEntityId.get,
+ rating = event.properties.get[Double]("rating"), // ADDED
+ t = event.eventTime.getMillis
+ )
} catch {
- case e: Exception => {
+ case e: Exception =>
logger.error(s"Cannot convert ${event} to RateEvent." + // MODIFIED
s" Exception: ${e}.")
throw e
- }
}
- rateEvent
- }.cache()
+ }
+
+ ...
new TrainingData(
users = usersRDD,
items = itemsRDD,
- rateEvents = rateEventsRDD // MODIFIED
+ rateEvents = rateEventsRDD, // MODIFIED
+ buyEvents = buyEventsRDD
)
-
+ }
```
### Preparator.scala
-Modify Preparator to pass rateEvents to algorithm as PreparedData (Replace all `ViewEvent` with `RateEvent`. Replace all `viewEvent` with `rateEvent`)
+Modify Preparator to pass rateEvents to algorithm as PreparedData (Replace all `ViewEvent` with `RateEvent`. Replace all `viewEvents` with `rateEvents`)
Modify Preparator's `parpare()` method:
@@ -124,10 +125,9 @@ Modify Preparator's `parpare()` method:
new PreparedData(
users = trainingData.users,
items = trainingData.items,
- rateEvents = trainingData.rateEvents) // MODIFIED
+ rateEvents = trainingData.rateEvents, // MODIFIED
+ buyEvents = trainingData.buyEvents)
}
-
- ...
```
Modify `PreparedData` class:
@@ -136,7 +136,8 @@ Modify `PreparedData` class:
class PreparedData(
val users: RDD[(String, User)],
val items: RDD[(String, Item)],
- val rateEvents: RDD[RateEvent] // MODIFIED
+ val rateEvents: RDD[RateEvent], // MODIFIED
+ val buyEvents: RDD[BuyEvent]
) extends Serializable
```
@@ -147,23 +148,31 @@ Modify `train()` method to train with rate event.
```scala
- def train(sc: SparkContext, data: PreparedData): ALSModel = {
+ def train(sc: SparkContext, data: PreparedData): ECommModel = {
require(!data.rateEvents.take(1).isEmpty, // MODIFIED
s"rateEvents in PreparedData cannot be empty." + // MODIFIED
" Please check if DataSource generates TrainingData" +
" and Preprator generates PreparedData correctly.")
...
+ }
+
+ def genMLlibRating(
+ userStringIntMap: BiMap[String, Int],
+ itemStringIntMap: BiMap[String, Int],
+ data: PreparedData): RDD[MLlibRating] = {
val mllibRatings = data.rateEvents // MODIFIED
.map { r =>
...
((uindex, iindex), (r.rating, r.t)) // MODIFIED
- }.filter { case ((u, i), v) =>
+ }
+ .filter { case ((u, i), v) =>
// keep events with valid user and item index
(u != -1) && (i != -1)
- }.reduceByKey { case (v1, v2) => // MODIFIED
+ }
+ .reduceByKey { case (v1, v2) => // MODIFIED
// if a user may rate same item with different value at different times,
// use the latest value for this case.
// Can remove this reduceByKey() if no need to support this case.
@@ -175,9 +184,10 @@ Modify `train()` method to train with rate event.
.map { case ((u, i), (rating, t)) => // MODIFIED
// MLlibRating requires integer index for user and item
MLlibRating(u, i, rating) // MODIFIED
- }.cache()
+ }
+ .cache()
- ...
+ mllibRatings
}
```
@@ -194,7 +204,7 @@ Change the following from:
iterations = ap.numIterations,
lambda = ap.lambda,
blocks = -1,
- alpha = 1.0, // WILL BE REMOVED
+ alpha = 1.0,
seed = seed)
...
@@ -204,6 +214,7 @@ to:
```scala
...
+
val m = ALS.train( // MODIFIED
ratings = mllibRatings,
rank = ap.rank,
@@ -211,8 +222,8 @@ to:
lambda = ap.lambda,
blocks = -1,
seed = seed)
-
...
+
```
That's it! Now your engine can train model with rate events.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/javaecommercerecommendation/quickstart.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/javaecommercerecommendation/quickstart.html.md.erb b/docs/manual/source/templates/javaecommercerecommendation/quickstart.html.md.erb
index d7d4430..700ab1c 100644
--- a/docs/manual/source/templates/javaecommercerecommendation/quickstart.html.md.erb
+++ b/docs/manual/source/templates/javaecommercerecommendation/quickstart.html.md.erb
@@ -412,11 +412,11 @@ A Python import script `import_eventserver.py` is provided to import sample data
<%= partial 'shared/quickstart/install_python_sdk' %>
-Make sure you are under the `MyECommerceRecommendation` directory. Execute the following to import the data (Replace the value of access_key parameter with your **Access Key**):
+Make sure you are under the `MyECommerceRecommendation` directory. Execute the following to import the data:
```
$ cd MyECommerceRecommendation
-$ python data/import_eventserver.py --access_key 3mZWDzci2D5YsqAnqNnXH9SB6Rg3dsTBs8iHkK6X2i54IQsIZI1eEeQQyMfs7b3F
+$ python data/import_eventserver.py --access_key $ACCESS_KEY
```
You should see the following output:
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/leadscoring/quickstart.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/leadscoring/quickstart.html.md.erb b/docs/manual/source/templates/leadscoring/quickstart.html.md.erb
index 03413f8..5df6e13 100644
--- a/docs/manual/source/templates/leadscoring/quickstart.html.md.erb
+++ b/docs/manual/source/templates/leadscoring/quickstart.html.md.erb
@@ -288,11 +288,11 @@ A Python import script `import_eventserver.py` is provided to import sample data
<%= partial 'shared/quickstart/install_python_sdk' %>
-Make sure you are under the `MyLeadScoring` directory. Execute the following to import the data (Replace the value of access_key parameter with your **Access Key**):
+Make sure you are under the `MyLeadScoring` directory. Execute the following to import the data:
```
$ cd MyLeadScoring
-$ python data/import_eventserver.py --access_key 3mZWDzci2D5YsqAnqNnXH9SB6Rg3dsTBs8iHkK6X2i54IQsIZI1eEeQQyMfs7b3F
+$ python data/import_eventserver.py --access_key $ACCESS_KEY
```
You should see the following output:
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/productranking/quickstart.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/productranking/quickstart.html.md.erb b/docs/manual/source/templates/productranking/quickstart.html.md.erb
index bb0802e..1e299ae 100644
--- a/docs/manual/source/templates/productranking/quickstart.html.md.erb
+++ b/docs/manual/source/templates/productranking/quickstart.html.md.erb
@@ -299,11 +299,11 @@ A Python import script `import_eventserver.py` is provided to import sample data
<%= partial 'shared/quickstart/install_python_sdk' %>
-Make sure you are under the `MyProductRanking` directory. Execute the following to import the data (Replace the value of access_key parameter with your **Access Key**):
+Make sure you are under the `MyProductRanking` directory. Execute the following to import the data:
```
$ cd MyProductRanking
-$ python data/import_eventserver.py --access_key 3mZWDzci2D5YsqAnqNnXH9SB6Rg3dsTBs8iHkK6X2i54IQsIZI1eEeQQyMfs7b3F
+$ python data/import_eventserver.py --access_key $ACCESS_KEY
```
You should see the following output:
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/recommendation/blacklist-items.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/recommendation/blacklist-items.html.md b/docs/manual/source/templates/recommendation/blacklist-items.html.md
index b5e0e38..70d1585 100644
--- a/docs/manual/source/templates/recommendation/blacklist-items.html.md
+++ b/docs/manual/source/templates/recommendation/blacklist-items.html.md
@@ -21,9 +21,11 @@ limitations under the License.
Let's say you want to supply a backList for each query to exclude some items from recommendation (For example, in the browsing session, the user just added some items to shopping cart, or you have a list of items you want to filter out, you may want to supply blackList in Query). This how-to will demonstrate how you can do it.
-Note that you may also use [E-Commerce Recommendation Template](/gallery/template-gallery#recommender-systems) which supports this feature by default.
+You can find the complete modified source code [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-recommendation/blacklist-items).
-If you are looking for filtering out items based on the specific user-to-item events logged by EventServer (eg. filter all items which the user has "buy" events on), you can use the [E-Commerce Recommendation Template](/gallery/template-gallery#recommender-systems). Please refer to the algorithm parameters "unseenOnly" and "seenEvents" of the E-Commerce Recommenation Template.
+Note that you may also use [E-Commerce Recommendation Template](/templates/ecommercerecommendation/quickstart/) which supports this feature by default.
+
+If you are looking for filtering out items based on the specific user-to-item events logged by EventServer (eg. filter all items which the user has "buy" events on), you can use the [E-Commerce Recommendation Template](/templates/ecommercerecommendation/quickstart/). Please refer to the algorithm parameters "unseenOnly" and "seenEvents" of the E-Commerce Recommenation Template.
## Add Query Parameter
@@ -34,7 +36,7 @@ Lets modify `case class Query` in MyRecommendation/src/main/scala/***Engine.scal
case class Query(
user: String,
num: Int,
- blackList: Set[String] // NOTE: line added
+ blackList: Set[String] // ADDED
) extends Serializable
```
@@ -44,31 +46,35 @@ Then we need to change the code that computes recommendation score to filter out
Lets modify class MyRecommendation/src/main/scala/***ALSModel.scala***. Just add the following two methods to that class.
```scala
- def recommendProductsWithFilter(
- user: Int,
- num: Int,
- productIdFilter: Set[Int]) = {
+import com.github.fommil.netlib.BLAS.{getInstance => blas} // ADDED
+
+...
+
+ // ADDED
+ def recommendProductsWithFilter(user: Int, num: Int, productIdFilter: Set[Int]) = {
val filteredProductFeatures = productFeatures
- .filter(features => !productIdFilter.contains(features._1)) // (*)
+ .filter { case (id, _) => !productIdFilter.contains(id) } // (*)
recommend(userFeatures.lookup(user).head, filteredProductFeatures, num)
.map(t => Rating(user, t._1, t._2))
}
+ // ADDED
private def recommend(
recommendToFeatures: Array[Double],
recommendableFeatures: RDD[(Int, Array[Double])],
num: Int): Array[(Int, Double)] = {
- val recommendToVector = new DoubleMatrix(recommendToFeatures)
- val scored = recommendableFeatures.map { case (id,features) =>
- (id, recommendToVector.dot(new DoubleMatrix(features)))
+ val scored = recommendableFeatures.map { case (id, features) =>
+ (id, blas.ddot(features.length, recommendToFeatures, 1, features, 1))
}
scored.top(num)(Ordering.by(_._2))
}
+
+...
+
```
-Also it’s required to add import of the `org.jblas.DoubleMatrix` class.
Please make attention that method `recommend` is the copy of method `org.apache.spark.mllib.recommendation.MatrixFactorizationModel#recommend`. We can't reuse this because it’s private.
-Method `recommendProductsWithFilter` is the almost full copy of `org.apache.spark.mllib.recommendation.MatrixFactorizationModel#recommendProduct` method. The difference only is the line with commentary ‘(*)’ where we apply filtering.
+Method `recommendProductsWithFilter` is the almost full copy of `org.apache.spark.mllib.recommendation.MatrixFactorizationModel#recommendProducts` method. The difference only is the line with commentary ‘(*)’ where we apply filtering.
## Put It All Together
@@ -81,12 +87,12 @@ Lets modify method `predict` in MyRecommendation/src/main/scala/***ALSAlgorithm.
model.userStringIntMap.get(query.user).map { userInt =>
// create inverse view of itemStringIntMap
val itemIntStringMap = model.itemStringIntMap.inverse
- // recommendProducts() returns Array[MLlibRating], which uses item Int
+ // recommendProductsWithFilter() returns Array[MLlibRating], which uses item Int
// index. Convert it to String ID for returning PredictedResult
- val blackListedIds = query.blackList.flatMap(model.itemStringIntMap.get) // NOTE: line added
+ val blackList = query.blackList.flatMap(model.itemStringIntMap.get) // ADDED
val itemScores = model
- .recommendProductsWithFilter(userInt, query.num, blackListedIds) // NOTE: line modified
- .map(r => ItemScore(itemIntStringMap(r.product), r.rating))
+ .recommendProductsWithFilter(userInt, query.num, blackList) // MODIFIED
+ .map (r => ItemScore(itemIntStringMap(r.product), r.rating))
new PredictedResult(itemScores)
}.getOrElse{
logger.info(s"No prediction for unknown user ${query.user}.")
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/recommendation/customize-data-prep.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/recommendation/customize-data-prep.html.md b/docs/manual/source/templates/recommendation/customize-data-prep.html.md
index 1a48f1e..e353e85 100644
--- a/docs/manual/source/templates/recommendation/customize-data-prep.html.md
+++ b/docs/manual/source/templates/recommendation/customize-data-prep.html.md
@@ -34,11 +34,11 @@ A sample black list file containing the items to be excluded is provided in
`./data/sample_not_train_data.txt`.
A full end-to-end example can be found on
-[GitHub](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-recommendation/custom-prepartor).
+[GitHub](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-recommendation/customize-data-prep).
## The Data Preparator Component
-Recall [the DASE Architecture](/start/engines/), data is prepared by 2
+Recall [the DASE Architecture](/customize/), data is prepared by 2
components sequentially: *Data Source* and *Data Preparator*. *Data Source*
reads data from the data store of Event Server and then *Data Preparator*
prepares `RDD[Rating]` for the ALS algorithm.
@@ -147,7 +147,7 @@ Preparator!
Optionally, you may want to take the hardcoded path
(`./data/sample_not_train_data.txt`) away from the source code.
-PredictionIO offers `PreparatorParams` so you can read variable values from
+PredictionIO offers preparator params so you can read variable values from
`engine.json` instead.
Modify `src/main/scala/Preparator.scala` again in the *MyRecommendation*
@@ -156,7 +156,7 @@ directory to:
```scala
import org.apache.predictionio.controller.Params // ADDED
- // ADDED CustomPreparatorParams case class
+// ADDED CustomPreparatorParams case class
case class CustomPreparatorParams(
filepath: String
) extends Params
@@ -165,7 +165,7 @@ class Preparator(pp: CustomPreparatorParams) // ADDED CustomPreparatorParams
extends PPreparator[TrainingData, PreparedData] {
def prepare(sc: SparkContext, trainingData: TrainingData): PreparedData = {
- val noTrainItems = Source.fromFile(pp.filepath).getLines.toSet //CHANGED
+ val noTrainItems = Source.fromFile(pp.filepath).getLines.toSet // CHANGED
val ratings = trainingData.ratings.filter( r =>
!noTrainItems.contains(r.item)
)
@@ -199,4 +199,4 @@ $ pio deploy
You can change the `filepath` value without re-building the code next time.
-#### [Next: Customizing Serving](customize-serving.html)
+#### [Next: Customizing Serving Component](customize-serving.html)
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/recommendation/customize-serving.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/recommendation/customize-serving.html.md b/docs/manual/source/templates/recommendation/customize-serving.html.md
index 3dcdc6c..b3a3ec2 100644
--- a/docs/manual/source/templates/recommendation/customize-serving.html.md
+++ b/docs/manual/source/templates/recommendation/customize-serving.html.md
@@ -26,23 +26,11 @@ currently in stock from the list of recommendation.
This section is based on the [Recommendation Engine Template](/templates/recommendation/quickstart/).
A full end-to-end example can be found on
-[GitHub](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-recommendation/custom-serving).
-
-<!--
-This section demonstrates how to add a custom filtering logic to exclude a list
-of blacklisted movies from the [Movie Recommendation Engine](/quickstart.html)
-based on the Recommendation Engine Template. It is highly recommended to go
-through the Quckstart guide first.
-
-Complete code example can be found in
-`examples/scala-parallel-recommendation-howto`.
-
-If you simply want to use this customized code, you can skip to the last section.
--->
+[GitHub](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-recommendation/customize-serving).
## The Serving Component
-Recall [the DASE Architecture](/start/engines/), a PredictionIO engine has
+Recall [the DASE Architecture](/customize/), a PredictionIO engine has
4 main components: Data Source, Data Preparator, Algorithm, and Serving
components. When a Query comes in, it is passed to the Algorithm component for
making Predictions.
@@ -51,7 +39,8 @@ The Engine's serving component can be found in `src/main/scala/Serving.scala` in
the *MyRecommendation* directory. By default, it looks like the following:
```scala
-class Serving extends LServing[Query, PredictedResult] {
+class Serving
+ extends LServing[Query, PredictedResult] {
override
def serve(query: Query,
@@ -73,10 +62,13 @@ is called, it loads the file and removes items in the disabled list from
```scala
import scala.io.Source // ADDED
-class Serving extends LServing[Query, PredictedResult] {
+class Serving
+ extends LServing[Query, PredictedResult] {
- override def serve(query: Query, predictedResults: Seq[PredictedResult])
- : PredictedResult = {
+ override
+ def serve(query: Query,
+ predictedResults: Seq[PredictedResult]): PredictedResult = {
+ // MODIFIED HERE
// Read the disabled item from file.
val disabledProducts: Set[String] = Source
.fromFile("./data/sample_disabled_items.txt")
@@ -101,16 +93,16 @@ implementation for production deployment.
## Deploy the Modified Engine
-Now you can deploy the modified engine as described in the [Quick
-Start](/templates/recommendation/quickstart/) guide.
+Now you can deploy the modified engine as described in [Quick
+Start](quickstart.html).
-Make sure the `appId` defined in the file `engine.json` match your *App ID*:
+Make sure the `appName` defined in the file `engine.json` matches your *App Name*:
```
...
"datasource": {
- "params": {
- "appId": 1
+ "params" : {
+ "appName": "YourAppName"
}
},
...
@@ -178,7 +170,7 @@ logic to your Serving component!
Optionally, you may want to take the hardcoded path
(`./data/sample_disabled_items.txt`) away from the source code.
-PredictionIO offers `ServingParams` so you can read variable values from
+PredictionIO offers serving params so you can read variable values from
`engine.json` instead. PredictionIO transforms the JSON object specified in
`engine.json`'s `serving` field into the `ServingParams` class.
@@ -210,9 +202,7 @@ class Serving(val params: ServingParams)
}
```
-In `engine.json`, you specify the parameter `serving` for the Serving component
-(the JSON4S library automatically extract the JSON object into a Scala class
-under the hood):
+In `engine.json`, you define the parameters `serving` for the Serving component:
```json
{
@@ -226,7 +216,7 @@ under the hood):
}
```
-Again, to build *MyRecommendation* and deploy it as a service:
+Try to build *MyRecommendation* and deploy it again:
```
$ pio build
@@ -236,4 +226,4 @@ $ pio deploy
You can change the `filepath` value without re-building the code next time.
-#### [Next: Filter Recommended Items by Blacklist in Query](blacklist-items.html)
+#### [Next: Training with Implicit Preference](training-with-implicit-preference.html)
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/recommendation/how-to.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/recommendation/how-to.html.md b/docs/manual/source/templates/recommendation/how-to.html.md
index 0481913..54a6851 100644
--- a/docs/manual/source/templates/recommendation/how-to.html.md
+++ b/docs/manual/source/templates/recommendation/how-to.html.md
@@ -22,8 +22,8 @@ limitations under the License.
Here are the pages that show you how you can customize the Recommendation engine template.
- [Read Custom Events](/templates/recommendation/reading-custom-events/)
-- [Train with Implicit Preference](/templates/recommendation/training-with-implicit-preference/)
- [Customize Data Preparator](/templates/recommendation/customize-data-prep/)
- [Customize Serving](/templates/recommendation/customize-serving/)
+- [Train with Implicit Preference](/templates/recommendation/training-with-implicit-preference/)
- [Filter Recommended Items by Blacklist in Query](/templates/recommendation/blacklist-items/)
- [Batch Persistable Evaluator](/templates/recommendation/batch-evaluator/)
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/recommendation/quickstart.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/recommendation/quickstart.html.md.erb b/docs/manual/source/templates/recommendation/quickstart.html.md.erb
index fa81f22..bc0372d 100644
--- a/docs/manual/source/templates/recommendation/quickstart.html.md.erb
+++ b/docs/manual/source/templates/recommendation/quickstart.html.md.erb
@@ -263,8 +263,7 @@ Event Server using Python SDK. Please upgrade to the latest Python SDK.
<%= partial 'shared/quickstart/install_python_sdk' %>
-Replace the value of access_key parameter by your
-**Access Key** and run:
+Execute the following to import the data:
WARNING: These commands must be executed in the Engine directory, for example: `MyRecomendation`.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/recommendation/reading-custom-events.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/recommendation/reading-custom-events.html.md b/docs/manual/source/templates/recommendation/reading-custom-events.html.md
index ea603db..cdf502e 100644
--- a/docs/manual/source/templates/recommendation/reading-custom-events.html.md
+++ b/docs/manual/source/templates/recommendation/reading-custom-events.html.md
@@ -24,9 +24,11 @@ You can modify the [default DataSource](dase.html#data) to read
- Custom events other than the default **rate** and **buy** events.
- Events which involve different entity types other than the default **user** and **item**.
+You can find the complete modified source code [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-recommendation/reading-custom-events).
+
## Add the Custom Event
-To read custom events, modify the function call `eventsDb.find()` in MyRecommendation/src/main/scala/***DataSource.scala***:
+To read custom events, modify the function call `PEventStore.find()` in MyRecommendation/src/main/scala/***DataSource.scala***:
- Specify the names of events in `eventNames` parameters
- Specify the entity types involved in the events in the `entityType` and `targetEntityType` parameters accordingly
@@ -35,11 +37,10 @@ In this example below, we modify DataSource to read custom **like** and **dislik
```scala
-val eventsRDD: RDD[Event] = eventsDb.find(
- appId = dsp.appId,
+val eventsRDD: RDD[Event] = PEventStore.find(
+ appName = dsp.appName,
entityType = Some("customer"), // MODIFIED
eventNames = Some(List("like", "dislike")), // MODIFIED
-
// targetEntityType is optional field of an event.
targetEntityType = Some(Some("product")))(sc) // MODIFIED
```
@@ -56,8 +57,6 @@ val ratingsRDD: RDD[Rating] = eventsRDD.map { event =>
val ratingValue: Double = event.event match {
// MODIFIED
case "like" => 4.0 // map a like event to a rating of 4.0
-
-
case "dislike" => 1.0 // map a like event to a rating of 1.0
case _ => throw new Exception(s"Unexpected event ${event} is read.")
}
@@ -72,7 +71,7 @@ val ratingsRDD: RDD[Rating] = eventsRDD.map { event =>
}
}
rating
- }
+ }.cache()
```
That's it! Your engine can read custom **like** and **dislike** event.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/recommendation/training-with-implicit-preference.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/recommendation/training-with-implicit-preference.html.md b/docs/manual/source/templates/recommendation/training-with-implicit-preference.html.md
index 24e8bde..1279b47 100644
--- a/docs/manual/source/templates/recommendation/training-with-implicit-preference.html.md
+++ b/docs/manual/source/templates/recommendation/training-with-implicit-preference.html.md
@@ -24,17 +24,17 @@ There are two types of user preferences:
- explicit preference (also referred as "explicit feedback"), such as "rating" given to item by users.
- implicit preference (also referred as "implicit feedback"), such as "view" and "buy" history.
-MLlib ALS provides two functions, `ALS.train()` and `ALS.trainImplicit()` to handle these two cases, respectively. The ALS algorithm takes RDD[Rating] as training data input. The Rating class is defined in Spark MLlib library as:
+MLlib ALS provides the `setImplicitPrefs()` function to set whether to use implicit preference. The ALS algorithm takes RDD[Rating] as training data input. The Rating class is defined in Spark MLlib library as:
```
case class Rating(user: Int, product: Int, rating: Double)
```
-By default, the recommendation template uses `ALS.train()` which expects explicit rating values which the user has rated the item.
+By default, the recommendation template sets `setImplicitPrefs()` to `false` which expects explicit rating values which the user has rated the item.
-To handle implicit preference, `ALS.trainImplicit()` can be used. In this case, the "rating" value input to ALS is used to calculate the confidence level that the user likes the item. Higher "rating" means a stronger indication that the user likes the item.
+To handle implicit preference, you can set `setImplicitPrefs()` to `true`. In this case, the "rating" value input to ALS is used to calculate the confidence level that the user likes the item. Higher "rating" means a stronger indication that the user likes the item.
-The following provides an example of using implicit preference.
+The following provides an example of using implicit preference. You can find the complete modified source code [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-recommendation/train-with-view-event).
### Training with view events
@@ -44,17 +44,10 @@ First, we can modify `DataSource.scala` to aggregate the number of views of the
```scala
-class DataSource(val dsp: DataSourceParams)
- extends PDataSource[TrainingData,
- EmptyEvaluationInfo, Query, EmptyActualResult] {
+ def getRatings(sc: SparkContext): RDD[Rating] = {
- @transient lazy val logger = Logger[this.type]
-
- override
- def readTraining(sc: SparkContext): TrainingData = {
- val eventsDb = Storage.getPEvents()
- val eventsRDD: RDD[Event] = eventsDb.find(
- appId = dsp.appId,
+ val eventsRDD: RDD[Event] = PEventStore.find(
+ appName = dsp.appName,
entityType = Some("user"),
eventNames = Some(List("view")), // MODIFIED
// targetEntityType is optional field of an event.
@@ -84,16 +77,20 @@ class DataSource(val dsp: DataSourceParams)
Rating(uid, iid, r)
}.cache()
- new TrainingData(ratingsRDD)
+ ratingsRDD
+ }
+
+ override
+ def readTraining(sc: SparkContext): TrainingData = {
+ new TrainingData(getRatings(sc))
}
-}
```
NOTE: You may put the view count aggregation logic in `ALSAlgorithm`'s `train()` instead, depending on your needs.
-Then, we can modify ALSAlgorithm.scala to call `ALS.trainImplicit()` instead of `ALS.train()`:
+Then, we can modify ALSAlgorithm.scala to set `setImplicitPrefs` to `true`:
```scala
@@ -106,15 +103,21 @@ class ALSAlgorithm(val ap: ALSAlgorithmParams)
...
+ // If you only have one type of implicit event (Eg. "view" event only),
+ // set implicitPrefs to true
// MODIFIED
- val m = ALS.trainImplicit(
- ratings = mllibRatings,
- rank = ap.rank,
- iterations = ap.numIterations,
- lambda = ap.lambda,
- blocks = -1,
- alpha = 1.0,
- seed = seed)
+ val implicitPrefs = true
+ val als = new ALS()
+ als.setUserBlocks(-1)
+ als.setProductBlocks(-1)
+ als.setRank(ap.rank)
+ als.setIterations(ap.numIterations)
+ als.setLambda(ap.lambda)
+ als.setImplicitPrefs(implicitPrefs)
+ als.setAlpha(1.0)
+ als.setSeed(seed)
+ als.setCheckpointInterval(10)
+ val m = als.run(mllibRatings)
new ALSModel(
rank = m.rank,
@@ -131,3 +134,5 @@ class ALSAlgorithm(val ap: ALSAlgorithmParams)
```
Now the recommendation engine can train a model with implicit preference events.
+
+#### [Next: Filter Recommended Items by Blacklist in Query](blacklist-items.html)
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/similarproduct/how-to.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/similarproduct/how-to.html.md b/docs/manual/source/templates/similarproduct/how-to.html.md
index f47c73b..4ce2893 100644
--- a/docs/manual/source/templates/similarproduct/how-to.html.md
+++ b/docs/manual/source/templates/similarproduct/how-to.html.md
@@ -21,4 +21,8 @@ limitations under the License.
Here are the pages that show you how you can customize the Similar Product engine template.
-- [Multiple Events and Multiple Algorithms](/templates/similarproduct/multi-events-multi-algos)
+- [Multiple Events and Multiple Algorithms](/templates/similarproduct/multi-events-multi-algos/)
+- [Returns Item Properties](/templates/similarproduct/return-item-properties/)
+- [Train with Rate Event](/templates/similarproduct/train-with-rate-event/)
+- [Get Rid of Events for Users](/templates/similarproduct/rid-user-set-event/)
+- [Recommend Users](/templates/similarproduct/recommended-user/)
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/similarproduct/multi-events-multi-algos.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/similarproduct/multi-events-multi-algos.html.md.erb b/docs/manual/source/templates/similarproduct/multi-events-multi-algos.html.md.erb
index 25b154d..9c93750 100644
--- a/docs/manual/source/templates/similarproduct/multi-events-multi-algos.html.md.erb
+++ b/docs/manual/source/templates/similarproduct/multi-events-multi-algos.html.md.erb
@@ -33,17 +33,15 @@ This example will demonstrate the following:
- Use positive and negative implicit events such as like and dislike with MLlib ALS algorithm
- Integrate multiple algorithms into one engine
-The complete source code of this examlpe can be found in `examples/scala-parallel-similarproduct-multi`.
-
-INFO: Require PredictionIO version >= 0.8.6 to run this example.
+The complete source code of this examlpe can be found in [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-similarproduct/multi-events-multi-algos).
### Step 1. Read "like" and "dislike" events as TrainingData
Modify the following in DataSource.scala:
- In addition to the original `ViewEvent` class, add a new class `LikeEvent` which has a boolean `like` field to represent it's like or dislike event.
-- Add a new field `likeEvents` into `TrainingData` class to store the `RDD[LikeEvents]`.
-- Modidy DataSource's `read()` function to read "like" and "dislike" events from the Event Store.
+- Add a new field `likeEvents` into `TrainingData` class to store the `RDD[LikeEvent]`.
+- Modidy DataSource's `readTraining()` function to read "like" and "dislike" events from the Event Store.
The modification is shown below:
@@ -63,8 +61,8 @@ class DataSource(val dsp: DataSourceParams)
// ADDED
// get all "user" "like" and "dislike" "item" events
- val likeEventsRDD: RDD[LikeEvent] = eventsDb.find(
- appId = dsp.appId,
+ val likeEventsRDD: RDD[LikeEvent] = PEventStore.find(
+ appName = dsp.appName,
entityType = Some("user"),
eventNames = Some(List("like", "dislike")),
// targetEntityType is optional field of an event.
@@ -88,7 +86,7 @@ class DataSource(val dsp: DataSourceParams)
}
}
likeEvent
- }
+ }.cache()
new TrainingData(
users = usersRDD,
@@ -177,13 +175,12 @@ In summary, this new `LikeAlgorithm` does the following:
It is shown in the code below:
```scala
-package org.template.similarproduct
+package org.apache.predictionio.examples.similarproduct
import org.apache.predictionio.data.storage.BiMap
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
-import org.apache.spark.rdd.RDD
import org.apache.spark.mllib.recommendation.ALS
import org.apache.spark.mllib.recommendation.{Rating => MLlibRating}
@@ -236,7 +233,6 @@ class LikeAlgorithm(ap: ALSAlgorithmParams) extends ALSAlgorithm(ap) {
// key is (uindex, iindex) tuple, value is (like, t) tuple
((uindex, iindex), (r.like, r.t))
}.filter { case ((u, i), v) =>
- //val = d
// keep events with valid user and item index
(u != -1) && (i != -1)
}.reduceByKey { case (v1, v2) => // MODIFIED
@@ -272,7 +268,7 @@ class LikeAlgorithm(ap: ALSAlgorithmParams) extends ALSAlgorithm(ap) {
seed = seed)
new ALSModel(
- productFeatures = m.productFeatures,
+ productFeatures = m.productFeatures.collectAsMap.toMap,
itemStringIntMap = itemStringIntMap,
items = items
)
@@ -346,18 +342,20 @@ NOTE: You may combine results of multiple algorithms in different ways based on
### Step 5. Modify Engine.scala and engine.json
-Modify Engine.scala to include the new algorithm `LikeAlgorithm` class and give it the name `"likealgo"` as shown below. This defines an Engine with two available algorithms named as `"als"` and `"likealgo"`:
+Modify Engine.scala to include the new algorithm `LikeAlgorithm` class and give it the name `"likealgo"` as shown below:
```scala
...
-object SimilarProductEngine extends IEngineFactory {
+object SimilarProductEngine extends EngineFactory {
def apply() = {
new Engine(
classOf[DataSource],
classOf[Preparator],
- Map("als" -> classOf[ALSAlgorithm],
+ Map(
+ "als" -> classOf[ALSAlgorithm],
+ "cooccurrence" -> classOf[CooccurrenceAlgorithm],
"likealgo" -> classOf[LikeAlgorithm]), // ADDED
classOf[Serving])
}
@@ -406,11 +404,11 @@ That's it! Now you have a engine configured with two algorithms.
### Sample data with "like" and "dislike" events
-For demonstration purpose, a sample import script is also provided for you to quickly test this engine. The script is modified from the original one used in [Quick Start](quickstart.html#sample-data-for-quick-start) with the addition of importing like and dislike events.
+For demonstration purpose, a sample import script is also provided for you to quickly test this engine. The script is modified from the original one used in [Quick Start](quickstart.html#import-more-sample-data) with the addition of importing like and dislike events.
-You could find the import script in `examples/scala-parallel-similarproduct-multi/data/import_eventserver.py`.
+You could find the import script in `data/import_eventserver.py`.
-You can create a new App. Go to the directory `examples/scala-parallel-similarproduct-multi`. Execute the following to import the data (Replace the value of access_key parameter with your **Access Key**):
+Make sure you are under the App directory. Execute the following to import the data (Replace the value of access_key parameter with your **Access Key**):
```
$ python data/import_eventserver.py --access_key 3mZWDzci2D5YsqAnqNnXH9SB6Rg3dsTBs8iHkK6X2i54IQsIZI1eEeQQyMfs7b3F
@@ -420,4 +418,4 @@ WARNING: If you see error **TypeError: __init__() got an unexpected keyword argu
please update the Python SDK to the latest version.
-You are ready to run pio build, train and deploy as described in the [Quick Start](quickstart.html#deploy-the-engine-as-a-service).
+You are ready to run pio build, train and deploy as described in the [Quick Start](quickstart.html#5.-deploy-the-engine-as-a-service).
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/similarproduct/quickstart.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/similarproduct/quickstart.html.md.erb b/docs/manual/source/templates/similarproduct/quickstart.html.md.erb
index d2bf180..32f3706 100644
--- a/docs/manual/source/templates/similarproduct/quickstart.html.md.erb
+++ b/docs/manual/source/templates/similarproduct/quickstart.html.md.erb
@@ -335,11 +335,11 @@ A Python import script `import_eventserver.py` is provided to import sample data
<%= partial 'shared/quickstart/install_python_sdk' %>
-Make sure you are under the `MySimilarProduct` directory. Execute the following to import the data (Replace the value of access_key parameter with your **Access Key**):
+Make sure you are under the `MySimilarProduct` directory. Execute the following to import the data:
```
$ cd MySimilarProduct
-$ python data/import_eventserver.py --access_key 3mZWDzci2D5YsqAnqNnXH9SB6Rg3dsTBs8iHkK6X2i54IQsIZI1eEeQQyMfs7b3F
+$ python data/import_eventserver.py --access_key $ACCESS_KEY
```
You should see the following output:
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/similarproduct/recommended-user.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/similarproduct/recommended-user.html.md.erb b/docs/manual/source/templates/similarproduct/recommended-user.html.md.erb
new file mode 100644
index 0000000..b7f586f
--- /dev/null
+++ b/docs/manual/source/templates/similarproduct/recommended-user.html.md.erb
@@ -0,0 +1,188 @@
+---
+title: Recommend Users (Similar Product)
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+This examples demonstrates how to recommend users instead of items.
+
+Instead of using user-to-item events to find similar items, user-to-user events are used to find similar users you may also follow, like, etc (depending on which events are used in training and how the events are used). By default, "follow" events are used.
+
+You can find the complete modified source code [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-similarproduct/recommended-user).
+
+
+## Modification
+
+### Engine.scala
+
+In Query, change `items` to `users` and remove categories. Change `ItemScore` case class to SimilarUserScore. In PredictedResult, change `Array[ItemScore]` to `Array[SimilarUserScore]`.
+
+```scala
+case class Query(
+ users: List[String],
+ num: Int,
+ whiteList: Option[Set[String]],
+ blackList: Option[Set[String]]
+)
+
+case class PredictedResult(
+ similarUserScores: Array[SimilarUserScore]
+){
+ override def toString: String = similarUserScores.mkString(",")
+}
+
+case class SimilarUserScore(
+ user: String,
+ score: Double
+)
+```
+
+### DataSource.scala
+
+In DataSource, change `ViewEvent` case class to FollowEvent. Remove `Item` case class.
+
+Change
+
+```scala
+case class ViewEvent(user: String, item: String, t: Long)
+```
+
+to
+
+```scala
+// MODIFIED
+case class FollowEvent(user: String, followedUser: String, t: Long)
+```
+
+Modify TrainingData class to use followEvent
+
+```scala
+class TrainingData(
+ val users: RDD[(String, User)],
+ val followEvents: RDD[FollowEvent] // MODIFIED
+) extends Serializable {
+ override def toString = {
+ s"users: [${users.count()} (${users.take(2).toList}...)]" +
+ // MODIFIED
+ s"followEvents: [${followEvents.count()}] (${followEvents.take(2).toList}...)"
+ }
+}
+```
+
+Modify `readTraining()` function of `DataSource` to read "follow" events (commented with "// MODIFIED"). Remove the RDD of (entityID, Item):
+
+```scala
+
+ override
+ def readTraining(sc: SparkContext): TrainingData = {
+
+ // create a RDD of (entityID, User)
+ val usersRDD: RDD[(String, User)] = ...
+
+ // MODIFIED
+ // get all "user" "follow" "followedUser" events
+ val followEventsRDD: RDD[FollowEvent] = PEventStore.find(
+ appName = dsp.appName,
+ entityType = Some("user"),
+ eventNames = Some(List("follow")),
+ // targetEntityType is optional field of an event.
+ targetEntityType = Some(Some("user")))(sc)
+ // eventsDb.find() returns RDD[Event]
+ .map { event =>
+ val followEvent = try {
+ event.event match {
+ case "follow" => FollowEvent(
+ user = event.entityId,
+ followedUser = event.targetEntityId.get,
+ t = event.eventTime.getMillis)
+ case _ => throw new Exception(s"Unexpected event $event is read.")
+ }
+ } catch {
+ case e: Exception => {
+ logger.error(s"Cannot convert $event to FollowEvent." +
+ s" Exception: $e.")
+ throw e
+ }
+ }
+ followEvent
+ }.cache()
+
+ new TrainingData(
+ users = usersRDD,
+ followEvents = followEventsRDD // MODIFIED
+ )
+ }
+```
+
+### Preparator.scala
+
+Modify Preparator to pass followEvents to algorithm as PreparedData.
+
+Modify Preparator's `parpare()` method:
+
+```scala
+
+ ...
+
+ def prepare(sc: SparkContext, trainingData: TrainingData): PreparedData = {
+ new PreparedData(
+ users = trainingData.users,
+ followEvents = trainingData.followEvents) // MODIFIED
+ }
+```
+
+Modify `PreparedData` class:
+
+```scala
+class PreparedData(
+ val users: RDD[(String, User)],
+ val followEvents: RDD[FollowEvent] // MODIFIED
+) extends Serializable
+
+```
+
+### ALSAlgorithm.scala
+
+Modify ALSModel class to use similar user. Modify `train()` method to train with follow event. Modify `predict()` method to predict similar users.
+
+### Test the Result
+
+Then we can build/train/deploy the engine and test the result:
+
+The query
+
+```bash
+$ curl -H "Content-Type: application/json" \
+-d '{ "users": ["u1"], "num": 4 }' \
+http://localhost:8000/queries.json
+```
+
+will return the result
+
+```json
+{
+ "similarUserScores":[
+ {"user":"u3","score":0.7574200014043541},
+ {"user":"u10","score":0.6484507108863744},
+ {"user":"u43","score":0.64741489488357},
+ {"user":"u29","score":0.5767264820728124}
+ ]
+}
+```
+
+That's it! Now your engine can recommend users.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/similarproduct/return-item-properties.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/similarproduct/return-item-properties.html.md.erb b/docs/manual/source/templates/similarproduct/return-item-properties.html.md.erb
new file mode 100644
index 0000000..a23518e
--- /dev/null
+++ b/docs/manual/source/templates/similarproduct/return-item-properties.html.md.erb
@@ -0,0 +1,146 @@
+---
+title: Returns Item Properties (Similar Product)
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+You can modify the [default DataSource](dase.html#data) to read your custom properties or different Entity Type.
+
+This explains how to add user defined properties to items returned by your engine. We add properties "title", "date" and "imdbUrl" for entity type "item".
+
+You can find the complete modified source code [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-similarproduct/return-item-properties).
+
+>> Note: you also need import events with these properties accordingly.
+
+## Modification
+
+### DataSource.scala
+
+- modify the `Item` parameters
+- modify how to create the `Item` object using the entity properties
+
+```scala
+
+// MODIFIED
+case class Item(
+ title: String,
+ date: String,
+ imdbUrl: String,
+ categories: Option[List[String]])
+
+...
+
+ override
+ def readTraining(sc: SparkContext): TrainingData = {
+ ...
+ // create a RDD of (entityID, Item)
+ val itemsRDD: RDD[(String, Item)] = PEventStore.aggregateProperties(
+ appName = dsp.appName,
+ entityType = "item"
+ )(sc).map { case (entityId, properties) =>
+ val item = try {
+ // Assume categories is optional property of item.
+ // MODIFIED
+ Item(
+ title = properties.get[String]("title"),
+ date = properties.get[String]("date"),
+ imdbUrl = properties.get[String]("imdbUrl"),
+ categories = properties.getOpt[List[String]]("categories"))
+ } catch {
+ case e: Exception => {
+ logger.error(s"Failed to get properties ${properties} of" +
+ s" item ${entityId}. Exception: ${e}.")
+ throw e
+ }
+ }
+ (entityId, item)
+ }.cache()
+
+ ...
+ }
+```
+
+### Engine.scala
+
+Modify the `ItemScore` parameters too.
+
+```scala
+// MODIFIED
+case class ItemScore(
+ item: String,
+ title: String,
+ date: String,
+ imdbUrl: String,
+ score: Double
+) extends Serializable
+```
+
+### ALSAlgorithm.scala
+
+Modify how to create the ItemScore object using the properties.
+
+```scala
+
+ def predict(model: ALSModel, query: Query): PredictedResult = {
+ ...
+
+ val itemScores = topScores.map { case (i, s) =>
+ // MODIFIED
+ val it = model.items(i)
+ new ItemScore(
+ item = model.itemIntStringMap(i),
+ title = it.title,
+ date = it.date,
+ imdbUrl = it.imdbUrl,
+ score = s
+ )
+ }
+
+ ...
+ }
+
+```
+
+Using `model.items(i)` you can receive corresponding object of the `Item` class, and now you can access its properties which you created during previous step.
+
+### Test the Result
+
+Then we can build/train/deploy the engine and test the result:
+
+The query
+
+```bash
+$ curl -H "Content-Type: application/json" \
+-d '{ "items": ["i1"], "num": 4 }' \
+http://localhost:8000/queries.json
+```
+
+will return the result
+
+```json
+{
+ "itemScores":[
+ {"item":"i3","title":"title for movie i3","date":"1947","imdbUrl":"http://imdb.com/fake-url/i3","score":0.5865418718902017},
+ {"item":"i44","title":"title for movie i44","date":"1941","imdbUrl":"http://imdb.com/fake-url/i44","score":0.5740199916714374},
+ {"item":"i37","title":"title for movie i37","date":"1940","imdbUrl":"http://imdb.com/fake-url/i37","score":0.5576820095310056},
+ {"item":"i6","title":"title for movie i6","date":"1947","imdbUrl":"http://imdb.com/fake-url/i6","score":0.45856345689769473}
+ ]
+}
+```
+
+That's it! Your engine can return more information.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/similarproduct/rid-user-set-event.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/similarproduct/rid-user-set-event.html.md.erb b/docs/manual/source/templates/similarproduct/rid-user-set-event.html.md.erb
new file mode 100644
index 0000000..551ad4f
--- /dev/null
+++ b/docs/manual/source/templates/similarproduct/rid-user-set-event.html.md.erb
@@ -0,0 +1,134 @@
+---
+title: Get Rid of Events for Users (Similar Product)
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+In some cases, if you don't need to keep track the user ID being created/deleted or user properties changes with events, then you can simplify the template as described in this example to get rid of '$set' events for users.
+
+You can find the complete modified source code [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-similarproduct/rid-user-set-event).
+
+
+## Modification
+
+### DataSource.scala
+
+Modify TrainingData class to remove the RDD of users.
+
+```scala
+class TrainingData(
+ val items: RDD[(String, Item)],
+ val viewEvents: RDD[ViewEvent]
+) extends Serializable {
+ override def toString = {
+ s"items: [${items.count()} (${items.take(2).toList}...)]" +
+ s"viewEvents: [${viewEvents.count()}] (${viewEvents.take(2).toList}...)"
+ }
+}
+```
+
+Modify `readTraining()` function of `DataSource` to remove the RDD of (entityID, User).
+
+```scala
+
+ override
+ def readTraining(sc: SparkContext): TrainingData = {
+
+ // REMOVED
+ // create a RDD of (entityID, User)
+ val usersRDD: RDD[(String, User)] = PEventStore.aggregateProperties(
+ appName = dsp.appName,
+ entityType = "user"
+ )(sc).map { case (entityId, properties) =>
+ val user = try {
+ User()
+ } catch {
+ case e: Exception => {
+ logger.error(s"Failed to get properties ${properties} of" +
+ s" user ${entityId}. Exception: ${e}.")
+ throw e
+ }
+ }
+ (entityId, user)
+ }.cache()
+
+ ...
+
+ new TrainingData(
+ items = itemsRDD,
+ viewEvents = viewEventsRDD
+ )
+ }
+```
+
+### Preparator.scala
+
+Modify Preparator to remove the RDD of users.
+
+Modify Preparator's `parpare()` method:
+
+```scala
+
+ ...
+
+ def prepare(sc: SparkContext, trainingData: TrainingData): PreparedData = {
+ new PreparedData(
+ items = trainingData.items,
+ viewEvents = trainingData.viewEvents)
+ }
+```
+
+Modify `PreparedData` class:
+
+```scala
+class PreparedData(
+ val items: RDD[(String, Item)],
+ val viewEvents: RDD[ViewEvent]
+) extends Serializable
+```
+
+### ALSAlgorithm.scala
+
+Modify `train()` method:
+
+- remove the check of users in PreparedData
+- modify user index BiMap to extract the user ID from the `viewEvents`
+
+```scala
+
+ def train(sc: SparkContext, data: PreparedData): ECommModel = {
+ ...
+ // REMOVED
+ require(!data.users.take(1).isEmpty,
+ s"users in PreparedData cannot be empty." +
+ " Please check if DataSource generates TrainingData" +
+ " and Preprator generates PreparedData correctly.")
+ ...
+ // create User and item's String ID to integer index BiMap
+ val userStringIntMap = BiMap.stringInt(data.viewEvents.map(_.user)) // MODIFIED
+ val itemStringIntMap = BiMap.stringInt(data.items.keys)
+
+ ...
+
+ }
+
+```
+
+You are ready to run pio build, train and deploy as described in the Quick Start. Simply send the same queries as described in the Quick Start. The result will be the same.
+
+That's it! Now your engine can get rid of '$set' events for users.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/75cfe461/docs/manual/source/templates/similarproduct/train-with-rate-event.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/templates/similarproduct/train-with-rate-event.html.md.erb b/docs/manual/source/templates/similarproduct/train-with-rate-event.html.md.erb
new file mode 100644
index 0000000..1f5c027
--- /dev/null
+++ b/docs/manual/source/templates/similarproduct/train-with-rate-event.html.md.erb
@@ -0,0 +1,217 @@
+---
+title: Train with Rate Event (Similar Product)
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+By default, the similar product template uses implicit preference, such as "view" event.
+
+To handle explicit preference, such as "rating" given to item by users, you can customize the template. Higher "rating" means a stronger indication that the user likes the item.
+
+This examples demonstrates how to modify similar product template to use "rate" event as Training Data. You can find the complete modified source code [here](https://github.com/apache/incubator-predictionio/tree/develop/examples/scala-parallel-similarproduct/train-with-rate-event).
+
+
+## Modification
+
+### DataSource.scala
+
+In DataSource, change `ViewEvent` case class to RateEvent. Add `rating: Double` is added to the RateEvent.
+
+Change
+
+```scala
+case class ViewEvent(user: String, item: String, t: Long)
+```
+
+to
+
+```scala
+// MODIFIED
+case class RateEvent(user: String, item: String, rating: Double, t: Long)
+```
+
+Modify TrainingData class to use rateEvent
+
+```scala
+class TrainingData(
+ val users: RDD[(String, User)],
+ val items: RDD[(String, Item)],
+ val rateEvents: RDD[RateEvent] // MODIFIED
+) extends Serializable {
+ override def toString = {
+ s"users: [${users.count()} (${users.take(2).toList}...)]" +
+ s"items: [${items.count()} (${items.take(2).toList}...)]" +
+ // MODIFIED
+ s"rateEvents: [${rateEvents.count()}] (${rateEvents.take(2).toList}...)"
+ }
+}
+```
+
+Modify `readTraining()` function of `DataSource` to read "rate" events (commented with "// MODIFIED"). Replace all `ViewEvent` with `RateEvent`. Replace all `viewEventsRDD` with `rateEventsRDD`. Retrieve the rating value from the event properties:
+
+```scala
+
+ override
+ def readTraining(sc: SparkContext): TrainingData = {
+ ...
+
+ // get all "user" "rate" "item" events
+ val rateEventsRDD: RDD[RateEvent] = PEventStore.find( // MODIFIED
+ appName = dsp.appName,
+ entityType = Some("user"),
+ eventNames = Some(List("rate")), // MODIFIED
+ // targetEntityType is optional field of an event.
+ targetEntityType = Some(Some("item")))(sc)
+ // eventsDb.find() returns RDD[Event]
+ .map { event =>
+ val rateEvent = try { // MODIFIED
+ event.event match {
+ case "rate" => RateEvent( // MODIFIED
+ user = event.entityId,
+ item = event.targetEntityId.get,
+ rating = event.properties.get[Double]("rating"), // ADDED
+ t = event.eventTime.getMillis)
+ case _ => throw new Exception(s"Unexpected event ${event} is read.")
+ }
+ } catch {
+ case e: Exception => {
+ logger.error(s"Cannot convert ${event} to RateEvent." + // MODIFIED
+ s" Exception: ${e}.")
+ throw e
+ }
+ }
+ rateEvent // MODIFIED
+ }.cache()
+
+ new TrainingData(
+ users = usersRDD,
+ items = itemsRDD,
+ rateEvents = rateEventsRDD // MODIFIED
+ )
+ }
+```
+
+### Preparator.scala
+
+Modify Preparator to pass rateEvents to algorithm as PreparedData (Replace all `ViewEvent` with `RateEvent`. Replace all `viewEvents` with `rateEvents`)
+
+Modify Preparator's `parpare()` method:
+
+```scala
+
+ ...
+
+ def prepare(sc: SparkContext, trainingData: TrainingData): PreparedData = {
+ new PreparedData(
+ users = trainingData.users,
+ items = trainingData.items,
+ rateEvents = trainingData.rateEvents) // MODIFIED
+ }
+```
+
+Modify `PreparedData` class:
+
+```scala
+class PreparedData(
+ val users: RDD[(String, User)],
+ val items: RDD[(String, Item)],
+ val rateEvents: RDD[RateEvent] // MODIFIED
+) extends Serializable
+
+```
+
+### ALSAlgorithm.scala
+
+Modify `train()` method to train with rate event.
+
+```scala
+
+ def train(sc: SparkContext, data: PreparedData): ECommModel = {
+ require(!data.rateEvents.take(1).isEmpty, // MODIFIED
+ s"rateEvents in PreparedData cannot be empty." + // MODIFIED
+ " Please check if DataSource generates TrainingData" +
+ " and Preprator generates PreparedData correctly.")
+
+ ...
+
+ val mllibRatings = data.rateEvents // MODIFIED
+ .map { r =>
+ ...
+
+ ((uindex, iindex), (r.rating,r.t)) //MODIFIED
+ }.filter { case ((u, i), v) =>
+ // keep events with valid user and item index
+ (u != -1) && (i != -1)
+ }
+ .reduceByKey { case (v1, v2) => // MODIFIED
+ // if a user may rate same item with different value at different times,
+ // use the latest value for this case.
+ // Can remove this reduceByKey() if no need to support this case.
+ val (rating1, t1) = v1
+ val (rating2, t2) = v2
+ // keep the latest value
+ if (t1 > t2) v1 else v2
+ }
+ .map { case ((u, i), (rating, t)) => // MODIFIED
+ // MLlibRating requires integer index for user and item
+ MLlibRating(u, i, rating) // MODIFIED
+ }
+ .cache()
+
+ ...
+
+ }
+
+```
+
+Modify `train()` method to use `ALS.trainImplicit()`:
+
+Change the following from:
+
+```scala
+ ...
+
+ val m = ALS.trainImplicit(
+ ratings = mllibRatings,
+ rank = ap.rank,
+ iterations = ap.numIterations,
+ lambda = ap.lambda,
+ blocks = -1,
+ alpha = 1.0,
+ seed = seed)
+ ...
+
+```
+
+to:
+
+```scala
+ ...
+
+ val m = ALS.train( // MODIFIED
+ ratings = mllibRatings,
+ rank = ap.rank,
+ iterations = ap.numIterations,
+ lambda = ap.lambda,
+ blocks = -1,
+ seed = seed)
+ ...
+
+```
+
+That's it! Now your engine can train model with rate events.
[2/2] incubator-predictionio git commit: Merge branch 'livedoc' into
develop
Posted by sh...@apache.org.
Merge branch 'livedoc' into develop
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/b6f168af
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/b6f168af
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/b6f168af
Branch: refs/heads/develop
Commit: b6f168afb17efcff01379dbc61783ffc4acd5f1f
Parents: 76f3409 75cfe46
Author: shimamoto <sh...@apache.org>
Authored: Mon Jul 10 13:48:59 2017 +0900
Committer: shimamoto <sh...@apache.org>
Committed: Mon Jul 10 13:48:59 2017 +0900
----------------------------------------------------------------------
docs/manual/data/nav/main.yml | 18 +-
.../classification/add-algorithm.html.md | 8 +-
.../classification/quickstart.html.md.erb | 4 +-
.../reading-custom-properties.html.md | 21 +-
.../classification/transform-data.html.md | 16 --
.../quickstart.html.md.erb | 4 +-
.../adjust-score.html.md.erb | 186 ++++++++++++++++
.../ecommercerecommendation/how-to.html.md | 1 +
.../quickstart.html.md.erb | 4 +-
.../train-with-rate-event.html.md.erb | 89 ++++----
.../quickstart.html.md.erb | 4 +-
.../leadscoring/quickstart.html.md.erb | 4 +-
.../productranking/quickstart.html.md.erb | 4 +-
.../recommendation/blacklist-items.html.md | 40 ++--
.../recommendation/customize-data-prep.html.md | 12 +-
.../recommendation/customize-serving.html.md | 48 ++--
.../templates/recommendation/how-to.html.md | 2 +-
.../recommendation/quickstart.html.md.erb | 3 +-
.../reading-custom-events.html.md | 13 +-
.../training-with-implicit-preference.html.md | 55 ++---
.../templates/similarproduct/how-to.html.md | 6 +-
.../multi-events-multi-algos.html.md.erb | 36 ++-
.../similarproduct/quickstart.html.md.erb | 4 +-
.../similarproduct/recommended-user.html.md.erb | 188 ++++++++++++++++
.../return-item-properties.html.md.erb | 146 +++++++++++++
.../rid-user-set-event.html.md.erb | 134 ++++++++++++
.../train-with-rate-event.html.md.erb | 217 +++++++++++++++++++
27 files changed, 1076 insertions(+), 191 deletions(-)
----------------------------------------------------------------------