You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@predictionio.apache.org by do...@apache.org on 2016/07/26 19:14:45 UTC
[01/15] incubator-predictionio git commit: Relax Ruby version
restriction
Repository: incubator-predictionio
Updated Branches:
refs/heads/develop 1e0d43580 -> 0e7563f83
Relax Ruby version restriction
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/3919d256
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/3919d256
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/3919d256
Branch: refs/heads/develop
Commit: 3919d2566d80510044704b2c7836465783f13565
Parents: da837da
Author: Donald Szeto <do...@prediction.io>
Authored: Fri Apr 8 14:42:14 2016 -0700
Committer: Donald Szeto <do...@prediction.io>
Committed: Fri Apr 8 14:42:14 2016 -0700
----------------------------------------------------------------------
docs/manual/Gemfile | 1 -
1 file changed, 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/3919d256/docs/manual/Gemfile
----------------------------------------------------------------------
diff --git a/docs/manual/Gemfile b/docs/manual/Gemfile
index 2a20508..f4df6bc 100644
--- a/docs/manual/Gemfile
+++ b/docs/manual/Gemfile
@@ -1,5 +1,4 @@
source 'https://rubygems.org'
-ruby '2.2.1'
gem 'middleman', '~> 3.3.10'
gem 'middleman-livereload', '~> 3.4.2'
[07/15] incubator-predictionio git commit: monit documentation
Posted by do...@apache.org.
monit documentation
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/28ca0618
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/28ca0618
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/28ca0618
Branch: refs/heads/develop
Commit: 28ca0618d8a6ba94ca4d5b2b5227bb7c0735a27c
Parents: 4f39d72
Author: alex9311 <al...@gmail.com>
Authored: Fri Apr 22 10:48:42 2016 +0200
Committer: alex9311 <al...@gmail.com>
Committed: Fri Apr 22 10:48:42 2016 +0200
----------------------------------------------------------------------
docs/manual/data/nav/main.yml | 3 +
docs/manual/source/deploy/monitoring.html.md | 154 ++++++++++++++++++++++
docs/manual/source/images/monit.png | Bin 0 -> 192080 bytes
3 files changed, 157 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/28ca0618/docs/manual/data/nav/main.yml
----------------------------------------------------------------------
diff --git a/docs/manual/data/nav/main.yml b/docs/manual/data/nav/main.yml
index c9f4485..66fa14e 100644
--- a/docs/manual/data/nav/main.yml
+++ b/docs/manual/data/nav/main.yml
@@ -62,6 +62,9 @@ root:
body: 'Engine Command-line Interface'
url: '/cli/#engine-commands'
-
+ body: 'Monitoring Engine'
+ url: '/deploy/monitoring/'
+ -
body: 'Setting Engine Parameters'
url: '/deploy/engineparams/'
-
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/28ca0618/docs/manual/source/deploy/monitoring.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/deploy/monitoring.html.md b/docs/manual/source/deploy/monitoring.html.md
new file mode 100644
index 0000000..c566bce
--- /dev/null
+++ b/docs/manual/source/deploy/monitoring.html.md
@@ -0,0 +1,154 @@
+---
+title: Monitoring an Engine
+---
+
+If you're using PredictionIO in a production setting, you'll want some way to make sure it is always up. [Monit](https://mmonit.com/monit/) is a tool which will monitor important processes and programs. This guide will show how to set up monit on your PredictionIO server to keep an engine always up and running.
+
+You can install monit on ubuntu with
+
+```bash
+sudo apt-get install monit
+```
+
+##Configure Basics
+Now we can configure monit by the configuration file `/etc/monit/monitrc` with your favorite editor. You will notice that this file contains quite a bit already, most of which is commented intructions/examples.
+
+First, choose the interval on which you want monit to check the status of your system. Use the `set daemon` command for this, it should already exist in the configuration file.
+
+```
+set daemon 60 #checks at 1-minute intervals
+```
+
+The `check system` block should also already be present, under the services block.
+
+```
+ check system 127.0.0.1
+ if memory usage > 75% then alert
+ if swap usage > 25% then alert
+ if loadavg (1min) > 4 then alert
+ if loadavg (5min) > 2 then alert
+ if cpu usage (user) > 70% then alert
+ if cpu usage (system) > 30% then alert
+ if cpu usage (wait) > 20% then alert
+```
+
+You might also want to configure the built in web server.
+
+```
+set httpd port 2812
+ allow admin:yourpassword # require user 'admin' with password 'yourpassword'
+```
+More examples on configuring the web server are included in the default config file.
+
+Configuration blocks for common services like apache, nginx, or PostgreSQL can be found [here](http://www.stuartellis.eu/articles/monit/)
+
+##Configure for PredictionIO
+###Event Server
+Now the interesting stuff, lets add monitoring for the event server.
+
+```
+check process eventserver
+ matching "Console eventserver"
+ start program = "/etc/monit/modebug /home/ubuntu/event_scripts.sh start"
+ stop program = "/etc/monit/modebug /home/ubuntu/event_scripts.sh stop"
+ if cpu usage > 95% for 10 cycles then restart
+```
+This block references a script, event_scripts.sh. This script tell monit how to restart the engine and event server if they go down.
+
+The script might differ slightly depending on your environment but it should look something like what is shown below. Assume SimilarProduct is the your pio app directory.
+
+```bash
+#!/bin/bash
+ case $1 in
+ start)
+ cd /home/ubuntu/SimilarProduct/
+ nohup /opt/PredictionIO/bin/pio eventserver > /home/ubuntu/events.log &
+ ;;
+ stop)
+ event_pid=`pgrep -f "Console eventserver"`
+ kill "$event_pid"
+ ;;
+ *)
+ esac
+ exit 0
+```
+Note that this is dumping output to an events log at `/home/ubuntu/events.log`. Also, be sure that this file is executable with `sudo chmod +x event_scripts.sh`
+
+###Engine
+The first step here is similar to checking the engine process.
+
+```
+check process pioengine
+ matching "Console deploy"
+ start program = "/etc/monit/modebug /home/ubuntu/engine_scripts.sh start"
+ stop program = "/etc/monit/modebug /home/ubuntu/engine_scripts.sh stop"
+ if cpu usage > 95% for 10 cycles then restart
+```
+Be sure to adjust your deploy command to your environment (driver-memry, postgres jar path)
+
+```bash
+#!/bin/bash
+ case $1 in
+ start)
+ cd /home/ubuntu/SimilarProduct/
+ nohup /opt/PredictionIO/bin/pio deploy -- --driver-class-path /home/ubuntu/postgresql-9.4.1208.jre6.jar --driver-memory 16G > /home/ubuntu/deploy.log &
+ ;;
+ stop)
+ deploy_pid=`pgrep -f "Console deploy"`
+ kill "$deploy_pid"
+ ;;
+ *)
+ esac
+ exit 0
+```
+
+There can be cases when the process is running but the engine is down however. If the spray REST API used by PredictionIO crashes, the engine process continues but the engine to fail when queried.
+
+This sort of crash can be taken care of by using monits `check program` capability.
+
+```
+check program pioengine-http with path "/etc/monit/bin/check_engine.sh"
+ start program = "/etc/monit/modebug /home/ubuntu/engine_scripts.sh start"
+ stop program = "/etc/monit/modebug /home/ubuntu/engine_scripts.sh stop"
+ if status != 1
+ then restart
+```
+This block executes the script at /etc/monit/bin/check_engine.sh and reads the exit status. Depending on the exit status, the block can run a restart script. The restart script can be the same as what is used in the process monitor, but we need a check_engine script.
+
+```bash
+#!/bin/bash
+# source: /etc/monit/bin/check_engine.sh
+url="http://127.0.0.1:8000/queries.json"
+check_string="itemScores"
+response=$(curl -H "Content-Type: application/json" -d '{ "user": "1", "num": 0}' $url)
+
+if [[ "$response" =~ "$check_string" ]]
+then
+ exit 1
+else
+ exit 0
+fi
+```
+This script does a curl request and checks the response. In this example, a user known to exist is used and then check make sure the json returned has "itemScores". This can vary between use cases but the idea should be similar.
+
+Again, make sure this file is executable.
+
+##Start it All Up
+Now we can get monit running with
+
+```bash
+sudo service monit restart
+```
+
+Navigate to http://\<your ip\>:2812/ to check out your status page
+
+
+
+##Testing
+To test, try killing your deployed engine or event server and see if monit brings it back up. You can even use the scripts we described above to do this
+
+```
+sudo ./engine_scripts.sh stop
+```
+
+Remember that monit checks only as often as you tell it to, so it may need a few minutes.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/28ca0618/docs/manual/source/images/monit.png
----------------------------------------------------------------------
diff --git a/docs/manual/source/images/monit.png b/docs/manual/source/images/monit.png
new file mode 100644
index 0000000..f8b35ac
Binary files /dev/null and b/docs/manual/source/images/monit.png differ
[02/15] incubator-predictionio git commit: Update bug tracker link
Posted by do...@apache.org.
Update bug tracker link
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/acd0e2f0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/acd0e2f0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/acd0e2f0
Branch: refs/heads/develop
Commit: acd0e2f0234a8d082c341f00d9b1b8dba459c76c
Parents: 3919d25
Author: Donald Szeto <do...@prediction.io>
Authored: Mon Apr 11 10:27:51 2016 -0700
Committer: Donald Szeto <do...@prediction.io>
Committed: Mon Apr 11 10:27:51 2016 -0700
----------------------------------------------------------------------
docs/manual/Gemfile | 1 -
docs/manual/Gemfile.lock | 6 ------
docs/manual/source/community/contribute-code.html.md | 11 +++++------
docs/manual/source/partials/_action_call.html.slim | 2 +-
docs/manual/source/partials/_footer.html.slim | 2 +-
5 files changed, 7 insertions(+), 15 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/acd0e2f0/docs/manual/Gemfile
----------------------------------------------------------------------
diff --git a/docs/manual/Gemfile b/docs/manual/Gemfile
index f4df6bc..a0c588c 100644
--- a/docs/manual/Gemfile
+++ b/docs/manual/Gemfile
@@ -9,7 +9,6 @@ gem 'middleman-s3_sync'
gem 'middleman-search_engine_sitemap'
gem 'slim'
-gem 'therubyracer'
gem 'oj'
gem 'redcarpet', '>= 3.2.3'
gem 'travis'
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/acd0e2f0/docs/manual/Gemfile.lock
----------------------------------------------------------------------
diff --git a/docs/manual/Gemfile.lock b/docs/manual/Gemfile.lock
index 5cfff32..495ec1c 100644
--- a/docs/manual/Gemfile.lock
+++ b/docs/manual/Gemfile.lock
@@ -91,7 +91,6 @@ GEM
kramdown (1.6.0)
launchy (2.4.3)
addressable (~> 2.3)
- libv8 (3.16.14.7)
listen (2.10.0)
celluloid (~> 0.16.0)
rb-fsevent (>= 0.9.3)
@@ -182,7 +181,6 @@ GEM
rb-inotify (0.9.5)
ffi (>= 0.5.0)
redcarpet (3.3.4)
- ref (1.0.5)
rouge (1.8.0)
ruby-progressbar (1.7.5)
sass (3.4.22)
@@ -200,9 +198,6 @@ GEM
sprockets (~> 2.0)
tilt (~> 1.1)
temple (0.7.6)
- therubyracer (0.12.2)
- libv8 (~> 3.16.14.0)
- ref
thor (0.19.1)
thread_safe (0.3.5)
tilt (1.4.1)
@@ -247,7 +242,6 @@ DEPENDENCIES
rainbow
redcarpet (>= 3.2.3)
slim
- therubyracer
travis
tzinfo-data
wdm (~> 0.1.0)
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/acd0e2f0/docs/manual/source/community/contribute-code.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/community/contribute-code.html.md b/docs/manual/source/community/contribute-code.html.md
index a9e8767..b424646 100644
--- a/docs/manual/source/community/contribute-code.html.md
+++ b/docs/manual/source/community/contribute-code.html.md
@@ -20,8 +20,7 @@ show features that are highly sought after by the community.
## How to Report an Issue
-If you wish to report an issue you found, you can do so on [GitHub Issues](https://github.com/PredictionIO/PredictionIO/issues)
-or [PredictionIO JIRA](https://predictionio.atlassian.net).
+If you wish to report an issue you found, you can do so on [GitHub Issues](https://github.com/PredictionIO/PredictionIO/issues).
## How to Help Resolve Existing Issues
@@ -36,7 +35,7 @@ bug fixes will follow a different path.
3. Verify the patch and issue a pull request to the main repository.
4. Once merged to the main repository, critical bug fixes will be merged to the
"master" branch and new binary will be built and distributed.
-
+
## How to Add / Propose a New Feature
@@ -45,12 +44,12 @@ bug fixes will follow a different path.
(https://groups.google.com/forum/#!forum/predictionio-dev) or email us directly at support@prediction.io.
2. Discuss with the community and the core development team on what needs to be
done, and lay down concrete plans on deliverables.
-3. Once solid plans are made, start creating tickets in the [issue tracker]
- (https://predictionio.atlassian.net/secure/RapidBoard.jspa?rapidView=1).
+3. Once solid plans are made, start creating tickets on [GitHub
+ Issues](https://github.com/PredictionIO/PredictionIO/issues).
4. Work side by side with other developers using PredictionIO Development Google
Group as primary mode of communication. You never know if someone else has a
better idea. ;)
-
+
## How to Issue a Pull Request
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/acd0e2f0/docs/manual/source/partials/_action_call.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_action_call.html.slim b/docs/manual/source/partials/_action_call.html.slim
index 72ea27b..7554540 100644
--- a/docs/manual/source/partials/_action_call.html.slim
+++ b/docs/manual/source/partials/_action_call.html.slim
@@ -24,4 +24,4 @@
a> href="//github.com/PredictionIO/PredictionIO" target="blank" GitHub
| and report issues on the
= succeed "." do
- a href="//predictionio.atlassian.net/secure/Dashboard.jspa" target="blank" Bug Tracker
\ No newline at end of file
+ a href="//github.com/PredictionIO/PredictionIO/issues" target="blank" Bug Tracker
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/acd0e2f0/docs/manual/source/partials/_footer.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_footer.html.slim b/docs/manual/source/partials/_footer.html.slim
index 3d9a528..e40f2f4 100644
--- a/docs/manual/source/partials/_footer.html.slim
+++ b/docs/manual/source/partials/_footer.html.slim
@@ -23,7 +23,7 @@ footer
ul
li: a href="//docs.prediction.io/community/contribute-code/" target="blank" Contribute
li: a href="//github.com/PredictionIO/PredictionIO" target="blank" Source Code
- li: a href="//predictionio.atlassian.net/secure/Dashboard.jspa" target="blank" Bug Tracker
+ li: a href="//github.com/PredictionIO/PredictionIO/issues" target="blank" Bug Tracker
li: a href="//groups.google.com/forum/#!forum/predictionio-dev" target="blank" Contributors’ Forum
li: a href="//prediction.io/cla" Contributor Agreement
li: a href="//predictionio.uservoice.com/forums/219398-general/filters/top" Request Features
[11/15] incubator-predictionio git commit: [PIO-6] Clean up for ASF
Posted by do...@apache.org.
[PIO-6] Clean up for ASF
Cleaned up links
Cleaned up unused JavaScript
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/0cebfb7b
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/0cebfb7b
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/0cebfb7b
Branch: refs/heads/develop
Commit: 0cebfb7b64cc285f14c2b1408b87b49dc4c55767
Parents: e0a73d2
Author: Donald Szeto <do...@apache.org>
Authored: Fri Jul 22 14:02:04 2016 -0700
Committer: Donald Szeto <do...@apache.org>
Committed: Fri Jul 22 14:02:04 2016 -0700
----------------------------------------------------------------------
docs/manual/source/layouts/layout.html.slim | 2 -
docs/manual/source/partials/_adroll.html.slim | 18 --------
docs/manual/source/partials/_facebook.html.slim | 11 -----
docs/manual/source/partials/_footer.html.slim | 43 +++++---------------
docs/manual/source/partials/_func.html.slim | 4 --
.../source/partials/_google_analytics.html.slim | 10 -----
docs/manual/source/partials/_header.html.slim | 4 +-
.../manual/source/partials/_quantcast.html.slim | 18 --------
docs/manual/source/partials/_rcx.html.slim | 7 ----
.../source/partials/_subscribe_form.html.slim | 6 ---
.../source/partials/head/_javascripts.html.slim | 11 ++---
11 files changed, 15 insertions(+), 119 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0cebfb7b/docs/manual/source/layouts/layout.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/layouts/layout.html.slim b/docs/manual/source/layouts/layout.html.slim
index 982eb43..d83b447 100644
--- a/docs/manual/source/layouts/layout.html.slim
+++ b/docs/manual/source/layouts/layout.html.slim
@@ -26,6 +26,4 @@ html
= yield
= partial 'footer'
= partial 'swiftype'
- = partial 'quantcast'
- = partial 'adroll'
= javascript_include_tag 'application'
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0cebfb7b/docs/manual/source/partials/_adroll.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_adroll.html.slim b/docs/manual/source/partials/_adroll.html.slim
deleted file mode 100644
index 46a5fb6..0000000
--- a/docs/manual/source/partials/_adroll.html.slim
+++ /dev/null
@@ -1,18 +0,0 @@
-javascript:
- adroll_adv_id = "CPSSMJFFZ5DDHITC2STA54";
- adroll_pix_id = "UWX4N2WIMJADVHJGOFTM44";
- (function () {
- var _onload = function(){
- if (document.readyState && !/loaded|complete/.test(document.readyState)){setTimeout(_onload, 10);return}
- if (!window.__adroll_loaded){__adroll_loaded=true;setTimeout(_onload, 50);return}
- var scr = document.createElement("script");
- var host = (("https:" == document.location.protocol) ? "https://s.adroll.com" : "http://a.adroll.com");
- scr.setAttribute('async', 'true');
- scr.type = "text/javascript";
- scr.src = host + "/j/roundtrip.js";
- ((document.getElementsByTagName('head') || [null])[0] ||
- document.getElementsByTagName('script')[0].parentNode).appendChild(scr);
- };
- if (window.addEventListener) {window.addEventListener('load', _onload, false);}
- else {window.attachEvent('onload', _onload)}
- }());
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0cebfb7b/docs/manual/source/partials/_facebook.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_facebook.html.slim b/docs/manual/source/partials/_facebook.html.slim
deleted file mode 100644
index 6816ed9..0000000
--- a/docs/manual/source/partials/_facebook.html.slim
+++ /dev/null
@@ -1,11 +0,0 @@
-- if build?
- javascript:
- !function(f,b,e,v,n,t,s){if(f.fbq)return;n=f.fbq=function(){n.callMethod?
- n.callMethod.apply(n,arguments):n.queue.push(arguments)};if(!f._fbq)f._fbq=n;
- n.push=n;n.loaded=!0;n.version='2.0';n.queue=[];t=b.createElement(e);t.async=!0;
- t.src=v;s=b.getElementsByTagName(e)[0];s.parentNode.insertBefore(t,s)}(window,
- document,'script','//connect.facebook.net/en_US/fbevents.js');
-
- fbq('init', '1073028432707778');
- fbq('track', "PageView");
-
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0cebfb7b/docs/manual/source/partials/_footer.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_footer.html.slim b/docs/manual/source/partials/_footer.html.slim
index 3d9a528..e9f5c7d 100644
--- a/docs/manual/source/partials/_footer.html.slim
+++ b/docs/manual/source/partials/_footer.html.slim
@@ -4,46 +4,23 @@ footer
.container
.seperator
.row
- .col-md-4.col-md-push-8.col-xs-12
- = partial 'subscribe_form'
-
- .col-md-2.col-md-pull-4.col-xs-6.footer-link-column
+ .col-md-6.col-xs-6.footer-link-column
.footer-link-column-row
h4 Community
ul
li: a href="//docs.prediction.io/install/" target="blank" Download
li: a href="//docs.prediction.io/" target="blank" Docs
- li: a href="//github.com/PredictionIO/PredictionIO" target="blank" GitHub
- li: a href="//groups.google.com/forum/#!forum/predictionio-user" target="blank" Support Forum
+ li: a href="//github.com/apache/incubator-predictionio" target="blank" GitHub
+ li: a href="mailto:user-subscribe@predictionio.incubator.apache.org" target="blank" Subscribe to User Mailing List
li: a href="//stackoverflow.com/questions/tagged/predictionio" target="blank" Stackoverflow
- li: a href="mailto:support@prediction.io" target="blank" Contact Us
- .col-md-2.col-md-pull-4.col-xs-6.footer-link-column
+ .col-md-6.col-xs-6.footer-link-column
.footer-link-column-row
h4 Contribute
ul
- li: a href="//docs.prediction.io/community/contribute-code/" target="blank" Contribute
- li: a href="//github.com/PredictionIO/PredictionIO" target="blank" Source Code
- li: a href="//predictionio.atlassian.net/secure/Dashboard.jspa" target="blank" Bug Tracker
- li: a href="//groups.google.com/forum/#!forum/predictionio-dev" target="blank" Contributors’ Forum
- li: a href="//prediction.io/cla" Contributor Agreement
- li: a href="//predictionio.uservoice.com/forums/219398-general/filters/top" Request Features
- .col-md-2.col-md-pull-4.col-xs-6.footer-link-column
- .footer-link-column-row
- h4 Enterprise
- ul
- li: a href="//docs.prediction.io/support/" target="blank" Support
- li: a href="//prediction.io/enterprise" Enterprise
- li: a href="//prediction.io/products/predictionio-enterprise" Services
- .footer-link-column-row
- h4 Connect
- ul
- li: a href="//blog.prediction.io/" target="blank" Blog
- li: a href="//predictionio.theresumator.com/" target="blank" Careers
- .col-md-2.col-md-pull-4.col-xs-6.footer-link-column
- .footer-link-column-row
- h4 Partnership
- ul
- li: a href="//prediction.io/partners/program" Partner Program
+ li: a href="//predictionio.incubator.apache.org/community/contribute-code/" target="blank" Contribute
+ li: a href="//github.com/apache/incubator-predictionio" target="blank" Source Code
+ li: a href="//issues.apache.org/jira/browse/PIO" target="blank" Bug Tracker
+ li: a href="mailto:dev-subscribe@predictionio.incubator.apache.org" target="blank" Subscribe to Development Mailing List
#footer-bottom
.container
@@ -52,8 +29,8 @@ footer
#footer-logo-wrapper
= image_tag 'logos/logo-white.png', alt: 'PredictionIO'
#social-icons-wrapper
- a.github-button> href="https://github.com/PredictionIO/PredictionIO" data-style="mega" data-count-href="/PredictionIO/PredictionIO/stargazers" data-count-api="/repos/PredictionIO/PredictionIO#stargazers_count" data-count-aria-label="# stargazers on GitHub" aria-label="Star PredictionIO/PredictionIO on GitHub" Star
- a.github-button> href="https://github.com/PredictionIO/PredictionIO/fork" data-icon="octicon-git-branch" data-style="mega" data-count-href="/PredictionIO/PredictionIO/network" data-count-api="/repos/PredictionIO/PredictionIO#forks_count" data-count-aria-label="# forks on GitHub" aria-label="Fork PredictionIO/PredictionIO on GitHub" Fork
+ a.github-button> href="https://github.com/apache/incubator-predictionio" data-style="mega" data-count-href="/apache/incubator-predictionio/stargazers" data-count-api="/repos/apache/incubator-predictionio#stargazers_count" data-count-aria-label="# stargazers on GitHub" aria-label="Star apache/incubator-predictionio on GitHub" Star
+ a.github-button> href="https://github.com/apache/incubator-predictionio/fork" data-icon="octicon-git-branch" data-style="mega" data-count-href="/apache/incubator-predictionio/network" data-count-api="/repos/apache/incubator-predictionio#forks_count" data-count-aria-label="# forks on GitHub" aria-label="Fork apache/incubator-predictionio on GitHub" Fork
script#github-bjs(async defer src="https://buttons.github.io/buttons.js")
a> href="//www.facebook.com/predictionio" target="blank"
= image_tag 'icons/twitter.png', alt: 'PredictionIO on Twitter'
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0cebfb7b/docs/manual/source/partials/_func.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_func.html.slim b/docs/manual/source/partials/_func.html.slim
deleted file mode 100644
index d2169d3..0000000
--- a/docs/manual/source/partials/_func.html.slim
+++ /dev/null
@@ -1,4 +0,0 @@
-- if build?
- javascript:
- function t(e){analytics.identify(e); analytics.track("newsletter signup");
- rcx.track("newsletter signup", { '_email': e });}
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0cebfb7b/docs/manual/source/partials/_google_analytics.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_google_analytics.html.slim b/docs/manual/source/partials/_google_analytics.html.slim
deleted file mode 100644
index 4456bc4..0000000
--- a/docs/manual/source/partials/_google_analytics.html.slim
+++ /dev/null
@@ -1,10 +0,0 @@
-- if build?
- javascript:
- (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
- (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
- m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
- })(window,document,'script','//www.google-analytics.com/analytics.js','ga');
-
- ga('create', 'UA-38306178-1', 'auto');
- ga('require', 'linkid', 'linkid.js');
- ga('send', 'pageview');
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0cebfb7b/docs/manual/source/partials/_header.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_header.html.slim b/docs/manual/source/partials/_header.html.slim
index 0212098..ed78bed 100644
--- a/docs/manual/source/partials/_header.html.slim
+++ b/docs/manual/source/partials/_header.html.slim
@@ -5,7 +5,7 @@ header
#logo-wrapper
span#drawer-toggle
a href="#"
- = link_to 'http://prediction.io/' do
+ = link_to 'http://predictionio.incubator.apache.org/' do
= image_tag 'logos/logo.png', alt: 'PredictionIO', id: 'logo'
#menu-wrapper
#header-nav-options-wrapper
@@ -14,5 +14,5 @@ header
li>: a href="/support" Support
#pill-wrapper
a.pill.left> href="//templates.prediction.io/" TEMPLATES
- a.pill.right href="//github.com/PredictionIO/PredictionIO/" OPEN SOURCE
+ a.pill.right href="//github.com/apache/incubator-predictionio/" OPEN SOURCE
= image_tag 'icons/search-glass.png', class: 'mobile-search-bar-toggler hidden-md hidden-lg'
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0cebfb7b/docs/manual/source/partials/_quantcast.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_quantcast.html.slim b/docs/manual/source/partials/_quantcast.html.slim
deleted file mode 100644
index c30ddfa..0000000
--- a/docs/manual/source/partials/_quantcast.html.slim
+++ /dev/null
@@ -1,18 +0,0 @@
-javascript:
- var _qevents = _qevents || [];
- (function() {
- var elem = document.createElement('script');
- elem.src = (document.location.protocol == "https:" ? "https://secure" : "http://edge") + ".quantserve.com/quant.js";
- elem.async = true;
- elem.type = "text/javascript";
- var scpt = document.getElementsByTagName('script')[0];
- scpt.parentNode.insertBefore(elem, scpt);
- })();
- _qevents.push({
- qacct:"p-stVMxuw8H5EPX"
- });
-
-noscript
- <div style="display:none;">
- <img src="//pixel.quantserve.com/pixel/p-stVMxuw8H5EPX.gif" border="0" height="1" width="1" alt="Quantcast"/>
- </div>
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0cebfb7b/docs/manual/source/partials/_rcx.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_rcx.html.slim b/docs/manual/source/partials/_rcx.html.slim
deleted file mode 100644
index 30531dc..0000000
--- a/docs/manual/source/partials/_rcx.html.slim
+++ /dev/null
@@ -1,7 +0,0 @@
-- if build?
- javascript:
- RCX_CUSTOM_LIB="https://cdn.recontext.com/staging/rcx.min.js";
- (function(b,d,a){b.RCX_OBJECT=a;a=b[a]||[];if(!a.snipV&&!a.libV){b.rcx=a;a.snipV="0.2.0";var g=function(a,b,c,d){a[b]=a[b]||function(){c.push([d].concat(Array.prototype.slice.call(arguments)))}};b="init page track identify link setUserProperty unsetUserProperty".split(" ");for(var f=0;f<b.length;f++){var e,c;e=b[f];c=e.split(".");2==c.length?(a[c[0]]=a[c[0]]||[],g(a[c[0]],c[1],a,e)):g(a,e,a,e)}a=d.createElement("script");a.type="text/javascript";a.async=!0;a.src="undefined"!==typeof RCX_CUSTOM_LIB?
- RCX_CUSTOM_LIB:"https://cdn.recontext.com/rcx.min.js";d=d.getElementsByTagName("script")[0];d.parentNode.insertBefore(a,d)}})(window,document,"rcx");
- rcx.init("kTxFcI3IWdXYfRsh6uuYuej4qYl8m8LVMePM2hdIkM9YjHqkAFC6mqdqO9fpp8p9");
- rcx.page();
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0cebfb7b/docs/manual/source/partials/_subscribe_form.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_subscribe_form.html.slim b/docs/manual/source/partials/_subscribe_form.html.slim
deleted file mode 100644
index b2dd6b2..0000000
--- a/docs/manual/source/partials/_subscribe_form.html.slim
+++ /dev/null
@@ -1,6 +0,0 @@
-.subscription-form-wrapper
- h4 Subscribe to our Newsletter
- form.ajax-form#subscribe-form method="POST" action="https://script.google.com/macros/s/AKfycbwhzeKCQJjQ52eVAqNT_vcklH07OITUO7wzOMDXvK6EGAWgaZgF/exec"
- input.required.underlined-input type="email" placeholder="Your email address" name="subscription_email" id="subscription_email"
- input.pill-button value="SUBSCRIBE" type="submit" data-state-normal="SUBSCRIBE" data-state-sucess="SUBSCRIBED!" data-state-loading="SENDING..." onclick="t($('#subscription_email').val());"
- p.result
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0cebfb7b/docs/manual/source/partials/head/_javascripts.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/head/_javascripts.html.slim b/docs/manual/source/partials/head/_javascripts.html.slim
index 1ff0ed8..14685fd 100644
--- a/docs/manual/source/partials/head/_javascripts.html.slim
+++ b/docs/manual/source/partials/head/_javascripts.html.slim
@@ -1,13 +1,8 @@
/[if lt IE 9]
- = javascript_include_tag '//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.2/html5shiv.min.js'
+= javascript_include_tag '//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.2/html5shiv.min.js'
= javascript_include_tag '//cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML'
-= partial 'google_analytics'
-= partial 'segment'
-= partial 'rcx'
-= partial 'func'
-= partial 'facebook'
/azo sans font
-= javascript_include_tag '//use.typekit.net/mut4mjx.js'
+= javascript_include_tag '//use.typekit.net/pqo0itb.js'
javascript:
- try{Typekit.load();}catch(e){}
+ try{Typekit.load({ async: true });}catch(e){}
[08/15] incubator-predictionio git commit: [PIO-6] Initial doc site
Posted by do...@apache.org.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/demo/textclassification.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/demo/textclassification.html.md b/docs/manual/source/demo/textclassification.html.md
deleted file mode 100644
index 7b819e3..0000000
--- a/docs/manual/source/demo/textclassification.html.md
+++ /dev/null
@@ -1,663 +0,0 @@
----
-title: Text Classification Engine Tutorial
----
-
-(Updated for Text Classification Template version 3.1)
-
-## Introduction
-
-In the real world, there are many applications that collect text as data. For example, spam detectors take email and header content to automatically determine what is or is not spam; applications can gague the general sentiment in a geographical area by analyzing Twitter data; and news articles can be automatically categorized based solely on the text content.There are a wide array of machine learning models you can use to create, or train, a predictive model to assign an incoming article, or query, to an existing category. Before you can use these techniques you must first transform the text data (in this case the set of news articles) into numeric vectors, or feature vectors, that can be used to train your model.
-
-The purpose of this tutorial is to illustrate how you can go about doing this using PredictionIO's platform. The advantages of using this platform include: a dynamic engine that responds to queries in real-time; [separation of concerns](http://en.wikipedia.org/wiki/Separation_of_concerns), which offers code re-use and maintainability, and distributed computing capabilities for scalability and efficiency. Moreover, it is easy to incorporate non-trivial data modeling tasks into the DASE architecture allowing Data Scientists to focus on tasks related to modeling. This tutorial will exemplify some of these ideas by guiding you through PredictionIO's [text classification template(http://templates.prediction.io/PredictionIO/template-scala-parallel-textclassification/).
-
-
-
-## Prerequisites
-
-Before getting started, please make sure that you have the latest version of PredictionIO [installed](https://docs.prediction.io/install/). We emphasize here that this is an engine template written in **Scala** and can be more generally thought of as an SBT project containing all the necessary components.
-
-You should also download the engine template named Text Classification Engine that accompanies this tutorial by cloning the template repository:
-
-```
-pio template get PredictionIO/template-scala-parallel-textclassification < Your new engine directory >
-```
-
-
-## Engine Overview
-
-The engine follows the DASE architecture which we briefly review here. As a user, you are tasked with collecting data for your web or application, and importing it into PredictionIO's Event Server. Once the data is in the server, it can be read and processed by the engine via the Data Source and Preparation components, respectively. The Algorithm component uses the processed, or prepared, data to train a set of predictive models. Once you have trained these models, you are ready to deploy your engine and respond to real-time queries via the Serving component which combines the results from different fitted models. The Evaluation component is used to compute an appropriate metric to test the performance of a fitted model, as well as aid in the tuning of model hyper parameters.
-
-This engine template is meant to handle text classification which means you will be working with text data. This means that a query, or newly observed documents, will be of the form:
-
-`{text : String}`.
-
-In the running example, a query would be an incoming news article. Once the engine is deployed it can process the query, and then return a Predicted Result of the form
-
-`{category : String, confidence : Double}`.
-
-Here category is the model's class assignment for this new text document (i.e. the best guess for this article's categorization), and confidence, a value between 0 and 1 representing your confidence in the category prediction (0 meaning you have no confidence in the prediction). The Actual Result is of the form
-
-`{category : String}`.
-
-This is used in the evaluation stage when estimating the performance of your predictive model (how well does the model predict categories). Please refer to the [following tutorial](https://docs.prediction.io/customize/) for a more detailed explanation of how your engine will interact with your web application, as well as an in depth-overview of DASE.
-
-
-## Quick Start
-
-This is a quick start guide in case you want to start using the engine right away. Sample email data for spam classification will be used. For more detailed information, read the subsequent sections.
-
-
-### 1. Create a new application.
-
-After the application is created, you will be given an access key and application ID for the application.
-
-```
-$ pio app new MyTextApp
-```
-
-### 2. Import the tutorial data.
-
-There are three different data sets available, each giving a different use case for this engine. Please refer to the **Data Source: Reading Event Data** section to see how to appropriate modify the `DataSource` class for use with each respective data set. The default data set is an e-mail spam data set.
-
-These data sets have already been processed and are ready for [batch import](/datacollection/batchimport/). Replace `***` with your actual application ID.
-
-```
-$ pio import --appid *** --input data/stopwords.json
-
-$ pio import --appid *** --input data/emails.json
-```
-
-### 3. Set the engine parameters in the file `engine.json`.
-
-The default settings are shown below. By default, it uses the algorithm name "lr" which is logstic regression. Please see later section for more detailed explanation of engine.json setting.
-
-Make sure the "appName" is same as the app you created in step1.
-
-
-```
-{
- "id": "default",
- "description": "Default settings",
- "engineFactory": "org.template.textclassification.TextClassificationEngine",
- "datasource": {
- "params": {
- "appName": "MyTextApp"
- }
- },
- "preparator": {
- "params": {
- "nGram": 1,
- "numFeatures": 500,
- "SPPMI": false
- }
- },
- "algorithms": [
- {
- "name": "lr",
- "params": {
- "regParam": 0.00000005
- }
- }
- ]
-}
-
-```
-
-### 4. Build your engine.
-
-```
-$ pio build --verbose
-```
-
-This command should take few minutes for the first time; all subsequent builds should be less than a minute. You can also run it without `--verbose` if you don't want to see all the log messages.
-
-Upon successful build, you should see a console message similar to the following
-
-```
-[INFO] [RegisterEngine$] Registering engine 6wxDy2hxLbvaMJra927ahFdQHDIVXeQz 266bae678c570dee58154b2338cef7aa1646e0d3
-[INFO] [Console$] Your engine is ready for training.
-```
-
-### 5.a. Train your model and deploy.
-
-```
-$ pio train
-```
-
-When your engine is trained successfully, you should see a console message similar to the following.
-
-```
-[INFO] [CoreWorkflow$] Training completed successfully.
-```
-
-Now your engine is ready to deploy. Run:
-
-```
-$ pio deploy
-```
-
-When the engine is deployed successfully and running, you should see a console message similar to the following:
-
-```
-[INFO] [HttpListener] Bound to /0.0.0.0:8000
-[INFO] [MasterActor] Engine is deployed and running. Engine API is live at http://0.0.0.0:8000.
-```
-
-Now you can send query to the engine. Open another terminal and send the following http request to the deployed engine:
-
-```
-$ curl -H "Content-Type: application/json" -d '{ "text":"I like speed and fast motorcycles." }' http://localhost:8000/queries.json
-```
-
-you should see following outputs returned by the engine:
-
-```
-{"category":"not spam","confidence":0.852619510921587}
-```
-
-Try another query:
-
-```
-$ curl -H "Content-Type: application/json" -d '{ "text":"Earn extra cash!" }' http://localhost:8000/queries.json
-```
-
-you should see following outputs returned by the engine:
-
-```
-{"category":"spam","confidence":0.5268770133242983}
-```
-
-
-### 5.b.Evaluate your training model and tune parameters.
-
-```
-$ pio eval org.template.textclassification.AccuracyEvaluation org.template.textclassification.EngineParamsList
-```
-
-**Note:** Training and evaluation stages are generally different stages of engine development. Evaluation is there to help you choose the best [algorithm parameters](/evaluation/paramtuning/) to use for training an engine that is to be deployed as a web service.
-
-Depending on your needs, in steps (5.x.) above, you can configure your Spark settings by typing a command of the form:
-
-```
-$ pio command command_parameters -- --master url --driver-memory {0}G --executor-memory {1}G --conf spark.akka.framesize={2} --total_executor_cores {3}
-```
-
-Only the latter commands are listed as these are some of the more commonly modified values. See the [Spark documentation](https://spark.apache.org/docs/latest/spark-standalone.html) and the [PredictionIO FAQ's](https://docs.prediction.io/resources/faq/) for more information.
-
-**Note:** We recommend you set your driver memory to `1G` or `2G` as the data size when dealing with text can be very large.
-
-
-# Detailed Explanation of DASE
-
-## Importing Data
-
-In the quick start, email spam classification is used. This template can easily be modified for other types text classification.
-
-If you want to import different sets of data, follow the Quick Start instructions to import data from different files. Make sure that the Data Source is modified accordingly to match the `event`, `entityType`, and `properties` fields set for the specific dataset. The following section explains this in more detail.
-
-## Data Source: Reading Event Data
-
-Now that the data has been imported into PredictionIO's Event Server, it needs to be read from storage to be used by the engine. This is precisely what the DataSource engine component is for, which is implemented in the template script `DataSource.scala`. The class `Observation` serves as a wrapper for storing the information about a news document needed to train a model. The attribute label refers to the label of the category a document belongs to, and text, stores the actual document content as a string. The class TrainingData is used to store an RDD of Observation objects along with the set of stop words.
-
-The class `DataSourceParams` is used to specify the parameters needed to read and prepare the data for processing. This class is initialized with two parameters `appName` and `evalK`. The first parameter specifies your application name (i.e. MyTextApp), which is needed so that the DataSource component knows where to pull the event data from. The second parameter is used for model evaluation and specifies the number of folds to use in [cross-validation](http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29) when estimating a model performance metric.
-
-The final and most important ingredient is the DataSource class. This is initialized with its corresponding parameter class, and extends `PDataSource`. This **must** implement the method `readTraining` which returns an instance of type TrainingData. This method completely relies on the defined private methods readEventData and readStopWords. Both of these functions read data observations as Event instances, create an RDD containing these events and finally transforms the RDD of events into an object of the appropriate type as seen below:
-
-```scala
-...
-private def readEventData(sc: SparkContext) : RDD[Observation] = {
- //Get RDD of Events.
- PEventStore.find(
- appName = dsp.appName,
- entityType = Some("content"), // specify data entity type
- eventNames = Some(List("e-mail")) // specify data event name
-
- // Convert collected RDD of events to and RDD of Observation
- // objects.
- )(sc).map(e => {
- val label : String = e.properties.get[String]("label")
- Observation(
- if (label == "spam") 1.0 else 0.0,
- e.properties.get[String]("text"),
- label
- )
- }).cache
- }
-
- // Helper function used to store stop words from
- // event server.
- private def readStopWords(sc : SparkContext) : Set[String] = {
- PEventStore.find(
- appName = dsp.appName,
- entityType = Some("resource"),
- eventNames = Some(List("stopwords"))
-
- //Convert collected RDD of strings to a string set.
- )(sc)
- .map(e => e.properties.get[String]("word"))
- .collect
- .toSet
- }
-...
-```
-
-Note that `readEventData` and `readStopWords` use different entity types and event names, but use the same application name. This is because the sample import script imports two different data types, documents and stop words. These field distinctions are required for distinguishing between the two. The method `readEval` is used to prepare the different cross-validation folds needed for evaluating your model and tuning hyper parameters.
-
-Now, the default dataset used for training is contained in the file `data/emails.json` and contains a set of e-mail spam data. If we want to switch over to one of the other data sets we must make sure that the `eventNames` and `entityType` fields are changed accordingly.
-
-In the data/ directory, you will find different sets of data files for different types of text classifcaiton application. The following show one observation from each of the provided data files:
-
-- `emails.json`:
-
-```
-{"eventTime": "2015-06-08T16:45:00.590+0000", "entityId": 1, "properties": {"text": "Subject: dobmeos with hgh my energy level has gone up ! stukm\nintroducing\ndoctor - formulated\nhgh\nhuman growth hormone - also called hgh\nis referred to in medical science as the master hormone . it is very plentiful\nwhen we are young , but near the age of twenty - one our bodies begin to produce\nless of it . by the time we are forty nearly everyone is deficient in hgh ,\nand at eighty our production has normally diminished at least 90 - 95 % .\nadvantages of hgh :\n- increased muscle strength\n- loss in body fat\n- increased bone density\n- lower blood pressure\n- quickens wound healing\n- reduces cellulite\n- improved vision\n- wrinkle disappearance\n- increased skin thickness texture\n- increased energy levels\n- improved sleep and emotional stability\n- improved memory and mental alertness\n- increased sexual potency\n- resistance to common illness\n- strengthened heart muscle\n- controlle
d cholesterol\n- controlled mood swings\n- new hair growth and color restore\nread\nmore at this website\nunsubscribe\n", "label": "spam"}, "event": "e-mail", "entityType": "content"}
-
-```
-
-- `20newsgroups.json`:
-
-```
-{"entityType": "source", "eventTime": "2015-06-08T18:01:55.003+0000", "event": "documents", "entityId": 1, "properties": {"category": "sci.crypt", "text": "From: rj@ri.cadre.com (Rob deFriesse)\nSubject: Can DES code be shipped to Canada?\nArticle-I.D.: fripp.1993Apr22.125402.27561\nReply-To: rj@ri.cadre.com\nOrganization: Cadre Technologies Inc.\nLines: 13\nNntp-Posting-Host: 192.9.200.19\n\nSomeone in Canada asked me to send him some public domain DES file\nencryption code I have. Is it legal for me to send it?\n\nThanx.\n--\nEschew Obfuscation\n\nRob deFriesse Mail: rj@ri.cadre.com\nCadre Technologies Inc. Phone: (401) 351-5950\n222 Richmond St. Fax: (401) 351-7380\nProvidence, RI 02903\n\nI don't speak for my employer.\n", "label": 11.0}}
-```
-
-- `sentimentanalysis.json`:
-
-```
-{"eventTime": "2015-06-08T16:58:14.278+0000", "entityId": 23714, "entityType": "source", "properties": {"phrase": "Tosca 's intoxicating ardor", "sentiment": 3}, "event": "phrases"}
-```
-
-Now, note that the `entityType`, `event`, and `properties` fields for the `20newsgroups.json` dataset differ from the default `emails.json` set. Default DataSource implementation is to read from `email.json` data set. If you want to use others such as newsgroups data set, the engine's Data Source component must be modified accordingly. To do this, you need only modify the method `readEventData` as follows:
-
-### Modify DataSource to Read `20newsgroups.json`
-
-```scala
-private def readEventData(sc: SparkContext) : RDD[Observation] = {
- //Get RDD of Events.
- PEventStore.find(
- appName = dsp.appName,
- entityType = Some("source"), // specify data entity type
- eventNames = Some(List("documents")) // specify data event name
-
- // Convert collected RDD of events to and RDD of Observation
- // objects.
- )(sc).map(e => {
-
- Observation(
- e.properties.get[Double]("label"),
- e.properties.get[String]("text"),
- e.properties.get[String]("category")
- )
- }).cache
- }
-```
-
-### Modify DataSource to Read `sentimentanalysis.json`
-
-```scala
-private def readEventData(sc: SparkContext) : RDD[Observation] = {
- //Get RDD of Events.
- PEventStore.find(
- appName = dsp.appName,
- entityType = Some("source"), // specify data entity type
- eventNames = Some(List("phrases")) // specify data event name
-
- // Convert collected RDD of events to and RDD of Observation
- // objects.
- )(sc).map(e => {
- val label = e.properties.get[Double]("sentiment")
-
- Observation(
- label,
- e.properties.get[String]("phrase"),
- label.toString
- )
- }).cache
- }
-```
-
-Note that `event` field in the json file refers to the `eventNames` field in the `readEventData` method. When using this engine with a custom data set, you need to make sure that the respective json fields match with the corresponding fields in the DataSource component. We have included a data sanity check with this engine component that lets you know if your data is actually being read in. If you have 0 observations being read, you should see the following output when your training process performs the Training Data sanity check:
-
-`Data set is empty, make sure event fields match imported data.`
-
-This data sanity check is a PredictionIO feature available for your `TrainingData` and `PreparedData` classes. The following code block demonstrates how the sanity check is implemented:
-
-```scala
-class TrainingData(
- val data : RDD[Observation],
- val stopWords : Set[String]
-) extends Serializable with SanityCheck {
-
- // Sanity check to make sure your data is being fed in correctly.
-
- def sanityCheck {
- try {
- val obs : Array[Double] = data.takeSample(false, 5).map(_.label)
-
- println()
- (0 until 5).foreach(
- k => println("Observation " + (k + 1) +" label: " + obs(k))
- )
- println()
- } catch {
- case (e : ArrayIndexOutOfBoundsException) => {
- println()
- println("Data set is empty, make sure event fields match imported data.")
- println()
- }
- }
-
- }
-
-}
-```
-
-## Preparator : Data Processing With DASE
-
-Recall that the Preparator stage is used for doing any prior data processing needed to fit a predictive model. In line with the separation of concerns, the Data Model implementation, PreparedData, is built to do the heavy lifting needed for this data processing. The Preparator must simply implement the prepare method which outputs an object of type PreparedData. This requires you to specify two n-gram window components, and two inverse i.d.f. window components (these terms will be defined in the following section). Therefore a custom class of parameters for the Preparator component, PreparatorParams, must be incorporated. The code defining the full Preparator component is given below:
-
-```scala
-// 1. Initialize Preparator parameters. Recall that for our data
-// representation we are only required to input the n-gram window
-// components.
-
-case class PreparatorParams(
- nGram: Int,
- numFeatures: Int = 5000,
- SPPMI: Boolean
-) extends Params
-
-
-
-// 2. Initialize your Preparator class.
-
-class Preparator(pp: PreparatorParams) extends PPreparator[TrainingData, PreparedData] {
-
- // Prepare your training data.
- def prepare(sc : SparkContext, td: TrainingData): PreparedData = {
- new PreparedData(td, pp.nGram)
- }
-}
-
-```
-
-The simplicity of this stage implementation truly exemplifies one of the benefits of using the PredictionIO platform. For developers, it is easy to incorporate different classes and tools into the DASE framework so that the process of creating an engine is greatly simplified which helps increase your productivity. For data scientists, the load of implementation details you need to worry about is minimized so that you can focus on what is important to you: training a good predictive model.
-
-The following subsection explains the class PreparedData, which actually handles the transformation of text documents to feature vectors.
-
-### PreparedData: Text Vectorization and Feature Reduction
-
-The Scala class PreparedData which takes the parameters td, nGram, where td is an object of class TrainingData. The other parameter specifies the n-gram parametrization which will be described shortly.
-
-It will be easier to explain the preparation process with an example, so consider the document \\(d\\):
-
-`"Hello, my name is Marco."`
-
-The first thing you need to do is break up \\(d\\) into an array of "allowed tokens." You can think of a token as a terminating sequence of characters that exist in a document (think of a word in a sentence). For example, the list of tokens that appear in \\(d\\) is:
-
-```scala
-val A = Array("Hello", ",", "my", "name", "is", "Marco", ".")
-```
-
-Recall that a set of stop words was also imported in the previous sections. This set of stop words contains all the words (or tokens) that you do not want to include once documents are tokenized. Those tokens that appear in \\(d\\) and are not contained in the set of stop words will be called allowed tokens. So, if the set of stop words is `{"my", "is"}`, then the list of allowed tokens appearing in \\(d\\) is:
-
-```scala
-val A = Array("Hello", ",", "name", "Marco", ".")
-```
-
-The next step in the data representation is to take the array of allowed tokens and extract a set of n-grams and a corresponding value indicating the number of times a given n-gram appears. The set of n-grams for n equal to 1 and 2 in the running example is the set of elements of the form `[A(`\\(i\\)`)]` and `[A(`\\(j\\)`), A(`\\(j + 1\\)`)]`, respectively. In the general case, the set of n-grams extracted from an array of allowed tokens `A` will be of the form `[A(`\\(i\\)`), A(`\\(i + 1\\)`), ..., A(`\\(i + n - 1\\)`)]` for \\(i = 0, 1, 2, ...,\\) `A.size` \\(- n\\). You can set `n` with the `nGram` parameter option in your `PreparatorParams`.
-
-We use MLLib's `HashingTF` class to implement the conversion from text to term frequency vectors, and can be seen in the following method of the class `PreparedData`:
-
-```scala
-...
- // 1. Hashing function: Text -> term frequency vector.
-
- private val hasher = new HashingTF()
-
- private def hashTF (text : String) : Vector = {
- val newList : Array[String] = text.split(" ")
- .sliding(nGram)
- .map(_.mkString)
- .toArray
-
- hasher.transform(newList)
- }
-
- // 2. Term frequency vector -> t.f.-i.d.f. vector.
-
- val idf : IDFModel = new IDF().fit(td.data.map(e => hashTF(e.text)))
-...
-```
-
-The next step is, once all of the observations have been hashed, to collect all n-grams and compute their corresponding [t.f.-i.d.f. value](http://en.wikipedia.org/wiki/Tf%E2%80%93idf). The t.f.-i.d.f. transformation is defined for n-grams, and helps to give less weight to those n-grams that appear with high frequency across all documents, and vice versa. This helps to leverage the predictive power of those words that appear rarely, but can make a big difference in the categorization of a given text document. This is implemented using MLLib's `IDF` and `IDFModel` classes:
-
-```scala
-// 2. Term frequency vector -> t.f.-i.d.f. vector.
-
- val idf : IDFModel = new IDF().fit(td.data.map(e => hashTF(e.text)))
-```
-
-
-The last two functions that will be mentioned are the methods you will actually use for the data transformation. The method transform takes a document and outputs a sparse vector (MLLib implementation). The transformData method simply transforms the TrainingData input (a corpus of documents) into a set of vectors that can now be used for training. The method transform is used both to transform the training data and future queries.
-
-```scala
-...
-// 3. Document Transformer: text => tf-idf vector.
-
- def transform(text : String): Vector = {
- // Map(n-gram -> document tf)
- idf.transform(hashTF(text))
- }
-
-
- // 4. Data Transformer: RDD[documents] => RDD[LabeledPoints]
-
- val transformedData: RDD[(LabeledPoint)] = {
- td.data.map(e => LabeledPoint(e.label, transform(e.text)))
- }
-```
-
-The last and final object implemented in this class simply creates a Map with keys being class labels and values, the corresponding category.
-
-```scala
- // 5. Finally extract category map, associating label to category.
- val categoryMap = td.data.map(e => (e.label, e.category)).collectAsMap
-```
-
-
-## Algorithm Component
-
-The algorithm components in this engine, `NBAlgorithm` and `LRAlgorithm`, actually follows a very general form. Firstly, a parameter class must again be initialized to feed in the corresponding Algorithm model parameters. For example, NBAlgorithm incorporates NBAlgorithmParams which holds the appropriate additive smoothing parameter lambda for the Naive Bayes model.
-
-
-The main class of interest in this component is the class that extends [P2LAlgorithm](https://docs.prediction.io/api/current/#io.prediction.controller.P2LAlgorithm). This class must implement a method named train which will output your predictive model (as a concrete object, this will be implemented via a Scala class). It must also implement a predict method that transforms a query to an appropriate feature vector, and uses this to predict with the fitted model. The vectorization function is implemented by a PreparedData object, and the categorization (prediction) is handled by an instance of the NBModel implementation. Again, this demonstrates the facility with which different models can be incorporated into PredictionIO's DASE architecture.
-
-The model class itself will be discussed in the following section, however, turn your attention to the TextManipulationEngine object defined in the script `Engine.scala`. You can see here that the engine is initialized by specifying the DataSource, Preparator, and Serving classes, as well as a Map of algorithm names to Algorithm classes. This tells the engine which algorithms to run. In practice, you can have as many statistical learning models as you'd like, you simply have to implement a new algorithm component to do this. However, this general design form will persist, and the main meat of the work should be in the implementation of your model class.
-
-The following subsection will go over our Naive Bayes implementation in NBModel.
-
-
-### Naive Bayes Classification
-
-This Training Model class only uses the Multinomial Naive Bayes [implementation](https://spark.apache.org/docs/latest/mllib-naive-bayes.html) found in the Spark MLLib library. However, recall that the predicted results required in the specifications listed in the overview are of the form:
-
-
-`{category: String, confidence: Double}`.
-
-The confidence value should really be interpreted as the probability that a document belongs to a category given its vectorized form. Note that MLLib's Naive Bayes model has the class members pi (\\(\pi\\)), and theta (\\(\theta\\)). \\(\pi\\) is a vector of log prior class probabilities, which shows your prior beliefs regarding the probability that an arbitrary document belongs in a category. \\(\theta\\) is a C \\(\times\\) D matrix, where C is the number of classes, and D, the number of features, giving the log probabilities that parametrize the Multinomial likelihood model assumed for each class. The multinomial model is easiest to think about as a problem of randomly throwing balls into bins, where the ball lands in each bin with a certain probability. The model treats each n-gram as a bin, and the corresponding t.f.-i.d.f. value as the number of balls thrown into it. The likelihood is the probability of observing a (vectorized) document given that it comes from a particular cl
ass.
-
-Now, letting \\(\mathbf{x}\\) be a vectorized text document, then it can be shown that the vector
-
-$$
-\frac{\exp\left(\pi + \theta\mathbf{x}\right)}{\left|\left|\exp\left(\pi + \theta\mathbf{x}\right)\right|\right|}
-$$
-
-is a vector with C components that represent the posterior class membership probabilities for the document given \\(\mathbf{x}\\). That is, the update belief regarding what category this document belongs to after observing its vectorized form. This is the motivation behind defining the class NBModel which uses Spark MLLib's NaiveBayesModel, but implements a separate prediction method.
-
-The private methods innerProduct and getScores are implemented to do the matrix computation above.
-
-```scala
-...
- // 2. Set up linear algebra framework.
-
- private def innerProduct (x : Array[Double], y : Array[Double]) : Double = {
- x.zip(y).map(e => e._1 * e._2).sum
- }
-
- val normalize = (u: Array[Double]) => {
- val uSum = u.sum
-
- u.map(e => e / uSum)
- }
-
-
-
- // 3. Given a document string, return a vector of corresponding
- // class membership probabilities.
-
- private def getScores(doc: String): Array[Double] = {
- // Helper function used to normalize probability scores.
- // Returns an object of type Array[Double]
-
- // Vectorize query,
- val x: Vector = pd.transform(doc)
-
- normalize(
- nb.pi
- .zip(nb.theta)
- .map(
- e => exp(innerProduct(e._2, x.toArray) + e._1))
- )
- }
-...
-```
-
-
-Once you have a vector of class probabilities, you can classify the text document to the category with highest posterior probability, and, finally, return both the category as well as the probability of belonging to that category (i.e. the confidence in the prediction) given the observed data. This is implemented in the method predict.
-
-```scala
-...
- // 4. Implement predict method for our model using
- // the prediction rule given in tutorial.
-
- def predict(doc : String) : PredictedResult = {
- val x: Array[Double] = getScores(doc)
- val y: (Double, Double) = (nb.labels zip x).maxBy(_._2)
- new PredictedResult(pd.categoryMap.getOrElse(y._1, ""), y._2)
- }
-```
-
-### Logistic Regression Classification
-
-To use the alternative multinomial logistic regression algorithm change your `engine.json` as follows:
-
-```json
- {
- "id": "default",
- "description": "Default settings",
- "engineFactory": "org.template.textclassification.TextClassificationEngine",
- "datasource": {
- "params": {
- "appName": "MyTextApp"
- }
- },
- "preparator": {
- "params": {
- "nGram": 2
- }
- },
- "algorithms": [
- {
- "name": "regParam",
- "params": {
- "regParam": 0.1
- }
- }
- ]
-}
-```
-
-
-## Serving: Delivering the Final Prediction
-
-The serving component is the final stage in the engine, and in a sense, the most important. This is the final stage in which you combine the results obtained from the different models you choose to run. The Serving class extends the [LServing](https://docs.prediction.io/api/current/#io.prediction.controller.LServing) class which must implement a method called serve. This takes a query and an associated sequence of predicted results, which contains the predicted results from the different algorithms that are implemented in your engine, and combines the results to yield a final prediction. It is this final prediction that you will receive after sending a query.
-
-For example, you could choose to slightly modify the implementation to return class probabilities coming from a mixture of model estimates for class probabilities, or any other technique you could conceive for combining your results. The default engine setting has this set to yield the label from the model predicting with greater confidence.
-
-
-
-## Evaluation: Model Assessment and Selection
-
- A predictive model needs to be evaluated to see how it will generalize to future observations. PredictionIO uses cross-validation to perform model performance metric estimates needed to assess your particular choice of model. The script `Evaluation.scala` available with the engine template exemplifies what a usual evaluator setup will look like. First, you must define an appropriate metric. In the engine template, since the topic is text classification, the default metric implemented is category accuracy.
-
- Second you must define an evaluation object (i.e. extends the class [Evaluation](https://docs.prediction.io/api/current/#io.prediction.controller.Evaluation)).
-Here, you must specify the actual engine and metric components that are to be used for the evaluation. In the engine template, the specified engine is the TextManipulationEngine object, and metric, Accuracy. Lastly, you must specify the parameter values that you want to test in the cross validation. You see in the following block of code:
-
-```scala
-object EngineParamsList extends EngineParamsGenerator {
-
- // Set data source and preparator parameters.
- private[this] val baseEP = EngineParams(
- dataSourceParams = DataSourceParams(appName = "marco-MyTextApp", evalK = Some(5)),
- preparatorParams = PreparatorParams(nMin = 1, nMax = 2)
- )
-
- // Set the algorithm params for which we will assess an accuracy score.
- engineParamsList = Seq(
- baseEP.copy(algorithmParamsList = Seq(("nb", NBAlgorithmParams(0.5)))),
- baseEP.copy(algorithmParamsList = Seq(("nb", NBAlgorithmParams(1.5)))),
- baseEP.copy(algorithmParamsList = Seq(("nb", NBAlgorithmParams(5))))
- )
-```
-
-
-## Engine Deployment
-
-Once an engine is ready for deployment it can interact with your web application in real-time. This section will cover how to send and receive queries from your engine, gather more data, and re-training your model with the newly gathered data.
-
-### Sending Queries
-
-Recall that one of the greatest advantages of using the PredictionIO platform is that once your engine is deployed, you can respond to queries in real-time. Recall that our queries are of the form
-
-`{"text" : "..."}`.
-
-To actually send a query you can use our REST API by typing in the following shell command:
-
-```
-curl -H "Content-Type: application/json" -d '{ "text":"I like speed and fast motorcycles." }' http://localhost:8000/queries.json
-```
-
-There are a number of [SDK's](https://github.com/PredictionIO) you can use to send your queries and obtain a response. Recall that our predicted response is of the form
-
-```
-{"category" : "class", "confidence" : 1.0}
-```
-
-which is what you should see upon inputting the latter command for querying.
-
-### Gathering More Data and Retraining Your Model
-
-The importing data section that is included in this tutorial uses a sample data set for illustration purposes, and uses the PredictionIO Python SDK to import the data. However, there are a variety of ways that you can [import](https://docs.prediction.io/datacollection/eventapi/) your collected data (via REST or other SDKs).
-
-
-As you continue to collect your data, it is quite easy to retrain your model once you actually import your data into the Event Server. You simply repeat the steps listed in the Quick Start guide. We re-list them here again:
-
-
-**1.** Build your engine.
-
-```
-$ pio build
-```
-
-**2.a.** Evaluate your training model and tune parameters.
-
-```
-$ pio eval org.template.textclassification.AccuracyEvaluation org.template.textclassification.EngineParamsList
-```
-
-**2.b.** Train your model and deploy.
-
-```
-$ pio train
-$ pio deploy
-```
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/demo/textclassification.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/demo/textclassification.html.md.erb b/docs/manual/source/demo/textclassification.html.md.erb
new file mode 100644
index 0000000..043db5a
--- /dev/null
+++ b/docs/manual/source/demo/textclassification.html.md.erb
@@ -0,0 +1,673 @@
+---
+title: Text Classification Engine Tutorial
+---
+
+(Updated for Text Classification Template version 3.1)
+
+## Introduction
+
+In the real world, there are many applications that collect text as data. For example, spam detectors take email and header content to automatically determine what is or is not spam; applications can gague the general sentiment in a geographical area by analyzing Twitter data; and news articles can be automatically categorized based solely on the text content.There are a wide array of machine learning models you can use to create, or train, a predictive model to assign an incoming article, or query, to an existing category. Before you can use these techniques you must first transform the text data (in this case the set of news articles) into numeric vectors, or feature vectors, that can be used to train your model.
+
+The purpose of this tutorial is to illustrate how you can go about doing this using PredictionIO's platform. The advantages of using this platform include: a dynamic engine that responds to queries in real-time; [separation of concerns](http://en.wikipedia.org/wiki/Separation_of_concerns), which offers code re-use and maintainability, and distributed computing capabilities for scalability and efficiency. Moreover, it is easy to incorporate non-trivial data modeling tasks into the DASE architecture allowing Data Scientists to focus on tasks related to modeling. This tutorial will exemplify some of these ideas by guiding you through PredictionIO's [text classification template(http://templates.prediction.io/PredictionIO/template-scala-parallel-textclassification/).
+
+
+
+## Prerequisites
+
+Before getting started, please make sure that you have the latest version of
+Apache PredictionIO (incubating)
+[installed](http://predictionio.incubator.apache.org/install/). We emphasize
+here that this is an engine template written in **Scala** and can be more
+generally thought of as an SBT project containing all the necessary components.
+
+You should also download the engine template named Text Classification Engine
+that accompanies this tutorial by cloning the template repository:
+
+```
+pio template get PredictionIO/template-scala-parallel-textclassification < Your new engine directory >
+```
+
+
+## Engine Overview
+
+The engine follows the DASE architecture which we briefly review here. As a user, you are tasked with collecting data for your web or application, and importing it into PredictionIO's Event Server. Once the data is in the server, it can be read and processed by the engine via the Data Source and Preparation components, respectively. The Algorithm component uses the processed, or prepared, data to train a set of predictive models. Once you have trained these models, you are ready to deploy your engine and respond to real-time queries via the Serving component which combines the results from different fitted models. The Evaluation component is used to compute an appropriate metric to test the performance of a fitted model, as well as aid in the tuning of model hyper parameters.
+
+This engine template is meant to handle text classification which means you will be working with text data. This means that a query, or newly observed documents, will be of the form:
+
+`{text : String}`.
+
+In the running example, a query would be an incoming news article. Once the engine is deployed it can process the query, and then return a Predicted Result of the form
+
+`{category : String, confidence : Double}`.
+
+Here category is the model's class assignment for this new text document (i.e. the best guess for this article's categorization), and confidence, a value between 0 and 1 representing your confidence in the category prediction (0 meaning you have no confidence in the prediction). The Actual Result is of the form
+
+`{category : String}`.
+
+This is used in the evaluation stage when estimating the performance of your predictive model (how well does the model predict categories). Please refer to the [following tutorial](https://docs.prediction.io/customize/) for a more detailed explanation of how your engine will interact with your web application, as well as an in depth-overview of DASE.
+
+
+## Quick Start
+
+This is a quick start guide in case you want to start using the engine right away. Sample email data for spam classification will be used. For more detailed information, read the subsequent sections.
+
+
+### 1. Create a new application.
+
+After the application is created, you will be given an access key and application ID for the application.
+
+```
+$ pio app new MyTextApp
+```
+
+### 2. Import the tutorial data.
+
+There are three different data sets available, each giving a different use case for this engine. Please refer to the **Data Source: Reading Event Data** section to see how to appropriate modify the `DataSource` class for use with each respective data set. The default data set is an e-mail spam data set.
+
+These data sets have already been processed and are ready for [batch import](/datacollection/batchimport/). Replace `***` with your actual application ID.
+
+```
+$ pio import --appid *** --input data/stopwords.json
+
+$ pio import --appid *** --input data/emails.json
+```
+
+### 3. Set the engine parameters in the file `engine.json`.
+
+The default settings are shown below. By default, it uses the algorithm name "lr" which is logstic regression. Please see later section for more detailed explanation of engine.json setting.
+
+Make sure the "appName" is same as the app you created in step1.
+
+
+```
+{
+ "id": "default",
+ "description": "Default settings",
+ "engineFactory": "org.template.textclassification.TextClassificationEngine",
+ "datasource": {
+ "params": {
+ "appName": "MyTextApp"
+ }
+ },
+ "preparator": {
+ "params": {
+ "nGram": 1,
+ "numFeatures": 500,
+ "SPPMI": false
+ }
+ },
+ "algorithms": [
+ {
+ "name": "lr",
+ "params": {
+ "regParam": 0.00000005
+ }
+ }
+ ]
+}
+
+```
+
+### 4. Build your engine.
+
+```
+$ pio build --verbose
+```
+
+This command should take few minutes for the first time; all subsequent builds should be less than a minute. You can also run it without `--verbose` if you don't want to see all the log messages.
+
+Upon successful build, you should see a console message similar to the following
+
+```
+[INFO] [RegisterEngine$] Registering engine 6wxDy2hxLbvaMJra927ahFdQHDIVXeQz 266bae678c570dee58154b2338cef7aa1646e0d3
+[INFO] [Console$] Your engine is ready for training.
+```
+
+### 5.a. Train your model and deploy.
+
+```
+$ pio train
+```
+
+When your engine is trained successfully, you should see a console message similar to the following.
+
+```
+[INFO] [CoreWorkflow$] Training completed successfully.
+```
+
+Now your engine is ready to deploy. Run:
+
+```
+$ pio deploy
+```
+
+When the engine is deployed successfully and running, you should see a console message similar to the following:
+
+```
+[INFO] [HttpListener] Bound to /0.0.0.0:8000
+[INFO] [MasterActor] Engine is deployed and running. Engine API is live at http://0.0.0.0:8000.
+```
+
+Now you can send query to the engine. Open another terminal and send the following http request to the deployed engine:
+
+```
+$ curl -H "Content-Type: application/json" -d '{ "text":"I like speed and fast motorcycles." }' http://localhost:8000/queries.json
+```
+
+you should see following outputs returned by the engine:
+
+```
+{"category":"not spam","confidence":0.852619510921587}
+```
+
+Try another query:
+
+```
+$ curl -H "Content-Type: application/json" -d '{ "text":"Earn extra cash!" }' http://localhost:8000/queries.json
+```
+
+you should see following outputs returned by the engine:
+
+```
+{"category":"spam","confidence":0.5268770133242983}
+```
+
+
+### 5.b.Evaluate your training model and tune parameters.
+
+```
+$ pio eval org.template.textclassification.AccuracyEvaluation org.template.textclassification.EngineParamsList
+```
+
+**Note:** Training and evaluation stages are generally different stages of engine development. Evaluation is there to help you choose the best [algorithm parameters](/evaluation/paramtuning/) to use for training an engine that is to be deployed as a web service.
+
+Depending on your needs, in steps (5.x.) above, you can configure your Spark settings by typing a command of the form:
+
+```
+$ pio command command_parameters -- --master url --driver-memory {0}G --executor-memory {1}G --conf spark.akka.framesize={2} --total_executor_cores {3}
+```
+
+Only the latter commands are listed as these are some of the more commonly
+modified values. See the [Spark
+documentation](https://spark.apache.org/docs/latest/spark-standalone.html) and
+the [PredictionIO
+FAQ's](http://predictionio.incubator.apache.org/resources/faq/) for more
+information.
+
+**Note:** We recommend you set your driver memory to `1G` or `2G` as the data size when dealing with text can be very large.
+
+
+# Detailed Explanation of DASE
+
+## Importing Data
+
+In the quick start, email spam classification is used. This template can easily be modified for other types text classification.
+
+If you want to import different sets of data, follow the Quick Start instructions to import data from different files. Make sure that the Data Source is modified accordingly to match the `event`, `entityType`, and `properties` fields set for the specific dataset. The following section explains this in more detail.
+
+## Data Source: Reading Event Data
+
+Now that the data has been imported into PredictionIO's Event Server, it needs to be read from storage to be used by the engine. This is precisely what the DataSource engine component is for, which is implemented in the template script `DataSource.scala`. The class `Observation` serves as a wrapper for storing the information about a news document needed to train a model. The attribute label refers to the label of the category a document belongs to, and text, stores the actual document content as a string. The class TrainingData is used to store an RDD of Observation objects along with the set of stop words.
+
+The class `DataSourceParams` is used to specify the parameters needed to read and prepare the data for processing. This class is initialized with two parameters `appName` and `evalK`. The first parameter specifies your application name (i.e. MyTextApp), which is needed so that the DataSource component knows where to pull the event data from. The second parameter is used for model evaluation and specifies the number of folds to use in [cross-validation](http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29) when estimating a model performance metric.
+
+The final and most important ingredient is the DataSource class. This is initialized with its corresponding parameter class, and extends `PDataSource`. This **must** implement the method `readTraining` which returns an instance of type TrainingData. This method completely relies on the defined private methods readEventData and readStopWords. Both of these functions read data observations as Event instances, create an RDD containing these events and finally transforms the RDD of events into an object of the appropriate type as seen below:
+
+```scala
+...
+private def readEventData(sc: SparkContext) : RDD[Observation] = {
+ //Get RDD of Events.
+ PEventStore.find(
+ appName = dsp.appName,
+ entityType = Some("content"), // specify data entity type
+ eventNames = Some(List("e-mail")) // specify data event name
+
+ // Convert collected RDD of events to and RDD of Observation
+ // objects.
+ )(sc).map(e => {
+ val label : String = e.properties.get[String]("label")
+ Observation(
+ if (label == "spam") 1.0 else 0.0,
+ e.properties.get[String]("text"),
+ label
+ )
+ }).cache
+ }
+
+ // Helper function used to store stop words from
+ // event server.
+ private def readStopWords(sc : SparkContext) : Set[String] = {
+ PEventStore.find(
+ appName = dsp.appName,
+ entityType = Some("resource"),
+ eventNames = Some(List("stopwords"))
+
+ //Convert collected RDD of strings to a string set.
+ )(sc)
+ .map(e => e.properties.get[String]("word"))
+ .collect
+ .toSet
+ }
+...
+```
+
+Note that `readEventData` and `readStopWords` use different entity types and event names, but use the same application name. This is because the sample import script imports two different data types, documents and stop words. These field distinctions are required for distinguishing between the two. The method `readEval` is used to prepare the different cross-validation folds needed for evaluating your model and tuning hyper parameters.
+
+Now, the default dataset used for training is contained in the file `data/emails.json` and contains a set of e-mail spam data. If we want to switch over to one of the other data sets we must make sure that the `eventNames` and `entityType` fields are changed accordingly.
+
+In the data/ directory, you will find different sets of data files for different types of text classifcaiton application. The following show one observation from each of the provided data files:
+
+- `emails.json`:
+
+```
+{"eventTime": "2015-06-08T16:45:00.590+0000", "entityId": 1, "properties": {"text": "Subject: dobmeos with hgh my energy level has gone up ! stukm\nintroducing\ndoctor - formulated\nhgh\nhuman growth hormone - also called hgh\nis referred to in medical science as the master hormone . it is very plentiful\nwhen we are young , but near the age of twenty - one our bodies begin to produce\nless of it . by the time we are forty nearly everyone is deficient in hgh ,\nand at eighty our production has normally diminished at least 90 - 95 % .\nadvantages of hgh :\n- increased muscle strength\n- loss in body fat\n- increased bone density\n- lower blood pressure\n- quickens wound healing\n- reduces cellulite\n- improved vision\n- wrinkle disappearance\n- increased skin thickness texture\n- increased energy levels\n- improved sleep and emotional stability\n- improved memory and mental alertness\n- increased sexual potency\n- resistance to common illness\n- strengthened heart muscle\n- controlle
d cholesterol\n- controlled mood swings\n- new hair growth and color restore\nread\nmore at this website\nunsubscribe\n", "label": "spam"}, "event": "e-mail", "entityType": "content"}
+
+```
+
+- `20newsgroups.json`:
+
+```
+{"entityType": "source", "eventTime": "2015-06-08T18:01:55.003+0000", "event": "documents", "entityId": 1, "properties": {"category": "sci.crypt", "text": "From: rj@ri.cadre.com (Rob deFriesse)\nSubject: Can DES code be shipped to Canada?\nArticle-I.D.: fripp.1993Apr22.125402.27561\nReply-To: rj@ri.cadre.com\nOrganization: Cadre Technologies Inc.\nLines: 13\nNntp-Posting-Host: 192.9.200.19\n\nSomeone in Canada asked me to send him some public domain DES file\nencryption code I have. Is it legal for me to send it?\n\nThanx.\n--\nEschew Obfuscation\n\nRob deFriesse Mail: rj@ri.cadre.com\nCadre Technologies Inc. Phone: (401) 351-5950\n222 Richmond St. Fax: (401) 351-7380\nProvidence, RI 02903\n\nI don't speak for my employer.\n", "label": 11.0}}
+```
+
+- `sentimentanalysis.json`:
+
+```
+{"eventTime": "2015-06-08T16:58:14.278+0000", "entityId": 23714, "entityType": "source", "properties": {"phrase": "Tosca 's intoxicating ardor", "sentiment": 3}, "event": "phrases"}
+```
+
+Now, note that the `entityType`, `event`, and `properties` fields for the `20newsgroups.json` dataset differ from the default `emails.json` set. Default DataSource implementation is to read from `email.json` data set. If you want to use others such as newsgroups data set, the engine's Data Source component must be modified accordingly. To do this, you need only modify the method `readEventData` as follows:
+
+### Modify DataSource to Read `20newsgroups.json`
+
+```scala
+private def readEventData(sc: SparkContext) : RDD[Observation] = {
+ //Get RDD of Events.
+ PEventStore.find(
+ appName = dsp.appName,
+ entityType = Some("source"), // specify data entity type
+ eventNames = Some(List("documents")) // specify data event name
+
+ // Convert collected RDD of events to and RDD of Observation
+ // objects.
+ )(sc).map(e => {
+
+ Observation(
+ e.properties.get[Double]("label"),
+ e.properties.get[String]("text"),
+ e.properties.get[String]("category")
+ )
+ }).cache
+ }
+```
+
+### Modify DataSource to Read `sentimentanalysis.json`
+
+```scala
+private def readEventData(sc: SparkContext) : RDD[Observation] = {
+ //Get RDD of Events.
+ PEventStore.find(
+ appName = dsp.appName,
+ entityType = Some("source"), // specify data entity type
+ eventNames = Some(List("phrases")) // specify data event name
+
+ // Convert collected RDD of events to and RDD of Observation
+ // objects.
+ )(sc).map(e => {
+ val label = e.properties.get[Double]("sentiment")
+
+ Observation(
+ label,
+ e.properties.get[String]("phrase"),
+ label.toString
+ )
+ }).cache
+ }
+```
+
+Note that `event` field in the json file refers to the `eventNames` field in the `readEventData` method. When using this engine with a custom data set, you need to make sure that the respective json fields match with the corresponding fields in the DataSource component. We have included a data sanity check with this engine component that lets you know if your data is actually being read in. If you have 0 observations being read, you should see the following output when your training process performs the Training Data sanity check:
+
+`Data set is empty, make sure event fields match imported data.`
+
+This data sanity check is a PredictionIO feature available for your `TrainingData` and `PreparedData` classes. The following code block demonstrates how the sanity check is implemented:
+
+```scala
+class TrainingData(
+ val data : RDD[Observation],
+ val stopWords : Set[String]
+) extends Serializable with SanityCheck {
+
+ // Sanity check to make sure your data is being fed in correctly.
+
+ def sanityCheck {
+ try {
+ val obs : Array[Double] = data.takeSample(false, 5).map(_.label)
+
+ println()
+ (0 until 5).foreach(
+ k => println("Observation " + (k + 1) +" label: " + obs(k))
+ )
+ println()
+ } catch {
+ case (e : ArrayIndexOutOfBoundsException) => {
+ println()
+ println("Data set is empty, make sure event fields match imported data.")
+ println()
+ }
+ }
+
+ }
+
+}
+```
+
+## Preparator : Data Processing With DASE
+
+Recall that the Preparator stage is used for doing any prior data processing needed to fit a predictive model. In line with the separation of concerns, the Data Model implementation, PreparedData, is built to do the heavy lifting needed for this data processing. The Preparator must simply implement the prepare method which outputs an object of type PreparedData. This requires you to specify two n-gram window components, and two inverse i.d.f. window components (these terms will be defined in the following section). Therefore a custom class of parameters for the Preparator component, PreparatorParams, must be incorporated. The code defining the full Preparator component is given below:
+
+```scala
+// 1. Initialize Preparator parameters. Recall that for our data
+// representation we are only required to input the n-gram window
+// components.
+
+case class PreparatorParams(
+ nGram: Int,
+ numFeatures: Int = 5000,
+ SPPMI: Boolean
+) extends Params
+
+
+
+// 2. Initialize your Preparator class.
+
+class Preparator(pp: PreparatorParams) extends PPreparator[TrainingData, PreparedData] {
+
+ // Prepare your training data.
+ def prepare(sc : SparkContext, td: TrainingData): PreparedData = {
+ new PreparedData(td, pp.nGram)
+ }
+}
+
+```
+
+The simplicity of this stage implementation truly exemplifies one of the benefits of using the PredictionIO platform. For developers, it is easy to incorporate different classes and tools into the DASE framework so that the process of creating an engine is greatly simplified which helps increase your productivity. For data scientists, the load of implementation details you need to worry about is minimized so that you can focus on what is important to you: training a good predictive model.
+
+The following subsection explains the class PreparedData, which actually handles the transformation of text documents to feature vectors.
+
+### PreparedData: Text Vectorization and Feature Reduction
+
+The Scala class PreparedData which takes the parameters td, nGram, where td is an object of class TrainingData. The other parameter specifies the n-gram parametrization which will be described shortly.
+
+It will be easier to explain the preparation process with an example, so consider the document \\(d\\):
+
+`"Hello, my name is Marco."`
+
+The first thing you need to do is break up \\(d\\) into an array of "allowed tokens." You can think of a token as a terminating sequence of characters that exist in a document (think of a word in a sentence). For example, the list of tokens that appear in \\(d\\) is:
+
+```scala
+val A = Array("Hello", ",", "my", "name", "is", "Marco", ".")
+```
+
+Recall that a set of stop words was also imported in the previous sections. This set of stop words contains all the words (or tokens) that you do not want to include once documents are tokenized. Those tokens that appear in \\(d\\) and are not contained in the set of stop words will be called allowed tokens. So, if the set of stop words is `{"my", "is"}`, then the list of allowed tokens appearing in \\(d\\) is:
+
+```scala
+val A = Array("Hello", ",", "name", "Marco", ".")
+```
+
+The next step in the data representation is to take the array of allowed tokens and extract a set of n-grams and a corresponding value indicating the number of times a given n-gram appears. The set of n-grams for n equal to 1 and 2 in the running example is the set of elements of the form `[A(`\\(i\\)`)]` and `[A(`\\(j\\)`), A(`\\(j + 1\\)`)]`, respectively. In the general case, the set of n-grams extracted from an array of allowed tokens `A` will be of the form `[A(`\\(i\\)`), A(`\\(i + 1\\)`), ..., A(`\\(i + n - 1\\)`)]` for \\(i = 0, 1, 2, ...,\\) `A.size` \\(- n\\). You can set `n` with the `nGram` parameter option in your `PreparatorParams`.
+
+We use MLLib's `HashingTF` class to implement the conversion from text to term frequency vectors, and can be seen in the following method of the class `PreparedData`:
+
+```scala
+...
+ // 1. Hashing function: Text -> term frequency vector.
+
+ private val hasher = new HashingTF()
+
+ private def hashTF (text : String) : Vector = {
+ val newList : Array[String] = text.split(" ")
+ .sliding(nGram)
+ .map(_.mkString)
+ .toArray
+
+ hasher.transform(newList)
+ }
+
+ // 2. Term frequency vector -> t.f.-i.d.f. vector.
+
+ val idf : IDFModel = new IDF().fit(td.data.map(e => hashTF(e.text)))
+...
+```
+
+The next step is, once all of the observations have been hashed, to collect all n-grams and compute their corresponding [t.f.-i.d.f. value](http://en.wikipedia.org/wiki/Tf%E2%80%93idf). The t.f.-i.d.f. transformation is defined for n-grams, and helps to give less weight to those n-grams that appear with high frequency across all documents, and vice versa. This helps to leverage the predictive power of those words that appear rarely, but can make a big difference in the categorization of a given text document. This is implemented using MLLib's `IDF` and `IDFModel` classes:
+
+```scala
+// 2. Term frequency vector -> t.f.-i.d.f. vector.
+
+ val idf : IDFModel = new IDF().fit(td.data.map(e => hashTF(e.text)))
+```
+
+
+The last two functions that will be mentioned are the methods you will actually use for the data transformation. The method transform takes a document and outputs a sparse vector (MLLib implementation). The transformData method simply transforms the TrainingData input (a corpus of documents) into a set of vectors that can now be used for training. The method transform is used both to transform the training data and future queries.
+
+```scala
+...
+// 3. Document Transformer: text => tf-idf vector.
+
+ def transform(text : String): Vector = {
+ // Map(n-gram -> document tf)
+ idf.transform(hashTF(text))
+ }
+
+
+ // 4. Data Transformer: RDD[documents] => RDD[LabeledPoints]
+
+ val transformedData: RDD[(LabeledPoint)] = {
+ td.data.map(e => LabeledPoint(e.label, transform(e.text)))
+ }
+```
+
+The last and final object implemented in this class simply creates a Map with keys being class labels and values, the corresponding category.
+
+```scala
+ // 5. Finally extract category map, associating label to category.
+ val categoryMap = td.data.map(e => (e.label, e.category)).collectAsMap
+```
+
+
+## Algorithm Component
+
+The algorithm components in this engine, `NBAlgorithm` and `LRAlgorithm`, actually follows a very general form. Firstly, a parameter class must again be initialized to feed in the corresponding Algorithm model parameters. For example, NBAlgorithm incorporates NBAlgorithmParams which holds the appropriate additive smoothing parameter lambda for the Naive Bayes model.
+
+
+The main class of interest in this component is the class that extends [P2LAlgorithm](https://docs.prediction.io/api/current/#io.prediction.controller.P2LAlgorithm). This class must implement a method named train which will output your predictive model (as a concrete object, this will be implemented via a Scala class). It must also implement a predict method that transforms a query to an appropriate feature vector, and uses this to predict with the fitted model. The vectorization function is implemented by a PreparedData object, and the categorization (prediction) is handled by an instance of the NBModel implementation. Again, this demonstrates the facility with which different models can be incorporated into PredictionIO's DASE architecture.
+
+The model class itself will be discussed in the following section, however, turn your attention to the TextManipulationEngine object defined in the script `Engine.scala`. You can see here that the engine is initialized by specifying the DataSource, Preparator, and Serving classes, as well as a Map of algorithm names to Algorithm classes. This tells the engine which algorithms to run. In practice, you can have as many statistical learning models as you'd like, you simply have to implement a new algorithm component to do this. However, this general design form will persist, and the main meat of the work should be in the implementation of your model class.
+
+The following subsection will go over our Naive Bayes implementation in NBModel.
+
+
+### Naive Bayes Classification
+
+This Training Model class only uses the Multinomial Naive Bayes [implementation](https://spark.apache.org/docs/latest/mllib-naive-bayes.html) found in the Spark MLLib library. However, recall that the predicted results required in the specifications listed in the overview are of the form:
+
+
+`{category: String, confidence: Double}`.
+
+The confidence value should really be interpreted as the probability that a document belongs to a category given its vectorized form. Note that MLLib's Naive Bayes model has the class members pi (\\(\pi\\)), and theta (\\(\theta\\)). \\(\pi\\) is a vector of log prior class probabilities, which shows your prior beliefs regarding the probability that an arbitrary document belongs in a category. \\(\theta\\) is a C \\(\times\\) D matrix, where C is the number of classes, and D, the number of features, giving the log probabilities that parametrize the Multinomial likelihood model assumed for each class. The multinomial model is easiest to think about as a problem of randomly throwing balls into bins, where the ball lands in each bin with a certain probability. The model treats each n-gram as a bin, and the corresponding t.f.-i.d.f. value as the number of balls thrown into it. The likelihood is the probability of observing a (vectorized) document given that it comes from a particular cl
ass.
+
+Now, letting \\(\mathbf{x}\\) be a vectorized text document, then it can be shown that the vector
+
+$$
+\frac{\exp\left(\pi + \theta\mathbf{x}\right)}{\left|\left|\exp\left(\pi + \theta\mathbf{x}\right)\right|\right|}
+$$
+
+is a vector with C components that represent the posterior class membership probabilities for the document given \\(\mathbf{x}\\). That is, the update belief regarding what category this document belongs to after observing its vectorized form. This is the motivation behind defining the class NBModel which uses Spark MLLib's NaiveBayesModel, but implements a separate prediction method.
+
+The private methods innerProduct and getScores are implemented to do the matrix computation above.
+
+```scala
+...
+ // 2. Set up linear algebra framework.
+
+ private def innerProduct (x : Array[Double], y : Array[Double]) : Double = {
+ x.zip(y).map(e => e._1 * e._2).sum
+ }
+
+ val normalize = (u: Array[Double]) => {
+ val uSum = u.sum
+
+ u.map(e => e / uSum)
+ }
+
+
+
+ // 3. Given a document string, return a vector of corresponding
+ // class membership probabilities.
+
+ private def getScores(doc: String): Array[Double] = {
+ // Helper function used to normalize probability scores.
+ // Returns an object of type Array[Double]
+
+ // Vectorize query,
+ val x: Vector = pd.transform(doc)
+
+ normalize(
+ nb.pi
+ .zip(nb.theta)
+ .map(
+ e => exp(innerProduct(e._2, x.toArray) + e._1))
+ )
+ }
+...
+```
+
+
+Once you have a vector of class probabilities, you can classify the text document to the category with highest posterior probability, and, finally, return both the category as well as the probability of belonging to that category (i.e. the confidence in the prediction) given the observed data. This is implemented in the method predict.
+
+```scala
+...
+ // 4. Implement predict method for our model using
+ // the prediction rule given in tutorial.
+
+ def predict(doc : String) : PredictedResult = {
+ val x: Array[Double] = getScores(doc)
+ val y: (Double, Double) = (nb.labels zip x).maxBy(_._2)
+ new PredictedResult(pd.categoryMap.getOrElse(y._1, ""), y._2)
+ }
+```
+
+### Logistic Regression Classification
+
+To use the alternative multinomial logistic regression algorithm change your `engine.json` as follows:
+
+```json
+ {
+ "id": "default",
+ "description": "Default settings",
+ "engineFactory": "org.template.textclassification.TextClassificationEngine",
+ "datasource": {
+ "params": {
+ "appName": "MyTextApp"
+ }
+ },
+ "preparator": {
+ "params": {
+ "nGram": 2
+ }
+ },
+ "algorithms": [
+ {
+ "name": "regParam",
+ "params": {
+ "regParam": 0.1
+ }
+ }
+ ]
+}
+```
+
+
+## Serving: Delivering the Final Prediction
+
+The serving component is the final stage in the engine, and in a sense, the most important. This is the final stage in which you combine the results obtained from the different models you choose to run. The Serving class extends the [LServing](https://docs.prediction.io/api/current/#io.prediction.controller.LServing) class which must implement a method called serve. This takes a query and an associated sequence of predicted results, which contains the predicted results from the different algorithms that are implemented in your engine, and combines the results to yield a final prediction. It is this final prediction that you will receive after sending a query.
+
+For example, you could choose to slightly modify the implementation to return class probabilities coming from a mixture of model estimates for class probabilities, or any other technique you could conceive for combining your results. The default engine setting has this set to yield the label from the model predicting with greater confidence.
+
+
+
+## Evaluation: Model Assessment and Selection
+
+ A predictive model needs to be evaluated to see how it will generalize to future observations. PredictionIO uses cross-validation to perform model performance metric estimates needed to assess your particular choice of model. The script `Evaluation.scala` available with the engine template exemplifies what a usual evaluator setup will look like. First, you must define an appropriate metric. In the engine template, since the topic is text classification, the default metric implemented is category accuracy.
+
+ Second you must define an evaluation object (i.e. extends the class [Evaluation](https://docs.prediction.io/api/current/#io.prediction.controller.Evaluation)).
+Here, you must specify the actual engine and metric components that are to be used for the evaluation. In the engine template, the specified engine is the TextManipulationEngine object, and metric, Accuracy. Lastly, you must specify the parameter values that you want to test in the cross validation. You see in the following block of code:
+
+```scala
+object EngineParamsList extends EngineParamsGenerator {
+
+ // Set data source and preparator parameters.
+ private[this] val baseEP = EngineParams(
+ dataSourceParams = DataSourceParams(appName = "marco-MyTextApp", evalK = Some(5)),
+ preparatorParams = PreparatorParams(nMin = 1, nMax = 2)
+ )
+
+ // Set the algorithm params for which we will assess an accuracy score.
+ engineParamsList = Seq(
+ baseEP.copy(algorithmParamsList = Seq(("nb", NBAlgorithmParams(0.5)))),
+ baseEP.copy(algorithmParamsList = Seq(("nb", NBAlgorithmParams(1.5)))),
+ baseEP.copy(algorithmParamsList = Seq(("nb", NBAlgorithmParams(5))))
+ )
+```
+
+
+## Engine Deployment
+
+Once an engine is ready for deployment it can interact with your web application in real-time. This section will cover how to send and receive queries from your engine, gather more data, and re-training your model with the newly gathered data.
+
+### Sending Queries
+
+Recall that one of the greatest advantages of using the PredictionIO platform is that once your engine is deployed, you can respond to queries in real-time. Recall that our queries are of the form
+
+`{"text" : "..."}`.
+
+To actually send a query you can use our REST API by typing in the following shell command:
+
+```
+curl -H "Content-Type: application/json" -d '{ "text":"I like speed and fast motorcycles." }' http://localhost:8000/queries.json
+```
+
+There are a number of [SDK's](https://github.com/PredictionIO) you can use to send your queries and obtain a response. Recall that our predicted response is of the form
+
+```
+{"category" : "class", "confidence" : 1.0}
+```
+
+which is what you should see upon inputting the latter command for querying.
+
+### Gathering More Data and Retraining Your Model
+
+The importing data section that is included in this tutorial uses a sample data set for illustration purposes, and uses the PredictionIO Python SDK to import the data. However, there are a variety of ways that you can [import](<%= url_root %>/datacollection/eventapi/) your collected data (via REST or other SDKs).
+
+
+As you continue to collect your data, it is quite easy to retrain your model once you actually import your data into the Event Server. You simply repeat the steps listed in the Quick Start guide. We re-list them here again:
+
+
+**1.** Build your engine.
+
+```
+$ pio build
+```
+
+**2.a.** Evaluate your training model and tune parameters.
+
+```
+$ pio eval org.template.textclassification.AccuracyEvaluation org.template.textclassification.EngineParamsList
+```
+
+**2.b.** Train your model and deploy.
+
+```
+$ pio train
+$ pio deploy
+```
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/deploy/index.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/deploy/index.html.md b/docs/manual/source/deploy/index.html.md
index be18a0d..5df5213 100644
--- a/docs/manual/source/deploy/index.html.md
+++ b/docs/manual/source/deploy/index.html.md
@@ -2,9 +2,11 @@
title: Deploying an Engine
---
-An engine must be **built** (i.e. `pio build`) and **trained** (i.e. `pio train`) before it can be deployed as a web service.
+An engine must be **built** (i.e. `pio build`) and **trained** (i.e. `pio
+train`) before it can be deployed as a web service.
-WARNING: The engine server is not protected by authentication, and the instructions below assume deployment in a trusted environment. See the section [Deploying with AWS](/system/deploy-cloudformation/) for a production deployment example.
+WARNING: The engine server is not protected by authentication, and the
+instructions below assume deployment in a trusted environment.
## Deploying an Engine the First Time
@@ -15,7 +17,7 @@ After you have [downloaded an Engine Template](/start/download/), you can deplo
2. Run `pio train` to train a predictive model with training data
3. Run `pio deploy` to deploy the engine as a service
-A deployed engine listens to port 8000 by default. Your application can [send query to retrieve prediction](/appintegration/) in real-time through the REST interface.
+A deployed engine listens to port 8000 by default. Your application can [send query to retrieve prediction](/appintegration/) in real-time through the REST interface.
**Note**: a new engine deployed as above will have no data to start with. Your engine may come with a `data/` directory with some sample data that you can import, not all have this. Check the quickstart instructions for your template.
@@ -55,7 +57,8 @@ pio deploy --port 8123 --ip 1.2.3.4
## Retrain and Deploy Script
-A retrain and deploy script is available [in the *examples/redeploy-script* directory](https://github.com/PredictionIO/PredictionIO/tree/develop/examples/redeploy-script).
+A retrain and deploy script is available [in the *examples/redeploy-script*
+directory](https://github.com/apache/incubator-predictionio/tree/develop/examples/redeploy-script).
To use the script, copy *local.sh.template* as *local.sh*, *redeploy.sh* as (say) *MyEngine_Redeploy_(production).sh* (Name of the script will appear as title of email) and put both files under the *scripts/* directory of your engine.
Then, modify the settings inside both file, filling in details like `PIO_HOME`, `LOG_DIR`, `TARGET_EMAIL`, `ENGINE_JSON` and others.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/index.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/index.html.md.erb b/docs/manual/source/index.html.md.erb
index 2b52143..db43ebb 100644
--- a/docs/manual/source/index.html.md.erb
+++ b/docs/manual/source/index.html.md.erb
@@ -1,13 +1,13 @@
---
-title: Welcome to PredictionIO!
-description: PredictionIO Open Source Machine Learning Server
+title: Welcome to Apache PredictionIO (incubating)!
+description: Apache PredictionIO (incubating) Open Source Machine Learning Server
---
-## What is PredictionIO?
+## What is Apache PredictionIO (incubating)?
-PredictionIO is an **open source Machine Learning Server** built on top of
-state-of-the-art open source stack for developers and data scientists create
-predictive engines for any machine learning task. It lets you:
+Apache PredictionIO (incubating) is an **open source Machine Learning Server**
+built on top of state-of-the-art open source stack for developers and data
+scientists create predictive engines for any machine learning task. It lets you:
* quickly build and deploy an engine as a web service on production with
[customizable templates](http://templates.prediction.io);
@@ -23,10 +23,10 @@ predictive engines for any machine learning task. It lets you:
into your engine;
* simplify data infrastructure management.
-PredictionIO can be [installed](https://docs.prediction.io/install/) as a full
-machine learning stack, bundled with **Apache Spark**, **MLlib**, **HBase**,
-**Spray** and **Elasticsearch**, which simplifies and accelerates scalable
-machine learning infrastructure management.
+Apache PredictionIO (incubating) can be [installed](/install/) as a full machine
+learning stack, bundled with **Apache Spark**, **MLlib**, **HBase**, **Spray**
+and **Elasticsearch**, which simplifies and accelerates scalable machine
+learning infrastructure management.
| Getting Started | Developer Guides | ML Education and Usage | PredictionIO SDKs |
| --------------- | ---------------- | ---------------------- | ----------------- |
@@ -42,16 +42,13 @@ machine learning infrastructure management.
## Release Notes
-A summary of the changes in each release in the current series are on GitHub
-[here](https://github.com/PredictionIO/PredictionIO/blob/master/RELEASE.md)
+A summary of the changes in each release before migrating to Apache Software
+Foundation are on GitHub
+[here](https://github.com/PredictionIO/PredictionIO/blob/master/RELEASE.md).
## Licensing
-PredictionIO is licensed under the Apache License, Version 2.0. See
-[LICENSE](https://github.com/PredictionIO/PredictionIO/blob/master/LICENSE.txt)
+Apache PredictionIO (incubating) is licensed under the Apache License, Version
+2.0. See
+[LICENSE](https://github.com/apache/incubator-predictionio/blob/master/LICENSE.txt)
for the full license text.
-
-### Documentation
-
-Documentation is under a [Creative Commons Attribution-NonCommercial-ShareAlike
-3.0 License](http://creativecommons.org/licenses/by-nc-sa/3.0/).
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/install/index.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/install/index.html.md.erb b/docs/manual/source/install/index.html.md.erb
index b5c5854..84e9719 100644
--- a/docs/manual/source/install/index.html.md.erb
+++ b/docs/manual/source/install/index.html.md.erb
@@ -1,30 +1,25 @@
---
-title: Installing PredictionIO
+title: Installing Apache PredictionIO (incubating)
---
-## Method 1: Launch an AWS Instance
+## Method 1: Quick Install
-You can [launch PredictionIO on Amazon Web Services](launch-aws.html).
-This is a hassle-free way to get a well-configured PredictionIO server quickly.
+On Linux / Mac OS X, Apache PredictionIO (incubating) can now be installed with a single command:
-There is also a [AWS Cloudformation version for PredictionIO](/system/deploy-cloudformation/).
-
-## Method 2: Quick Install
-
-On Linux / Mac OS X, PredictionIO can now be installed with a single command:
-<br />
-`$ bash -c "$(curl -s https://install.prediction.io/install.sh)"`
+```
+$ bash -c "$(curl -s https://raw.githubusercontent.com/apache/incubator-predictionio/master/bin/install.sh)"
+```
The above script will complete the installation for you.
<%= partial 'shared/install/proceed_template' %>
-## Method 3: Manual Install
+## Method 2: Manual Install
### Prerequisites
It is **very important** to meet the minimum version of the following
-technologies that power PredictionIO.
+technologies that power Apache PredictionIO (incubating).
* Apache Hadoop 2.4.0 (optional, required only if YARN and HDFS are needed)
* Apache Spark 1.3.0 for Hadoop 2.4
@@ -50,25 +45,25 @@ operation mode without further configuration, may not work. In that case,
configure your Apache Spark to run in [standalone cluster
mode](http://spark.apache.org/docs/latest/spark-standalone.html).
-### Installing PredictionIO
+### Installing Apache PredictionIO (incubating)
-PredictionIO runs on a Java virtual machine, so it runs on most platforms.
-Choose your platform below:
+Apache PredictionIO (incubating) runs on a Java virtual machine, so it runs on
+most platforms. Choose your platform below:
-* [Installing PredictionIO on Linux / Mac OS X](install-linux.html)
-* [Installing PredictionIO from Source Code](install-sourcecode.html)
-* [Installing PredictionIO with Vagrant (VirtualBox)](install-vagrant.html)
-* [Installing PredictionIO with Terminal.com Snap](https://www.terminal.com/snapshot/f444bfb7538dfc596485374f56167ec6f79cbc16f793f013ad120067070eb81a)
+* [Installing Apache PredictionIO (incubating) on Linux / Mac OS X](install-linux.html)
+* [Installing Apache PredictionIO (incubating) from Source Code](install-sourcecode.html)
+* [Installing Apache PredictionIO (incubating) with Vagrant (VirtualBox)](install-vagrant.html)
+* [Installing Apache PredictionIO (incubating) with Terminal.com Snap](https://www.terminal.com/snapshot/f444bfb7538dfc596485374f56167ec6f79cbc16f793f013ad120067070eb81a)
You may also use one of the community-contributed packages to install
-PredictionIO:
+Apache PredictionIO (incubating):
-* [Installing PredictionIO with
+* [Installing Apache PredictionIO (incubating) with
Docker](/community/projects.html#docker-installation-for-predictionio)
-[//]: # (* *(coming soon)* Installing PredictionIO with Homebrew)
+[//]: # (* *(coming soon)* Installing Apache PredictionIO (incubating) with Homebrew)
@@ -82,10 +77,10 @@ Elasticsearch. See more [here](/resources/upgrade/).**
[//]: # (For production environment setup, please refer to [Production)
[//]: # (Deployment](/production/deploy.html) guide.)
-## Method 4: Terminal.com
+## Method 3: Terminal.com
Public snaps are available on our Terminal.com
[page](https://www.terminal.com/user/predictionio).
-Instance type **must be** set to *medium* or higher for PredictionIO to function
-correctly.
+Instance type **must be** set to *medium* or higher for Apache PredictionIO
+(incubating) to function correctly.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/install/install-linux.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/install/install-linux.html.md.erb b/docs/manual/source/install/install-linux.html.md.erb
index 8c69e19..c100611 100644
--- a/docs/manual/source/install/install-linux.html.md.erb
+++ b/docs/manual/source/install/install-linux.html.md.erb
@@ -1,12 +1,14 @@
---
-title: Installing PredictionIO on Linux / Mac OS X
+title: Installing Apache PredictionIO (incubating) on Linux / Mac OS X
---
## Method 1: Quick Install
-On Linux / Mac OS X, PredictionIO can now be installed with a single command:
-<br />
-`$ bash -c "$(curl -s https://install.prediction.io/install.sh)"`
+On Linux / Mac OS X, Apache PredictionIO (incubating) can now be installed with a single command:
+
+```
+$ bash -c "$(curl -s https://raw.githubusercontent.com/apache/incubator-predictionio/master/bin/install.sh)"
+```
The above script will complete the installation for you.
@@ -15,9 +17,9 @@ The above script will complete the installation for you.
## Method 2: Manual Install
If you do not want to use the Quick Install script, you can follow the steps
-below to setup PredictionIO and its dependencies. In these instructions we will
-assume you are in your home directory. Wherever you see `/home/abc`, replace it
-with your own home directory.
+below to setup Apache PredictionIO (incubating) and its dependencies. In these
+instructions we will assume you are in your home directory. Wherever you see
+`/home/abc`, replace it with your own home directory.
### Java
@@ -32,24 +34,26 @@ Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)
```
-### Download PredictionIO
+### Download Apache PredictionIO (incubating)
-Download PredictionIO and extract it.
+Download Apache PredictionIO (incubating) and extract it.
```
$ cd
$ pwd
/home/abc
-$ wget https://d8k1yxp8elc6b.cloudfront.net/PredictionIO-<%= data.versions.pio %>.tar.gz
+$ wget http://download.prediction.io/PredictionIO-<%= data.versions.pio %>.tar.gz
$ tar zxvf PredictionIO-<%= data.versions.pio %>.tar.gz
```
+NOTE: Download instructions above apply to previous non-Apache releases only.
+Once we have made an Apache release, new instructions will be provided.
### Installing Dependencies
-Let us install dependencies inside a subdirectory of the PredictionIO
-installation. By following this convention, you can use PredictionIO's default
-configuration as is.
+Let us install dependencies inside a subdirectory of the Apache PredictionIO
+(incubating) installation. By following this convention, you can use
+PredictionIO's default configuration as is.
```
$ mkdir PredictionIO-<%= data.versions.pio %>/vendors
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/install/install-sourcecode.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/install/install-sourcecode.html.md.erb b/docs/manual/source/install/install-sourcecode.html.md.erb
index c21d885..ab78b1c 100644
--- a/docs/manual/source/install/install-sourcecode.html.md.erb
+++ b/docs/manual/source/install/install-sourcecode.html.md.erb
@@ -1,5 +1,5 @@
---
-title: Installing PredictionIO from Source Code
+title: Installing Apache PredictionIO (incubating) from Source Code
---
INFO: Assuming you are following the directory structure in the followoing,
@@ -7,11 +7,12 @@ replace `/home/abc` with your own home directory wherever you see it.
## Building
-Run the following to download and build PredictionIO from its source code.
+Run the following to download and build Apache PredictionIO (incubating) from
+its source code.
```
-$ git clone https://github.com/PredictionIO/PredictionIO.git
-$ cd PredictionIO
+$ git clone https://github.com/apache/incubator-predictionio.git
+$ cd incubator-predictionio
$ git checkout master
$ ./make-distribution.sh
```
@@ -36,9 +37,9 @@ $ tar zxvf PredictionIO-<%= data.versions.pio %>.tar.gz
## Installing Dependencies
-Let us install dependencies inside a subdirectory of the PredictionIO
-installation. By following this convention, you can use PredictionIO's default
-configuration as is.
+Let us install dependencies inside a subdirectory of the Apache PredictionIO
+(incubating) installation. By following this convention, you can use
+Apache PredictionIO (incubating)'s default configuration as is.
```
$ mkdir PredictionIO-<%= data.versions.pio %>/vendors
@@ -52,7 +53,7 @@ $ mkdir PredictionIO-<%= data.versions.pio %>/vendors
### Storage Setup
-#### PostgreSQL Setup
+#### PostgreSQL Setup
<%=partial 'shared/install/postgres' %>
@@ -88,4 +89,4 @@ export JAVA_HOME=/usr/lib/jvm/java-8-oracle
Now you have installed everything you need!
-<%= partial 'shared/install/proceed_template' %>
\ No newline at end of file
+<%= partial 'shared/install/proceed_template' %>
[06/15] incubator-predictionio git commit: Use GitHub for binaries
Posted by do...@apache.org.
Use GitHub for binaries
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/4f39d72c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/4f39d72c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/4f39d72c
Branch: refs/heads/develop
Commit: 4f39d72cbeb443578d5dfefb97b6d2d76223afbc
Parents: 2376545
Author: Donald Szeto <do...@prediction.io>
Authored: Mon Apr 11 16:06:54 2016 -0700
Committer: Donald Szeto <do...@prediction.io>
Committed: Mon Apr 11 16:06:54 2016 -0700
----------------------------------------------------------------------
docs/manual/source/install/install-linux.html.md.erb | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/4f39d72c/docs/manual/source/install/install-linux.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/install/install-linux.html.md.erb b/docs/manual/source/install/install-linux.html.md.erb
index 8c69e19..527ce43 100644
--- a/docs/manual/source/install/install-linux.html.md.erb
+++ b/docs/manual/source/install/install-linux.html.md.erb
@@ -40,7 +40,7 @@ Download PredictionIO and extract it.
$ cd
$ pwd
/home/abc
-$ wget https://d8k1yxp8elc6b.cloudfront.net/PredictionIO-<%= data.versions.pio %>.tar.gz
+$ wget https://github.com/PredictionIO/PredictionIO/releases/download/v0.9.6/PredictionIO-<%= data.versions.pio %>.tar.gz
$ tar zxvf PredictionIO-<%= data.versions.pio %>.tar.gz
```
[09/15] incubator-predictionio git commit: [PIO-6] Initial doc site
Posted by do...@apache.org.
[PIO-6] Initial doc site
Addresses pressing installation and mailing list issues
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/5be3cca8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/5be3cca8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/5be3cca8
Branch: refs/heads/develop
Commit: 5be3cca83e341ab7b2c5347004311c1de2b2aea4
Parents: 03cab0f
Author: Donald Szeto <do...@apache.org>
Authored: Wed Jul 20 12:04:14 2016 -0700
Committer: Donald Szeto <do...@apache.org>
Committed: Wed Jul 20 12:04:14 2016 -0700
----------------------------------------------------------------------
docs/manual/config.rb | 2 +-
docs/manual/data/nav/main.yml | 9 +-
docs/manual/helpers/application_helpers.rb | 4 +-
docs/manual/source/algorithm/index.html.md | 9 +-
docs/manual/source/appintegration/index.html.md | 24 +-
docs/manual/source/cli/index.html.md | 43 +-
.../source/community/contribute-code.html.md | 106 +--
.../community/contribute-documentation.html.md | 41 +-
.../source/community/contribute-sdk.html.md | 45 +-
.../source/community/contribute-webhook.html.md | 19 +-
docs/manual/source/community/index.html.md | 20 +-
docs/manual/source/community/projects.html.md | 12 +-
docs/manual/source/customize/dase.html.md.erb | 8 +-
docs/manual/source/customize/index.html.md | 6 +-
.../source/customize/troubleshooting.html.md | 12 +-
.../datacollection/analytics-ipynb.html.md.erb | 2 +-
.../analytics-tableau.html.md.erb | 4 +-
.../analytics-zeppelin.html.md.erb | 6 +-
.../source/datacollection/batchimport.html.md | 6 +-
.../source/datacollection/channel.html.md.erb | 15 +-
.../source/datacollection/eventapi.html.md | 33 +-
docs/manual/source/datacollection/index.html.md | 5 +-
.../source/datacollection/webhooks.html.md.erb | 7 +-
docs/manual/source/demo/community.html.md | 10 +-
docs/manual/source/demo/index.html.md.erb | 6 +-
docs/manual/source/demo/tapster.html.md | 76 ++-
.../source/demo/textclassification.html.md | 663 ------------------
.../source/demo/textclassification.html.md.erb | 673 +++++++++++++++++++
docs/manual/source/deploy/index.html.md | 11 +-
docs/manual/source/index.html.md.erb | 35 +-
docs/manual/source/install/index.html.md.erb | 47 +-
.../source/install/install-linux.html.md.erb | 30 +-
.../install/install-sourcecode.html.md.erb | 19 +-
33 files changed, 1042 insertions(+), 966 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/config.rb
----------------------------------------------------------------------
diff --git a/docs/manual/config.rb b/docs/manual/config.rb
index e9cbe75..c115891 100644
--- a/docs/manual/config.rb
+++ b/docs/manual/config.rb
@@ -28,7 +28,7 @@ set :markdown,
sprockets.append_path File.join root, 'bower_components'
# Sitemap
-set :url_root, 'https://docs.prediction.io'
+set :url_root, '//predictionio.incubator.apache.org'
activate :search_engine_sitemap, exclude_attr: 'hidden'
# Development Settings
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/data/nav/main.yml
----------------------------------------------------------------------
diff --git a/docs/manual/data/nav/main.yml b/docs/manual/data/nav/main.yml
index c9f4485..f9bd0fc 100644
--- a/docs/manual/data/nav/main.yml
+++ b/docs/manual/data/nav/main.yml
@@ -1,10 +1,10 @@
root:
-
- body: 'PredictionIO Docs'
+ body: 'Apache PredictionIO (incubating) Documentation'
url: '/'
children:
-
- body: 'Welcome to PredictionIO'
+ body: 'Welcome to Apache PredictionIO (incubating)'
url: '/'
-
body: 'Getting Started'
@@ -14,7 +14,7 @@ root:
body: 'A Quick Intro'
url: '/start/'
-
- body: 'Installing PredictionIO'
+ body: 'Installing Apache PredictionIO (incubating)'
url: '/install/'
-
body: 'Downloading an Engine Template'
@@ -154,9 +154,6 @@ root:
body: 'Architecture Overview'
url: '/system/'
-
- body: 'Production Deployment with AWS CloudFormation'
- url: '/system/deploy-cloudformation/'
- -
body: 'Using Another Data Store'
url: '/system/anotherdatastore/'
-
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/helpers/application_helpers.rb
----------------------------------------------------------------------
diff --git a/docs/manual/helpers/application_helpers.rb b/docs/manual/helpers/application_helpers.rb
index 07e7775..a20ea54 100644
--- a/docs/manual/helpers/application_helpers.rb
+++ b/docs/manual/helpers/application_helpers.rb
@@ -19,7 +19,7 @@ module ApplicationHelpers
end
def github_url
- base = 'https://github.com/PredictionIO/PredictionIO/tree/livedoc/docs/manual'
+ base = 'https://github.com/apache/incubator-predictionio/tree/livedoc/docs/manual'
path = current_page.source_file.sub(Middleman::Application.root_path.to_s, '')
base + path
end
@@ -54,7 +54,7 @@ module ApplicationHelpers
return current_page.data.title
end
else
- return "Welcome to PredictionIO Docs!"
+ return "Welcome to Apache PredictionIO (incubating) Documentation!"
end
end
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/algorithm/index.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/algorithm/index.html.md b/docs/manual/source/algorithm/index.html.md
index 7d09d9d..0ef9242 100644
--- a/docs/manual/source/algorithm/index.html.md
+++ b/docs/manual/source/algorithm/index.html.md
@@ -2,7 +2,10 @@
title: Built-in Algorithm Libraries
---
-An engine can virtually call any algorithm in the Algorithm class. PredictionIO currently offers native support to [Spark MLlib](http://spark.apache.org/docs/latest/mllib-guide.html) machine learning library.
-It is being used by some of the engine templates in the [template gallery](http://templates.prediction.io/).
+An engine can virtually call any algorithm in the Algorithm class. Apache
+PredictionIO (incubating) currently offers native support to [Spark
+MLlib](http://spark.apache.org/docs/latest/mllib-guide.html) machine learning
+library. It is being used by some of the engine templates in the [template
+gallery](http://templates.prediction.io/).
-More library support will be added soon.
\ No newline at end of file
+More library support will be added soon.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/appintegration/index.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/appintegration/index.html.md b/docs/manual/source/appintegration/index.html.md
index 8d9b08e..510ca62 100644
--- a/docs/manual/source/appintegration/index.html.md
+++ b/docs/manual/source/appintegration/index.html.md
@@ -2,20 +2,26 @@
title: App Integration Overview
---
-PredictionIO is designed as a machine learning server that integrates with your applications on production environments.
+Apache PredictionIO (incubating) is designed as a machine learning server that
+integrates with your applications on production environments.
A web or mobile app normally:
-1. Send event data to PredictionIO's Event Server for model training
-2. Send dynamic queries to deployed engine(s) to retrieve predicted results
+1. Send event data to Apache PredictionIO (incubating)'s Event Server for model
+ training
+2. Send dynamic queries to deployed engine(s) to retrieve predicted results
-
+
## Sending Event Data
-PredictionIO's Event Server receives event data from your application. The data can be used by engines as training data to build preditive models.
+Apache PredictionIO (incubating)'s Event Server receives event data from your
+application. The data can be used by engines as training data to build preditive
+models.
-Event Server listens to port 7070 by default. You can change the port with the [--port arg](/cli/#event-server-commands) when you launch the Event Server.
+Event Server listens to port 7070 by default. You can change the port with the
+[--port arg](/cli/#event-server-commands) when you launch the Event Server.
For further information, please read:
@@ -24,8 +30,10 @@ For further information, please read:
## Sending Query
-After you deploy an engine as a web service,
-it will wait for queries from your application and return predicted results in JSON format. An engine listens to port 8000 by default. If you want to deploy multiple engines, you can specific a different port for each of them.
+After you deploy an engine as a web service, it will wait for queries from your
+application and return predicted results in JSON format. An engine listens to
+port 8000 by default. If you want to deploy multiple engines, you can specific a
+different port for each of them.
For further information, please read:
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/cli/index.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/cli/index.html.md b/docs/manual/source/cli/index.html.md
index 0dbe777..9394807 100644
--- a/docs/manual/source/cli/index.html.md
+++ b/docs/manual/source/cli/index.html.md
@@ -4,17 +4,20 @@ title: Command Line
##Overview
-Interaction with PredictionIO is done through the command line interface. It follows the format of:
+Interaction with Apache PredictionIO (incubating) is done through the command
+line interface. It follows the format of:
```pio <command> [options] <args>...```
-You can run ```pio help``` to see a list of all available commands and ```pio help <command>``` to see details of the command.
+You can run ```pio help``` to see a list of all available commands and ```pio
+help <command>``` to see details of the command.
-PredictionIO commands can be separated into the following three categories.
+Apache PredictionIO (incubating) commands can be separated into the following
+three categories.
##General Commands
```pio help``` Display usage summary. `pio help <command>` to read about a specific subcommand.
-
+
```pio version``` Displays the version of the installed PredictionIO.
```pio status``` Displays install path and running status of PredictionIO system and its dependencies.
@@ -24,14 +27,14 @@ PredictionIO commands can be separated into the following three categories.
```pio eventserver``` Launch the Event Server.
-```pio app``` Manage apps that are used by the Event Server.
+```pio app``` Manage apps that are used by the Event Server.
+
+```pio app data-delete <name>``` deletes all data associated with the app.
+
+```pio app delete <name>``` deletes the app and its data.
-```pio app data-delete <name>``` deletes all data associated with the app.
-
-```pio app delete <name>``` deletes the app and its data.
+ ```--ip <value>``` IP to bind to. Default to localhost.
- ```--ip <value>``` IP to bind to. Default to localhost.
-
```--port <value>``` Port to bind to. Default to 7070.
@@ -39,22 +42,12 @@ PredictionIO commands can be separated into the following three categories.
##Engine Commands
-Engine commands need to be run from the directory that contains the engine project. ```--debug``` and ```--verbose``` flags will provide debug and third-party informational messages.
+Engine commands need to be run from the directory that contains the engine
+project. ```--debug``` and ```--verbose``` flags will provide debug and
+third-party informational messages.
-```pio build``` Build the engine at the current directory.
+```pio build``` Build the engine at the current directory.
```pio train``` Kick off a training using an engine.
-```pio deploy``` Deploy an engine as an engine server. If no instance ID is specified, it will deploy the latest instance.
-
-
-
-
-
-
-
-
-
-
-
-
+```pio deploy``` Deploy an engine as an engine server. If no instance ID is specified, it will deploy the latest instance.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/community/contribute-code.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/community/contribute-code.html.md b/docs/manual/source/community/contribute-code.html.md
index a9e8767..902091e 100644
--- a/docs/manual/source/community/contribute-code.html.md
+++ b/docs/manual/source/community/contribute-code.html.md
@@ -2,8 +2,13 @@
title: Contribute Code
---
-Thank you for your interest in contributing to PredictionIO. Our mission is to
-enable developers to build scalable machine learning applications easily. Here is how you can help with the project development. If you have questions at anytime, please free to post on the [Developer Fourm](https://groups.google.com/forum/#!forum/predictionio-dev) or email us at support@prediction.io.
+Thank you for your interest in contributing to Apache PredictionIO (incubating).
+Our mission is to enable developers to build scalable machine learning
+applications easily. Here is how you can help with the project development. If
+you have any question regarding development at anytime, please free to
+[subscribe](mailto:dev-subscribe@predictionio.incubator.apache.org) and post to
+the [Development Mailing
+List](mailto:dev-subscribe@predictionio.incubator.apache.org).
## Areas in Need of Help
@@ -20,83 +25,78 @@ show features that are highly sought after by the community.
## How to Report an Issue
-If you wish to report an issue you found, you can do so on [GitHub Issues](https://github.com/PredictionIO/PredictionIO/issues)
-or [PredictionIO JIRA](https://predictionio.atlassian.net).
+If you wish to report an issue you found, you can do so on [Apache PredictionIO
+(incubating) JIRA](https://issues.apache.org/jira/browse/PIO).
## How to Help Resolve Existing Issues
In general, bug fixes should be done the same way as new features, but critical
bug fixes will follow a different path.
-#### Critical Bug Fixes Only
-
-1. File an issue against the issue tracker if there isn't one already.
-2. Create a new hotfix branch described by the *git flow* methodology, and write
- the fix in that branch.
-3. Verify the patch and issue a pull request to the main repository.
-4. Once merged to the main repository, critical bug fixes will be merged to the
- "master" branch and new binary will be built and distributed.
-
-
## How to Add / Propose a New Feature
-1. To propose a new feature, simply post your proposal to [PredictionIO
- Development Google Group]
- (https://groups.google.com/forum/#!forum/predictionio-dev) or email us directly at support@prediction.io.
+1. To propose a new feature, simply
+ [subscribe](mailto:dev-subscribe@predictionio.incubator.apache.org) and post
+ your proposal to [Apache PredictionIO (incubating) Development Mailing List]
+ (mailto:dev@predictionio.incubator.apache.org).
2. Discuss with the community and the core development team on what needs to be
done, and lay down concrete plans on deliverables.
3. Once solid plans are made, start creating tickets in the [issue tracker]
- (https://predictionio.atlassian.net/secure/RapidBoard.jspa?rapidView=1).
-4. Work side by side with other developers using PredictionIO Development Google
- Group as primary mode of communication. You never know if someone else has a
- better idea. ;)
-
+ (https://issues.apache.org/jira/browse/PIO).
+4. Work side by side with other developers using Apache PredictionIO
+ (incubating) Development Mailing List as primary mode of communication. You
+ never know if someone else has a better idea. ;)
+
## How to Issue a Pull Request
When you have finished your code, you can [create a pull
request](https://help.github.com/articles/creating-a-pull-request/) against the
-**develop** branch. You also need to complete the [Contributor Agreement](http://prediction.io/cla). We cannot accept your PR without the agreement.
-
+**develop** branch.
+
+- The title must contain a tag associating with an existing JIRA ticket. You
+ must create a ticket so that the infrastructure can correctly track issues
+ across Apache JIRA and GitHub. If your ticket is `PIO-789`, your title must
+ look something like `[PIO-789] Some short description`.
+- Please also, in your commit message summary, include the JIRA ticket number
+ similar to above.
- Make sure the title and description are clear and concise. For more details on
writing a good commit message, check out [this
guide](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
- If the change is visual, make sure to include a screenshot or GIF.
-- If the PR closes an issue, make sure to put *Closes #X* at the end of the
- description on a new line.
- Make sure it is being opened into the right branch.
- Make sure it has been rebased on top of that branch.
-NOTE: When it is close to a release, and if there are major development ongoing, a
-release branch will be forked from the develop branch to stabilize the code for
-binary release. Please refer to the *git flow* methodology page for more
+NOTE: When it is close to a release, and if there are major development ongoing,
+a release branch will be forked from the develop branch to stabilize the code
+for binary release. Please refer to the *git flow* methodology page for more
information.
## Getting Started
-PredictionIO relies heavily on the [git flow methodology](
+Apache PredictionIO (incubating) relies heavily on the [git flow methodology](
http://nvie.com/posts/a-successful-git-branching-model/). Please make sure you
read and understand it before you start your development. By default, cloning
-PredictionIO will put you in the *develop* branch, which in most cases is where
-all the latest development go to.
+Apache PredictionIO (incubating) will put you in the *develop* branch, which in
+most cases is where all the latest development go to.
NOTE: For core development, please follow the [Scala Style Guide](http://docs.scala-lang.org/style/).
-### Create a Clone of PredictionIO\u2019s Repository
+### Create a Fork of the Apache PredictionIO (incubating) Repository
1. Start by creating a GitHub account if you do not already have one.
-2. Go to [PredictionIO\u2019s
- repository](https://github.com/PredictionIO/PredictionIO) and fork it to your
- own account.
+2. Go to [Apache PredictionIO (incubating)\u2019s GitHub
+ mirror](https://github.com/PredictionIO/PredictionIO) and fork it to your own
+ account.
3. Clone your fork to your local machine.
If you need additional help, please refer to
https://help.github.com/articles/fork-a-repo/.
-### Building PredictionIO from Source
+### Building Apache PredictionIO (incubating) from Source
-After the previous section, you should have a copy of PredictionIO in your local
-machine ready to be built.
+After the previous section, you should have a copy of Apache PredictionIO
+(incubating) in your local machine ready to be built.
1. Make sure you are on the *develop* branch. You can double check by `git
status` or simply `git checkout develop`.
@@ -105,15 +105,16 @@ machine ready to be built.
### Setting Up the Environment
-PredictionIO relies on 3rd party software to perform its tasks. To set them up,
-simply follow this [documentation](
+Apache PredictionIO (incubating) relies on 3rd party software to perform its
+tasks. To set them up, simply follow this [documentation](
http://docs.prediction.io/install/install-sourcecode/#installing-dependencies).
### Start Hacking
-You should have a PredictionIO development environment by now. Happy hacking!
+You should have a Apache PredictionIO (incubating) development environment by
+now. Happy hacking!
-## Anatomy of PredictionIO\u2019s Code Tree
+## Anatomy of Apache PredictionIO (incubating) Code Tree
The following describes each directory\u2019s purpose.
@@ -128,18 +129,18 @@ Configuration files that are used by both a source tree and binary distribution.
### core
-Core PredictionIO code that provides the DASE controller API, core data
-structures, and workflow creation and management code.
+Core Apache PredictionIO (incubating) code that provides the DASE controller
+API, core data structures, and workflow creation and management code.
### data
-PredictionIO Event Server, and backend-agnostic storage layer for event store
-and metadata store.
+Apache PredictionIO (incubating) Event Server, and backend-agnostic storage
+layer for event store and metadata store.
### docs
-Source code for docs.prediction.io site, and any other documentation support
-files.
+Source code for http://predictionio.incubator.apache.org site, and any other
+documentation support files.
### engines
@@ -147,7 +148,7 @@ Obsolete built-in engines. To be removed.
### examples
-Complete code examples showing PredictionIO\u2019s application.
+Complete code examples showing Apache PredictionIO (incubating)'s application.
### sbt
@@ -159,5 +160,6 @@ Starting point of building your custom engine.
### tools
-Tools for running PredictionIO. Contains primarily the CLI (command-line
-interface) and its supporting code, and the experimental evaluation dashboard.
+Tools for running Apache PredictionIO (incubating). Contains primarily the CLI
+(command-line interface) and its supporting code, and the experimental
+evaluation dashboard.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/community/contribute-documentation.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/community/contribute-documentation.html.md b/docs/manual/source/community/contribute-documentation.html.md
index 86dcfa8..fe97ffd 100644
--- a/docs/manual/source/community/contribute-documentation.html.md
+++ b/docs/manual/source/community/contribute-documentation.html.md
@@ -4,32 +4,30 @@ title: Contribute Documentation
## How to Write Documentation
-You can help improve the PredictionIO documentation by submitting tutorials,
-writing how-tos, fixing errors, and adding missing information. You can edit any page
-live on [GitHub](https://github.com/PredictionIO/PredictionIO) by clicking the pencil icon on any page or open a
-[Pull Request](https://help.github.com/articles/creating-a-pull-request/).
-
-<a href="https://gemnasium.com/PredictionIO/PredictionIO"><img src="https://gemnasium.com/PredictionIO/PredictionIO.svg" alt="Dependency Status" class="static" /></a>
-
+You can help improve the Apache PredictionIO (incubating) documentation by
+submitting tutorials, writing how-tos, fixing errors, and adding missing
+information. You can edit any page live on
+[GitHub](https://github.com/apache/incubator-predictionio) by clicking the
+pencil icon on any page or open a [Pull
+Request](https://help.github.com/articles/creating-a-pull-request/).
## Branching
Use the `livedoc` branch if you want to update the current documentation.
-<a href="https://travis-ci.org/PredictionIO/PredictionIO"><img src="https://travis-ci.org/PredictionIO/PredictionIO.svg?branch=livedoc" alt="Build Status" class="static" /></a>
-
-Use the `develop` branch if you want to write documentation for the next release.
-
-<a href="https://travis-ci.org/PredictionIO/PredictionIO"><img src="https://travis-ci.org/PredictionIO/PredictionIO.svg?branch=develop" alt="Build Status" class="static" /></a>
+Use the `develop` branch if you want to write documentation for the next
+release.
## Installing Locally
-PredictionIO documentation uses [Middleman](http://middlemanapp.com/) and is hosted on [S3](http://aws.amazon.com/s3/).
+Apache PredictionIO (incubating) documentation uses
+[Middleman](http://middlemanapp.com/) and is hosted on Apache.
-[Gems](http://rubygems.org/) are managed with [Bundler](http://bundler.io/). Front end code with [Bower](http://bower.io/).
+[Gems](http://rubygems.org/) are managed with [Bundler](http://bundler.io/).
+Front end code with [Bower](http://bower.io/).
-Requires [Ruby](https://www.ruby-lang.org/en/) 2.1 or greater.
-We recommend [RVM](http://rvm.io/) or [rbenv](https://github.com/sstephenson/rbenv).
+Requires [Ruby](https://www.ruby-lang.org/en/) 2.1 or greater. We recommend
+[RVM](http://rvm.io/) or [rbenv](https://github.com/sstephenson/rbenv).
WARNING: **OS X** users you will need to install [Xcode Command Line Tools](https://developer.apple.com/xcode/downloads/)
with: `$ xcode-select --install` first.
@@ -89,8 +87,9 @@ Internal links:
* Should end with / (S3 requirement).
* Should **not** end with .html.
-Following these rules helps keep everything consistent and allows our version parser to correctly version links.
-Middleman is configured for directory indexes. Linking to a file in `sources/samples/index.html` should be done with
+Following these rules helps keep everything consistent and allows our version
+parser to correctly version links. Middleman is configured for directory
+indexes. Linking to a file in `sources/samples/index.html` should be done with
`[Title](/sample/)`.
```md
@@ -170,9 +169,11 @@ NOTE: Files must end with a `.erb` extension to be processed as ERB.
## Going Live
-Pushing to the `livedoc` branch will update [docs.prediction.io](http://docs.prediction.io/) in about 5 minutes.
+Pushing to the `livedoc` branch will update
+http://predictionio.incubator.apache.org in about 5 minutes.
-You can check the progress of each build on [Travis CI](https://travis-ci.org/PredictionIO/PredictionIO).
+You can check the progress of each build on [Apache's
+Jenkins](https://builds.apache.org/).
```
$ git push origin livedoc
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/community/contribute-sdk.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/community/contribute-sdk.html.md b/docs/manual/source/community/contribute-sdk.html.md
index f88c966..1cc0357 100644
--- a/docs/manual/source/community/contribute-sdk.html.md
+++ b/docs/manual/source/community/contribute-sdk.html.md
@@ -2,28 +2,28 @@
title: Contribute a SDK
---
-A SDK should provide convenient methods for client applications to easily
-record users' behaviors in PredictionIO's Event Server and also query
-recommendations from the ML Engines. Therefore, a SDK typically has 2
+A SDK should provide convenient methods for client applications to easily
+record users' behaviors in Apache PredictionIO (incubating)'s Event Server and
+also query recommendations from the ML Engines. Therefore, a SDK typically has 2
corresponding clients: `Event Client` and `Engine Client`.
-The following guideline bases on the REST API provided by PredictionIO's
-Event Client which details can be found [here](http://docs.prediction.io/datacollection/eventapi/).
-
+The following guideline bases on the REST API provided by Apache PredictionIO
+(incubating)'s Event Client which details can be found
+[here](http://docs.prediction.io/datacollection/eventapi/).
## Event Client
Because the Event Server has only 1 connection point, the `Event Client`
-needs to implement this core request first. The core request has the
+needs to implement this core request first. The core request has the
following rules.
-- **URL**: `<base URL>/events.json?accessKey=<your access key>`
+- **URL**: `<base URL>/events.json?accessKey=<your access key>`
(e.g. http://localhost:7070/events.json?accessKey=1234567890)
- **Request**: `POST` + JSON data. Please refer to the [Event Creation API]
-(http://docs.prediction.io/datacollection/eventapi/) for the details
+(http://docs.prediction.io/datacollection/eventapi/) for the details
on the fields of the JSON data object.
-- **Response**:
+- **Response**:
+ **Success**: status code `201` with a JSON result containing
the `eventId`.
+ **Failure**: a JSON result containing a `message` field describing
@@ -32,8 +32,8 @@ on the fields of the JSON data object.
* Status code `400`: fail to parse the JSON request e.g. missing
required fields like `event`, or invalid `eventTime` format.
-Other convenient methods are just shortcut. They could simply build
-the event's parameters and call the core request. `Event Client` should
+Other convenient methods are just shortcut. They could simply build
+the event's parameters and call the core request. `Event Client` should
support the following 7 shorthand operations:
- **User entities**
@@ -127,8 +127,8 @@ sending the event to the server.
## Engine Client
-`Engine Client`'s main job is to retrieve recommendation or prediction
-results from PredictionIO's Engines. It has only a few rules on the
+`Engine Client`'s main job is to retrieve recommendation or prediction results
+from Apache PredictionIO (incubating)'s Engines. It has only a few rules on the
request and response type.
- **URL**: `<base URL>/queries.json` (e.g. http://localhost:8000/queries.json)
@@ -162,10 +162,10 @@ request and response type.
```
+ **Failure**: status code `400` e.g. fail to parse the query.
-The formats of JSON objects in both the request and response must be defined
-by the PredictionIO's Engine and are different across applications. The above
-examples are taken from the Recommendation Engine template in which the query
-and prediction results are defined as following.
+The formats of JSON objects in both the request and response must be defined by
+the Apache PredictionIO (incubating)'s Engine and are different across
+applications. The above examples are taken from the Recommendation Engine
+template in which the query and prediction results are defined as following.
```scala
case class Query(
@@ -179,10 +179,11 @@ case class PredictedResult(
```
## Testing Your SDK
-You can set up a local host PredictionIO environment to test your SDK.
-However, it is hard to set it up online to test your SDK automatically
-using services like Travis CI. In that case, you should consider
-using these lightweight [mock servers]
+
+You can set up a local host Apache PredictionIO (incubating) environment to test
+your SDK. However, it is hard to set it up online to test your SDK automatically
+using services like Travis CI. In that case, you should consider using these
+lightweight [mock servers]
(https://github.com/minhtule/PredictionIO-Mock-Server). Please see the
instructions in the repo how to use it. It takes less than 5 minutes!
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/community/contribute-webhook.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/community/contribute-webhook.html.md b/docs/manual/source/community/contribute-webhook.html.md
index 5919cc4..2876f01 100644
--- a/docs/manual/source/community/contribute-webhook.html.md
+++ b/docs/manual/source/community/contribute-webhook.html.md
@@ -2,7 +2,8 @@
title: Contribute a Webhooks Connector
---
-NOTE: Please check out the [latest develop branch](https://github.com/PredictionIO/PredictionIO).
+NOTE: Please check out the [latest develop
+branch](https://github.com/apache/incubator-predictionio).
Event server can collect data from other third-party sites or software through their webhooks services (for example, SegmentIO, MailChimp). To support that, a *Webhooks Connector* for the third-party data is needed to be integrated into Event Server. The job of the *Webhooks Connector* is as simply as converting the third-party data into Event JSON. You can find an example below.
@@ -161,7 +162,10 @@ private[prediction] object ExampleJsonConnector extends JsonConnector {
}
```
-You can find the complete example in [the GitHub repo](https://github.com/PredictionIO/PredictionIO/blob/develop/data/src/main/scala/io/prediction/data/webhooks/examplejson/ExampleJsonConnector.scala) and how to write [tests for the connector](https://github.com/PredictionIO/PredictionIO/blob/develop/data/src/test/scala/io/prediction/data/webhooks/examplejson/ExampleJsonConnectorSpec.scala).
+You can find the complete example in [the GitHub
+repo](https://github.com/apache/incubator-predictionio/blob/develop/data/src/main/scala/io/prediction/data/webhooks/examplejson/ExampleJsonConnector.scala)
+and how to write [tests for the
+connector](https://github.com/apache/incubator-predictionio/blob/develop/data/src/test/scala/io/prediction/data/webhooks/examplejson/ExampleJsonConnectorSpec.scala).
Please put the connector code in a separate directory for each site. For example, code for segmentio connector should be in
@@ -176,7 +180,10 @@ and tests should be in
data/src/test/scala/io/prediction/data/webhooks/segmentio/
```
-**For form-submission data**, you can find the comple example [the GitHub repo](https://github.com/PredictionIO/PredictionIO/blob/develop/data/src/main/scala/io/prediction/data/webhooks/exampleform/ExampleFormConnector.scala) and how to write [tests for the connector](https://github.com/PredictionIO/PredictionIO/blob/develop/data/src/test/scala/io/prediction/data/webhooks/exampleform/ExampleFormConnectorSpec.scala).
+**For form-submission data**, you can find the comple example [the GitHub
+repo](https://github.com/apache/incubator-predictionio/blob/develop/data/src/main/scala/io/prediction/data/webhooks/exampleform/ExampleFormConnector.scala)
+and how to write [tests for the
+connector](https://github.com/apache/incubator-predictionio/blob/develop/data/src/test/scala/io/prediction/data/webhooks/exampleform/ExampleFormConnectorSpec.scala).
## 2. Integrate the Connector into Event Server
@@ -184,7 +191,7 @@ data/src/test/scala/io/prediction/data/webhooks/segmentio/
Once we have the connector implemented, we can add this to the EventServer so we can collect real-time data.
Add the connector to [`WebhooksConnectors` object](
-https://github.com/PredictionIO/PredictionIO/blob/develop/data/src/main/scala/io/prediction/data/api/WebhooksConnectors.scala):
+https://github.com/apache/incubator-predictionio/blob/develop/data/src/main/scala/io/prediction/data/api/WebhooksConnectors.scala):
```scala
@@ -218,4 +225,6 @@ For `FormConnector`, the URL doesn't have `.json`. For example,
http://<EVENT SERVER URL>/webhooks/mailchimp?accessKey=<YOUR_ACCESS_KEY>&channel=<CHANNEL_NAME>
```
-That's it. Once you re-compile PredictionIO, you can send the ExampleJson data to the following URL and the data will be stored to the App of the corresponding Access Key.
+That's it. Once you re-compile Apache PredictionIO (incubating), you can send
+the ExampleJson data to the following URL and the data will be stored to the App
+of the corresponding Access Key.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/community/index.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/community/index.html.md b/docs/manual/source/community/index.html.md
index d988afd..e7b9c75 100644
--- a/docs/manual/source/community/index.html.md
+++ b/docs/manual/source/community/index.html.md
@@ -2,13 +2,15 @@
title: Community Page
---
-## User Forum
+## User Mailing List
-Join our [Google Group](https://groups.google.com/forum/#!forum/predictionio-user).
+This list is for users of Apache PredictionIO (incubating) to ask questions,
+share knowledge, and discuss issues. Do send mail to this list with usage and
+configuration questions and problems. Also, please send questions to this list
+to verify your problem before filing issues in JIRA.
-## Request a Feature
-
-Fill out the form [here](https://predictionio.uservoice.com/forums/219398-general/filters/top).
+[Subscribe](mailto:user-subscribe@predictionio.incubator.apache.org) to our User Mailing List.
+[Unsubscribe](mailto:user-unsubscribe@predictionio.incubator.apache.org) from our User Mailing List.
## Twitter
@@ -16,11 +18,11 @@ Follow us on Twitter [@predictionio](https://twitter.com/PredictionIO).
## Facebook Page
-Like us on Facebook at [https://www.facebook.com/predictionio](https://www.facebook.com/predictionio).
+Like us on Facebook at https://www.facebook.com/predictionio.
## GitHub
-View our code on GitHub at [https://github.com/PredictionIO](https://github.com/PredictionIO)
+View our code on GitHub at https://github.com/apache/incubator-predictionio.
-<iframe src="/github/?user=PredictionIO&repo=PredictionIO&type=fork&count=true&size=large" allowtransparency="true" frameborder="0" scrolling="0" width="170" height="30"></iframe>
-<iframe src="/github/?user=PredictionIO&repo=PredictionIO&type=watch&count=true&size=large" allowtransparency="true" frameborder="0" scrolling="0" width="170" height="30"></iframe>
+<iframe src="/github/?user=apache&repo=incubator-predictionio&type=fork&count=true&size=large" allowtransparency="true" frameborder="0" scrolling="0" width="170" height="30"></iframe>
+<iframe src="/github/?user=apache&repo=incubator-predictionio&type=watch&count=true&size=large" allowtransparency="true" frameborder="0" scrolling="0" width="170" height="30"></iframe>
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/community/projects.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/community/projects.html.md b/docs/manual/source/community/projects.html.md
index 29735d6..982e870 100644
--- a/docs/manual/source/community/projects.html.md
+++ b/docs/manual/source/community/projects.html.md
@@ -2,13 +2,13 @@
title: Community Powered Projects
---
-We are compiling a list of community-powered projects for PredictionIO 0.8.0 and above.
+Here you will find great projects contributed by the Apache PredictionIO
+(incubating) community. Stay tuned.
-Here you will find great projects contributed by the PredictionIO community.
-Stay tuned.
-
-INFO: If you have built a PredictionIO-related project, we would love to showcase it
-to the community! Simply edit [this page](https://github.com/PredictionIO/PredictionIO/blob/livedoc/docs/manual/source/community/projects.html.md) and submit a pull request.
+INFO: If you have built a Apache PredictionIO (incubating)-related project, we
+would love to showcase it to the community! Simply edit [this
+page](https://github.com/apache/incubator-predictionio/blob/livedoc/docs/manual/source/community/projects.html.md)
+and submit a pull request.
## SDKs
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/customize/dase.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/customize/dase.html.md.erb b/docs/manual/source/customize/dase.html.md.erb
index 78a0194..456a373 100644
--- a/docs/manual/source/customize/dase.html.md.erb
+++ b/docs/manual/source/customize/dase.html.md.erb
@@ -159,20 +159,22 @@ The two methods of the Algorithm class are train() and predict():
## train()
-train() is responsible for training a predictive model. It is called when you run `pio train`. PredictionIO will store this model.
+train() is responsible for training a predictive model. It is called when you
+run `pio train`. Apache PredictionIO (incubating) will store this model.
## predict()
predict() is responsible for using this model to make prediction. It is called when you send a JSON query to the engine. Note that predict() is called in real time.
-PredictionIO supports two types of algorithms:
+Apache PredictionIO (incubating) supports two types of algorithms:
- **[P2LAlgorithm](https://docs.prediction.io/api/current/#io.prediction.controller.P2LAlgorithm)**: trains a Model which does not contain RDD
- **[PAlgorithm](https://docs.prediction.io/api/current/#io.prediction.controller.PAlgorithm)**: trains a Model which contains RDD
## P2LAlgorithm
-For `P2LAlgorithm`, the Model is automatically serialized and persisted by PredictionIO after training.
+For `P2LAlgorithm`, the Model is automatically serialized and persisted by
+Apache PredictionIO (incubating) after training.
Implementing `IPersistentModel` and `IPersistentModelLoader` is optional for P2LAlgorithm.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/customize/index.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/customize/index.html.md b/docs/manual/source/customize/index.html.md
index c99ecde..00fdde0 100644
--- a/docs/manual/source/customize/index.html.md
+++ b/docs/manual/source/customize/index.html.md
@@ -27,9 +27,9 @@ customize the final returned results.
An Evaluation Metric quantifies prediction accuracy with a numerical score. It
can be used for comparing algorithms or algorithm parameter settings.
-> PredictionIO helps you modularize these components so you can build, for
-example, several Serving components for an Engine. You will be able to choose
-which one to be deployed when you create an Engine.
+> Apache PredictionIO (incubating) helps you modularize these components so you
+can build, for example, several Serving components for an Engine. You will be
+able to choose which one to be deployed when you create an Engine.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/customize/troubleshooting.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/customize/troubleshooting.html.md b/docs/manual/source/customize/troubleshooting.html.md
index 201a8de..2e9b130 100644
--- a/docs/manual/source/customize/troubleshooting.html.md
+++ b/docs/manual/source/customize/troubleshooting.html.md
@@ -2,7 +2,8 @@
title: Engine Development - Troubleshoot
---
-PredictionIO provides the following features to help you debug engines during development cycle.
+Apache PredictionIO (incubating) provides the following features to help you
+debug engines during development cycle.
## Stop Training between Stages
@@ -116,8 +117,9 @@ After run `pio deploy`, you can access the engine status page by go to same URL
## pio-shell
-PredictionIO also provides `pio-shell` in which you can easily access PredictionIO API,
-Spark context and Spark API for quickly testing code or debugging purposes.
+Apache PredictionIO (incubating) also provides `pio-shell` in which you can
+easily access Apache PredictionIO (incubating) API, Spark context and Spark API
+for quickly testing code or debugging purposes.
To bring up the shell, simply run:
@@ -125,7 +127,9 @@ To bring up the shell, simply run:
$ pio-shell --with-spark
```
-(`pio-shell` is available inside `bin/` directory of installed PredictionIO directory, you should be able to access it if you have added PredictionIO/bin into your environment variable `PATH`)
+(`pio-shell` is available inside `bin/` directory of installed Apache
+PredictionIO (incubating) directory, you should be able to access it if you have
+added PredictionIO/bin into your environment variable `PATH`)
Note that the Spark context is available as variable `sc` inside the shell.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/datacollection/analytics-ipynb.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/datacollection/analytics-ipynb.html.md.erb b/docs/manual/source/datacollection/analytics-ipynb.html.md.erb
index ee8aea2..9179e8c 100644
--- a/docs/manual/source/datacollection/analytics-ipynb.html.md.erb
+++ b/docs/manual/source/datacollection/analytics-ipynb.html.md.erb
@@ -4,7 +4,7 @@ title: Machine Learning Analytics with IPython Notebook
[IPython Notebook](http://ipython.org/notebook.html) is a very powerful
interactive computational environment, and with
-[PredictionIO](https://prediction.io),
+[Apache PredictionIO (incubating)](http://predictionio.incubator.apache.org),
[PySpark](http://spark.apache.org/docs/latest/api/python/) and [Spark
SQL](https://spark.apache.org/sql/), you can easily analyze your collected
events when you are developing or tuning your engine.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/datacollection/analytics-tableau.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/datacollection/analytics-tableau.html.md.erb b/docs/manual/source/datacollection/analytics-tableau.html.md.erb
index 0386c82..9d20365 100644
--- a/docs/manual/source/datacollection/analytics-tableau.html.md.erb
+++ b/docs/manual/source/datacollection/analytics-tableau.html.md.erb
@@ -2,8 +2,8 @@
title: Machine Learning Analytics with Tableau
---
-With Spark SQL, it is possible to connect Tableau to PredictionIO Event Server
-for interactive analysis of event data.
+With Spark SQL, it is possible to connect Tableau to Apache PredictionIO
+(incubating) Event Server for interactive analysis of event data.
## Prerequisites
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/datacollection/analytics-zeppelin.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/datacollection/analytics-zeppelin.html.md.erb b/docs/manual/source/datacollection/analytics-zeppelin.html.md.erb
index 94cb0ec..399d133 100644
--- a/docs/manual/source/datacollection/analytics-zeppelin.html.md.erb
+++ b/docs/manual/source/datacollection/analytics-zeppelin.html.md.erb
@@ -3,8 +3,8 @@ title: Machine Learning Analytics with Zeppelin
---
[Apache Zeppelin](http://zeppelin-project.org/) is an interactive computational
-environment built on Apache Spark like the IPython Notebook. With
-[PredictionIO](https://prediction.io) and [Spark
+environment built on Apache Spark like the IPython Notebook. With [Apache
+PredictionIO (incubating)](http://predictionio.incubator.apache.org) and [Spark
SQL](https://spark.apache.org/sql/), you can easily analyze your collected
events when you are developing or tuning your engine.
@@ -12,7 +12,7 @@ events when you are developing or tuning your engine.
The following instructions assume that you have the command `sbt` accessible in
your shell's search path. Alternatively, you can use the `sbt` command that
-comes with PredictionIO at `$PIO_HOME/sbt/sbt`.
+comes with Apache PredictionIO (incubating) at `$PIO_HOME/sbt/sbt`.
<%= partial 'shared/datacollection/parquet' %>
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/datacollection/batchimport.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/datacollection/batchimport.html.md b/docs/manual/source/datacollection/batchimport.html.md
index 9364d6b..ab31d41 100644
--- a/docs/manual/source/datacollection/batchimport.html.md
+++ b/docs/manual/source/datacollection/batchimport.html.md
@@ -29,7 +29,11 @@ import.
## Use SDK to Prepare Batch Input File
-Some of the PredictionIO SDKs also provides FileExporter client. You may use them to prepare the JSON file as described above. The FileExporter creates event in the same way as EventClient except that the events are written to a JSON file instead of being sent to EventSever. The written JSON file can then be used by batch import.
+Some of the Apache PredictionIO (incubating) SDKs also provides FileExporter
+client. You may use them to prepare the JSON file as described above. The
+FileExporter creates event in the same way as EventClient except that the events
+are written to a JSON file instead of being sent to EventSever. The written JSON
+file can then be used by batch import.
<div class="tabs">
<div data-tab="PHP SDK" data-lang="php">
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/datacollection/channel.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/datacollection/channel.html.md.erb b/docs/manual/source/datacollection/channel.html.md.erb
index af2a70c..92814a8 100644
--- a/docs/manual/source/datacollection/channel.html.md.erb
+++ b/docs/manual/source/datacollection/channel.html.md.erb
@@ -2,17 +2,24 @@
title: Channel
---
-Each App has a default channel (without name) which stores all incoming events. This "default" one is used when channel is not specified.
+Each App has a default channel (without name) which stores all incoming events.
+This "default" one is used when channel is not specified.
-You may create additional Channels for the App. Creating multiple Channels is advanced usage. You don't need to create any in order to use PredictionIO. The Channel is associated with one App only and must have unique name within the same App.
+You may create additional Channels for the App. Creating multiple Channels is
+advanced usage. You don't need to create any in order to use Apache PredictionIO
+(incubating). The Channel is associated with one App only and must have unique
+name within the same App.
-Creating multiple Channels allows you more easily to identify, manage and use specific event data if you may collect events from different multiple sources (eg. mobile, website, or third-party webhooks service) for the your application.
+Creating multiple Channels allows you more easily to identify, manage and use
+specific event data if you may collect events from different multiple sources
+(eg. mobile, website, or third-party webhooks service) for the your application.
(More usage details coming soon...)
## Create a new Channel
-For example, to create a new channel "myChannel" for app "myApp", run following `pio` command:
+For example, to create a new channel "myChannel" for app "myApp", run following
+`pio` command:
```
pio app channel-new myApp myChannel
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/datacollection/eventapi.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/datacollection/eventapi.html.md b/docs/manual/source/datacollection/eventapi.html.md
index 9fb3716..0e59c0a 100644
--- a/docs/manual/source/datacollection/eventapi.html.md
+++ b/docs/manual/source/datacollection/eventapi.html.md
@@ -2,28 +2,29 @@
title: Collecting Data through REST/SDKs
---
-**Event Server** is designed to collect data into PredictionIO in an event-based
-style. Once the Event Server is launched, your application can send data to it
-through its **Event API** with HTTP requests or with `EventClient`s of
-PredictionIO's SDKs.
-
-INFO: All PredictionIO-compliant engines support accessing the Event Store (i.e. the
-data store of Event Server) through [PredictionIO's Storage
+**Event Server** is designed to collect data into Apache PredictionIO
+(incubating) in an event-based style. Once the Event Server is launched, your
+application can send data to it through its **Event API** with HTTP requests or
+with `EventClient`s of PredictionIO's SDKs.
+
+INFO: All Apache PredictionIO (incubating)-compliant engines support accessing
+the Event Store (i.e. the data store of Event Server) through [Apache
+PredictionIO (incubating)'s Storage
API](http://docs.prediction.io/api/current/index.html#io.prediction.data.storage.package).
## Launching the Event Server
INFO: Before launching the Event Server, make sure that your event data store
-backend is properly configured and is running. By default, PredictionIO uses
-Apache HBase, and a quick configuration can be found
-[here](/install/install-linux.html#hbase). Please allow a minute
-(usually less than 30 seconds) after HBase is started for its initialization to
-complete before starting the Event Server.
+backend is properly configured and is running. By default, Apache PredictionIO
+(incubating) uses Apache HBase, and a quick configuration can be found
+[here](/install/install-linux.html#hbase). Please allow a minute (usually less
+than 30 seconds) after HBase is started for its initialization to complete
+before starting the Event Server.
-Everything about PredictionIO can be done through the `pio` command. Please add
-PIO binary command path to to your `PATH` first. Assuming PredictionIO is
-installed at `/home/yourname/PredictionIO/`, you can run
+Everything about Apache PredictionIO (incubating) can be done through the `pio`
+command. Please add PIO binary command path to to your `PATH` first. Assuming
+PredictionIO is installed at `/home/yourname/PredictionIO/`, you can run
```
$ PATH=$PATH:/home/yourname/PredictionIO/bin; export PATH
@@ -81,7 +82,7 @@ to use the Event API. You should see something like the following output:
### Creating Your First Event
You may connect to the Event Server with HTTP request or by using one of many
-**PredictionIO SDKs**.
+**Apache PredictionIO (incubating) SDKs**.
For example, the following shows how one can create an event involving a single entity.
Replace the value of `accessKey` by the *Access Key* generated for your App.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/datacollection/index.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/datacollection/index.html.md b/docs/manual/source/datacollection/index.html.md
index 55293fd..b304c3d 100644
--- a/docs/manual/source/datacollection/index.html.md
+++ b/docs/manual/source/datacollection/index.html.md
@@ -2,8 +2,9 @@
title: Event Server Overview
---
-PredictionIO offers an Event Server that collects data in an event-based style
-via a RESTful API. By default, Event Server uses Apache HBase as data store.
+Apache PredictionIO (incubating) offers an Event Server that collects data in an
+event-based style via a RESTful API. By default, Event Server uses Apache HBase
+as data store.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/datacollection/webhooks.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/datacollection/webhooks.html.md.erb b/docs/manual/source/datacollection/webhooks.html.md.erb
index df0f032..b5b7c4d 100644
--- a/docs/manual/source/datacollection/webhooks.html.md.erb
+++ b/docs/manual/source/datacollection/webhooks.html.md.erb
@@ -4,7 +4,6 @@ title: Unifying Multichannel Data with Webhooks
Webhooks are used to collects data for your application from multiple channels automatically.
-PredictionIO offers webhooks for Segment.io and MailChimp (backend only). Several users also have expressed interest in
-Magento, Shopify and KeenIO so we'd be happy to help if anyone wishes to implement these.
-
-Please contact us at <a href="mailto:support@prediction.io">support@prediction.io</a> if you are interested in trying out one of them.
+Apache PredictionIO (incubating) offers webhooks for Segment.io and MailChimp
+(backend only). Several users also have expressed interest in Magento, Shopify
+and KeenIO so we'd be happy to help if anyone wishes to implement these.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/demo/community.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/demo/community.html.md b/docs/manual/source/demo/community.html.md
index 6bb7a68..e1c1a0b 100644
--- a/docs/manual/source/demo/community.html.md
+++ b/docs/manual/source/demo/community.html.md
@@ -1,10 +1,12 @@
---
-title: Demo Tutorials with PredictionIO
+title: Demo Tutorials with Apache PredictionIO (incubating)
---
-Here is a list of demo tutorials for PredictionIO.
+Here is a list of demo tutorials for Apache PredictionIO (incubating).
-INFO: If you want to showcase your demo app here, simply edit [this page](https://github.com/PredictionIO/PredictionIO/blob/livedoc/docs/manual/source/community/showcase.html.md) and submit a pull request.
+INFO: If you want to showcase your demo app here, simply edit [this
+page](https://github.com/apache/incubator-predictionio/blob/livedoc/docs/manual/source/community/showcase.html.md)
+and submit a pull request.
## Tapster iOS Demo
@@ -37,7 +39,7 @@ URL: http://ontappapp.com/
Writeup: http://victorleungtw.com/prediction-io/
-## Yelpio
+## Yelpio
<img src="/images/showcase/yelpio-70x70.png" alt="OnTapp" class="static" />
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/demo/index.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/demo/index.html.md.erb b/docs/manual/source/demo/index.html.md.erb
index 76f4320..43da798 100644
--- a/docs/manual/source/demo/index.html.md.erb
+++ b/docs/manual/source/demo/index.html.md.erb
@@ -4,9 +4,11 @@ title: Demos
## Current Demos
-[Tapster](/demo/tapster/) - A Tinder like Rails application using PredictionIO.
+[Tapster](/demo/tapster/) - A Tinder like Rails application using Apache
+PredictionIO (incubating).
## Contribute
-Interested in writing a demo? Contact us at <%= mail_to 'support@prediction.io' %>.
+Interested in writing a demo? Contact us at <%= mail_to
+'dev@prediction.incubator.apache.org' %>.
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/5be3cca8/docs/manual/source/demo/tapster.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/demo/tapster.html.md b/docs/manual/source/demo/tapster.html.md
index efb0de6..4bf9917 100644
--- a/docs/manual/source/demo/tapster.html.md
+++ b/docs/manual/source/demo/tapster.html.md
@@ -66,20 +66,21 @@ $rails server
-## PredictionIO Setup
+## Apache PredictionIO (incubating) Setup
-### Install PredictionIO
-Follow the installation instructions [here](http://docs.prediction.io/install/) or simply run:
+### Install Apache PredictionIO (incubating)
+Follow the installation instructions [here](http://predictionio.incubator.apache.org/install/) or simply run:
```
-$ bash -c "$(curl -s https://install.prediction.io/install.sh)"
+$ bash -c "$(curl -s https://raw.githubusercontent.com/apache/incubator-predictionio/master/bin/install.sh)"
```
### Create a New App
-You will need to create a new app on PredictionIO to house the Tapster demo. You can do this with:
+You will need to create a new app on Apache PredictionIO (incubating) to house
+the Tapster demo. You can do this with:
```
$ pio app new tapster
@@ -172,7 +173,8 @@ Once everything is installed, start the event server by running: `$ pio eventser
-INFO: You can check the status of PredictionIO at any time by running: `$ pio status`
+INFO: You can check the status of Apache PredictionIO (incubating) at any time
+by running: `$ pio status`
ALERT: If your laptop goes to sleep you might manually need to restart HBase with:
@@ -182,9 +184,11 @@ $ ./stop-hbase.sh
$ ./start-hbase.sh
```
-The key event we are importing into PredictionIO event server is the "Like" event (for example, user X likes episode Y).
+The key event we are importing into Apache PredictionIO (incubating) event
+server is the "Like" event (for example, user X likes episode Y).
-We will send this data to PredictionIO by executing `$ rake import:predictionio` command.
+We will send this data to Apache PredictionIO (incubating) by executing `$ rake
+import:predictionio` command.
[View on GitHub](https://github.com/PredictionIO/Demo-Tapster/blob/master/lib/tasks/import/predictionio.rake)
@@ -200,7 +204,9 @@ INFO: If you forget your **Access Key** you can always run: `$ pio app list`
You can set these values in the `.env` file located in the application root directory and it will be automatically loaded into your environment each time Rails is run.
-The next part of the script loops through each line of the `data/user_list.csv` file and returns an array of unique user and episode IDs. Once we have those we can send the data to PredictionIO like this.
+The next part of the script loops through each line of the `data/user_list.csv`
+file and returns an array of unique user and episode IDs. Once we have those we
+can send the data to Apache PredictionIO (incubating) like this.
First the users:
@@ -253,7 +259,8 @@ CSV.foreach(USER_LIST, headers: true) do |row|
end
```
-In total the script takes about 4 minutes to run on a basic laptop. At this point all the data is now imported to PredictionIO.
+In total the script takes about 4 minutes to run on a basic laptop. At this
+point all the data is now imported to Apache PredictionIO (incubating).
@@ -268,7 +275,10 @@ $ pio train -- --driver-memory 4g
-Using the --driver-memory option to limit the memory used by PredictionIO. Without this PredictionIO can consume too much memory leading to a crash. You can adjust the 4g up or down depending on your system specs.
+Using the --driver-memory option to limit the memory used by Apache PredictionIO
+(incubating). Without this Apache PredictionIO (incubating) can consume too much
+memory leading to a crash. You can adjust the 4g up or down depending on your
+system specs.
You can set up a job to periodically retrain the engine so the model is updated with the latest dataset.
@@ -277,27 +287,33 @@ You can set up a job to periodically retrain the engine so the model is updated
You can deploy the model with: `$ pio deploy` from the `tapster-episode-similar` directory.
-At this point, you have an demo app with data and a PredictionIO server with a trained model all setup. Next, we will connect the two so you can log the live interaction (likes) events into PredictionIO event server and query the engine server for recommendation.
+At this point, you have an demo app with data and a Apache PredictionIO
+(incubating) server with a trained model all setup. Next, we will connect the
+two so you can log the live interaction (likes) events into Apache PredictionIO
+(incubating) event server and query the engine server for recommendation.
-## Connect Demo app with PredictionIO
+## Connect Demo app with Apache PredictionIO (incubating)
### Overview
-On a high level the application keeps a record of each like and dislike. It uses jQuery to send an array of both likes and dislikes to the server on each click. The server then queries PredictionIO for a similar episode which is relayed to jQuery and displayed to the user.
+On a high level the application keeps a record of each like and dislike. It uses jQuery to send an array of both likes and dislikes to the server on each click. The server then queries Apache PredictionIO (incubating) for a similar episode which is relayed to jQuery and displayed to the user.
Data flow:
- The user likes an episode.
-- Tapster sends the "Like" event to PredictionIO event server.
-- Tapster queries PredictionIO engine with all the episodes the user has rated (likes and dislikes) in this session.
-- PredictionIO returns 1 recommended episode.
+- Tapster sends the "Like" event to Apache PredictionIO (incubating) event
+ server.
+- Tapster queries Apache PredictionIO (incubating) engine with all the episodes
+ the user has rated (likes and dislikes) in this session.
+- Apache PredictionIO (incubating) returns 1 recommended episode.
### JavaScript
All the important code lives in `app/assets/javascripts/application.js` [View on GitHub](https://github.com/PredictionIO/Demo-Tapster/blob/master/app/assets/javascripts/application.js)
Most of this file is just handlers for click things, displaying the loading dialog and other such things.
-The most important function is to query the Rails server for results from PredictionIO.
+The most important function is to query the Rails server for results from Apache
+PredictionIO (incubating).
```
// Query the server for a comic based on previous likes. See episodes#query.
@@ -354,17 +370,21 @@ def query
end
```
-On the first line we make a connection to PredictionIO. You will need to set the `PIO_ENGINE_URL`. This can be done in the `.env` file. The default URL is: http://localhost:8000.
+On the first line we make a connection to Apache PredictionIO (incubating). You
+will need to set the `PIO_ENGINE_URL`. This can be done in the `.env` file. The
+default URL is: http://localhost:8000.
Next we decode the JSON sent from the browser.
After that we check to see if the user has liked anything yet. If not we just return a random episode.
-If the user has likes then we can send that data to PredictionIO event server.
+If the user has likes then we can send that data to Apache PredictionIO
+(incubating) event server.
We also blacklist the dislikes so that they are not returned.
-With our response from PredictionIO it\u2019s just a matter of looking it up in the database and rendering that object as JSON.
+With our response from Apache PredictionIO (incubating) it\u2019s just a matter of
+looking it up in the database and rendering that object as JSON.
Once the response is sent to the browser JavaScript is used to replace the existing comic and hide the loading message.
@@ -375,8 +395,16 @@ Source code is on GitHub at: [github.com/PredictionIO/Demo-Tapster](https://gith
## Conclusion
-Love this tutorial and PredictionIO? Both are open source (Apache 2 License). [Fork](https://github.com/PredictionIO/Demo-Tapster) this demo and build upon it. If you produce something cool shoot us an email and we will link to it from here.
-
-Found a typo? Think something should be explained better? This tutorial (and all our other documenation) live in the main repo [here](https://github.com/PredictionIO/PredictionIO/blob/livedoc/docs/manual/source/demo/tapster.html.md). Our documentation is in the `livedoc` branch. Find out how to contribute documentation at [docs.prediction.io/community/contribute-documentation/](http://docs.prediction.io/community/contribute-documentation/).
+Love this tutorial and Apache PredictionIO (incubating)? Both are open source
+(Apache 2 License). [Fork](https://github.com/PredictionIO/Demo-Tapster) this
+demo and build upon it. If you produce something cool shoot us an email and we
+will link to it from here.
+
+Found a typo? Think something should be explained better? This tutorial (and all
+our other documenation) live in the main repo
+[here](https://github.com/apache/incubator-predictionio/blob/livedoc/docs/manual/source/demo/tapster.html.md).
+Our documentation is in the `livedoc` branch. Find out how to contribute
+documentation at
+http://predictionio.incubator.apache.org/community/contribute-documentation/].
We ♥ pull requests!
[05/15] incubator-predictionio git commit: Merge branch 'master' into
livedoc
Posted by do...@apache.org.
Merge branch 'master' into livedoc
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/23765454
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/23765454
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/23765454
Branch: refs/heads/develop
Commit: 23765454f9f4e9c9e8540f94939882a16f145b1d
Parents: 9d49754 7cad71e
Author: Donald Szeto <do...@prediction.io>
Authored: Mon Apr 11 16:00:47 2016 -0700
Committer: Donald Szeto <do...@prediction.io>
Committed: Mon Apr 11 16:00:47 2016 -0700
----------------------------------------------------------------------
.travis.yml | 83 +++++----
RELEASE.md | 10 +-
bin/compute-classpath.sh | 7 +
bin/install.sh | 49 ++----
build.sbt | 22 +--
common/build.sbt | 5 +
common/src/main/resources/application.conf | 11 ++
.../authentication/KeyAuthentication.scala | 55 ++++++
.../configuration/SSLConfiguration.scala | 71 ++++++++
conf/keystore.jks | Bin 0 -> 2238 bytes
conf/pio-env.sh.template | 3 +
conf/server.conf | 13 ++
core/build.sbt | 16 +-
core/src/main/resources/application.conf | 4 -
.../io/prediction/controller/Deployment.scala | 2 +-
.../scala/io/prediction/controller/Engine.scala | 5 +-
.../io/prediction/controller/Evaluation.scala | 10 +-
.../prediction/controller/FastEvalEngine.scala | 38 ++--
.../io/prediction/controller/LServing.scala | 2 +-
.../scala/io/prediction/controller/Metric.scala | 14 +-
.../prediction/controller/MetricEvaluator.scala | 14 +-
.../io/prediction/controller/package.scala | 2 +-
.../io/prediction/core/BaseAlgorithm.scala | 2 +-
.../scala/io/prediction/core/BaseEngine.scala | 8 +-
.../io/prediction/core/BaseEvaluator.scala | 6 +-
.../io/prediction/workflow/CoreWorkflow.scala | 4 +-
.../io/prediction/workflow/CreateServer.scala | 58 ++++--
.../io/prediction/workflow/FakeWorkflow.scala | 26 +--
.../io/prediction/workflow/WorkflowUtils.scala | 6 +-
.../io/prediction/controller/EngineTest.scala | 106 +++++------
data/build.sbt | 20 +--
.../io/prediction/data/api/EventServer.scala | 95 ++++++----
.../scala/io/prediction/data/api/Webhooks.scala | 2 +-
.../io/prediction/data/storage/AccessKeys.scala | 12 +-
.../io/prediction/data/storage/BiMap.scala | 2 +-
.../data/storage/elasticsearch/ESChannels.scala | 2 +-
.../data/storage/hbase/PIOHBaseUtil.scala | 2 +-
.../data/storage/jdbc/JDBCEngineInstances.scala | 4 +-
.../storage/jdbc/JDBCEvaluationInstances.scala | 4 +-
.../data/storage/jdbc/JDBCLEvents.scala | 42 ++++-
.../data/storage/jdbc/JDBCPEvents.scala | 82 +++++----
.../data/storage/jdbc/StorageClient.scala | 7 +-
.../io/prediction/data/store/LEventStore.scala | 2 +-
.../webhooks/mailchimp/MailChimpConnector.scala | 22 +--
.../webhooks/segmentio/SegmentIOConnector.scala | 71 +++++---
.../prediction/data/api/EventServiceSpec.scala | 16 +-
.../prediction/data/api/SegmentIOAuthSpec.scala | 175 +++++++++++++++++++
.../segmentio/SegmentIOConnectorSpec.scala | 42 ++---
data/test-segmentio.sh | 91 ++++++++--
docs/manual/data/versions.yml | 4 +-
docs/manual/source/deploy/index.html.md | 2 +
.../evaluation/evaluationdashboard.html.md | 4 +
.../source/system/anotherdatastore.html.md | 12 ++
e2/build.sbt | 2 +-
.../prediction/e2/engine/BinaryVectorizer.scala | 3 +-
.../scala-local-regression/Run.scala | 3 +-
.../src/main/scala/DataSource.scala | 2 +-
.../src/main/scala/Engine.scala | 6 +-
.../src/main/scala/Engine.scala | 6 +-
.../src/main/scala/Serving.scala | 3 +-
.../src/main/scala/Engine.scala | 6 +-
.../src/main/scala/Serving.scala | 3 +-
.../src/main/scala/Engine.scala | 6 +-
.../src/main/scala/Engine.scala | 6 +-
.../src/main/scala/Serving.scala | 3 +-
.../src/main/scala/Engine.scala | 6 +-
.../src/main/scala/Serving.scala | 3 +-
.../src/main/scala/Engine.scala | 6 +-
.../src/main/scala/Serving.scala | 3 +-
.../src/main/scala/Engine.scala | 6 +-
.../src/main/scala/Serving.scala | 3 +-
.../src/main/scala/BackTestingMetrics.scala | 28 +--
.../scala-stock/src/main/scala/Data.scala | 13 +-
.../src/main/scala/YahooDataSource.scala | 42 ++---
.../src/main/scala/Engine.scala | 6 +-
.../weighted-items/src/main/scala/Engine.scala | 6 +-
.../src/main/scala/Engine.scala | 6 +-
.../src/main/scala/Serving.scala | 3 +-
.../custom-query/src/main/scala/Engine.scala | 4 +-
.../custom-serving/src/main/scala/Engine.scala | 6 +-
.../src/main/scala/Preparator.scala | 6 +-
.../src/main/scala/DataSource.scala | 8 +-
.../src/main/scala/Engine.scala | 6 +-
.../src/main/scala/DataSource.scala | 2 +-
.../src/main/scala/Engine.scala | 6 +-
.../src/main/scala/DataSource.scala | 10 +-
.../add-rateevent/src/main/scala/Engine.scala | 6 +-
.../filterbyyear/src/main/scala/Engine.scala | 6 +-
.../filterbyyear/src/main/scala/Serving.scala | 3 +-
.../multi/src/main/scala/Engine.scala | 6 +-
.../multi/src/main/scala/Serving.scala | 3 +-
.../no-set-user/src/main/scala/Engine.scala | 6 +-
.../no-set-user/src/main/scala/Serving.scala | 3 +-
.../src/main/scala/Engine.scala | 6 +-
project/assembly.sbt | 2 +-
project/plugins.sbt | 4 +-
scalastyle-config.xml | 3 +
tools/build.sbt | 22 ++-
.../scala/io/prediction/tools/RunWorkflow.scala | 2 +-
.../prediction/tools/admin/CommandClient.scala | 2 +-
.../scala/io/prediction/tools/console/App.scala | 9 +-
.../io/prediction/tools/console/Console.scala | 19 +-
.../io/prediction/tools/console/Template.scala | 6 +-
.../tools/dashboard/CorsSupport.scala | 16 +-
.../prediction/tools/dashboard/Dashboard.scala | 41 +++--
.../prediction/tools/imprt/FileToEvents.scala | 12 +-
.../io/prediction/tools/console/app.scala.txt | 12 +-
107 files changed, 1182 insertions(+), 602 deletions(-)
----------------------------------------------------------------------
[03/15] incubator-predictionio git commit: Add back JavaScript runtime
Posted by do...@apache.org.
Add back JavaScript runtime
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/28000835
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/28000835
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/28000835
Branch: refs/heads/develop
Commit: 2800083577e12de254bd7968f06119b0c15a34d8
Parents: acd0e2f
Author: Donald Szeto <do...@prediction.io>
Authored: Mon Apr 11 10:33:35 2016 -0700
Committer: Donald Szeto <do...@prediction.io>
Committed: Mon Apr 11 10:33:35 2016 -0700
----------------------------------------------------------------------
docs/manual/Gemfile | 1 +
1 file changed, 1 insertion(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/28000835/docs/manual/Gemfile
----------------------------------------------------------------------
diff --git a/docs/manual/Gemfile b/docs/manual/Gemfile
index a0c588c..f4df6bc 100644
--- a/docs/manual/Gemfile
+++ b/docs/manual/Gemfile
@@ -9,6 +9,7 @@ gem 'middleman-s3_sync'
gem 'middleman-search_engine_sitemap'
gem 'slim'
+gem 'therubyracer'
gem 'oj'
gem 'redcarpet', '>= 3.2.3'
gem 'travis'
[13/15] incubator-predictionio git commit: Fix fonts
Posted by do...@apache.org.
Fix fonts
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/1ed9b156
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/1ed9b156
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/1ed9b156
Branch: refs/heads/develop
Commit: 1ed9b1564b0353fde9b4e5fadc94578e8a596419
Parents: 6cab260
Author: Donald Szeto <do...@apache.org>
Authored: Mon Jul 25 14:25:36 2016 -0700
Committer: Donald Szeto <do...@apache.org>
Committed: Mon Jul 25 14:25:36 2016 -0700
----------------------------------------------------------------------
docs/manual/source/stylesheets/variables/_fonts.css.scss | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/1ed9b156/docs/manual/source/stylesheets/variables/_fonts.css.scss
----------------------------------------------------------------------
diff --git a/docs/manual/source/stylesheets/variables/_fonts.css.scss b/docs/manual/source/stylesheets/variables/_fonts.css.scss
index 8ef537e..6e6c886 100644
--- a/docs/manual/source/stylesheets/variables/_fonts.css.scss
+++ b/docs/manual/source/stylesheets/variables/_fonts.css.scss
@@ -1,3 +1,3 @@
-$standard-font-family: "azo-sans-web", 'Helvetica Neue', Helvetica, Arial, sans-serif;
-$body-font-family: 'PT Sans', 'Helvetica Neue', Helvetica, Arial, sans-serif;
+$standard-font-family: "pt-sans", 'Helvetica Neue', Helvetica, Arial, sans-serif;
+$body-font-family: "pt-sans", 'Helvetica Neue', Helvetica, Arial, sans-serif;
$code-font-family: Menlo, Monaco, Consolas, monospace;
[15/15] incubator-predictionio git commit: Merge branch 'livedoc'
into develop
Posted by do...@apache.org.
Merge branch 'livedoc' into develop
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/0e7563f8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/0e7563f8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/0e7563f8
Branch: refs/heads/develop
Commit: 0e7563f83584c6bddff99a195cf831f8c380eaa6
Parents: 1e0d435 8fc24eb
Author: Donald Szeto <do...@apache.org>
Authored: Tue Jul 26 12:14:35 2016 -0700
Committer: Donald Szeto <do...@apache.org>
Committed: Tue Jul 26 12:14:35 2016 -0700
----------------------------------------------------------------------
docs/manual/Gemfile | 1 -
docs/manual/Gemfile.lock | 4 +-
docs/manual/config.rb | 2 +-
docs/manual/data/nav/main.yml | 17 +-
docs/manual/helpers/application_helpers.rb | 4 +-
docs/manual/source/algorithm/index.html.md | 9 +-
docs/manual/source/appintegration/index.html.md | 24 +-
docs/manual/source/cli/index.html.md | 43 +-
.../source/community/contribute-code.html.md | 106 +--
.../community/contribute-documentation.html.md | 41 +-
.../source/community/contribute-sdk.html.md | 45 +-
.../source/community/contribute-webhook.html.md | 19 +-
docs/manual/source/community/index.html.md | 20 +-
docs/manual/source/community/projects.html.md | 12 +-
docs/manual/source/customize/dase.html.md.erb | 8 +-
docs/manual/source/customize/index.html.md | 6 +-
.../source/customize/troubleshooting.html.md | 12 +-
.../datacollection/analytics-ipynb.html.md.erb | 2 +-
.../analytics-tableau.html.md.erb | 4 +-
.../analytics-zeppelin.html.md.erb | 6 +-
.../source/datacollection/batchimport.html.md | 6 +-
.../source/datacollection/channel.html.md.erb | 15 +-
.../source/datacollection/eventapi.html.md | 33 +-
docs/manual/source/datacollection/index.html.md | 5 +-
.../source/datacollection/webhooks.html.md.erb | 7 +-
docs/manual/source/demo/community.html.md | 10 +-
docs/manual/source/demo/index.html.md.erb | 5 +-
docs/manual/source/demo/tapster.html.md | 76 ++-
.../source/demo/textclassification.html.md | 663 ------------------
.../source/demo/textclassification.html.md.erb | 673 +++++++++++++++++++
docs/manual/source/deploy/index.html.md | 11 +-
docs/manual/source/deploy/monitoring.html.md | 154 +++++
docs/manual/source/images/monit.png | Bin 0 -> 192080 bytes
docs/manual/source/index.html.md.erb | 35 +-
docs/manual/source/install/index.html.md.erb | 47 +-
.../source/install/install-linux.html.md.erb | 30 +-
.../install/install-sourcecode.html.md.erb | 19 +-
docs/manual/source/layouts/layout.html.slim | 2 -
.../source/partials/_action_call.html.slim | 2 +-
docs/manual/source/partials/_adroll.html.slim | 18 -
docs/manual/source/partials/_facebook.html.slim | 11 -
docs/manual/source/partials/_footer.html.slim | 43 +-
docs/manual/source/partials/_func.html.slim | 4 -
.../source/partials/_google_analytics.html.slim | 10 -
docs/manual/source/partials/_header.html.slim | 8 +-
.../manual/source/partials/_quantcast.html.slim | 18 -
docs/manual/source/partials/_rcx.html.slim | 7 -
.../source/partials/_subscribe_form.html.slim | 6 -
.../source/partials/head/_javascripts.html.slim | 11 +-
.../stylesheets/variables/_fonts.css.scss | 4 +-
docs/manual/source/support/index.html.md.erb | 19 +-
51 files changed, 1230 insertions(+), 1107 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0e7563f8/docs/manual/config.rb
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0e7563f8/docs/manual/data/nav/main.yml
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0e7563f8/docs/manual/source/community/contribute-webhook.html.md
----------------------------------------------------------------------
diff --cc docs/manual/source/community/contribute-webhook.html.md
index e2612be,2876f01..edd5cd7
--- a/docs/manual/source/community/contribute-webhook.html.md
+++ b/docs/manual/source/community/contribute-webhook.html.md
@@@ -173,10 -177,13 +177,13 @@@ data/src/main/scala/org.apache.predicti
and tests should be in
```
-data/src/test/scala/io/prediction/data/webhooks/segmentio/
+data/src/test/scala/org.apache.predictionio/data/webhooks/segmentio/
```
- **For form-submission data**, you can find the comple example [the GitHub repo](https://github.com/PredictionIO/PredictionIO/blob/develop/data/src/main/scala/org.apache.predictionio/data/webhooks/exampleform/ExampleFormConnector.scala) and how to write [tests for the connector](https://github.com/PredictionIO/PredictionIO/blob/develop/data/src/test/scala/org.apache.predictionio/data/webhooks/exampleform/ExampleFormConnectorSpec.scala).
+ **For form-submission data**, you can find the comple example [the GitHub
+ repo](https://github.com/apache/incubator-predictionio/blob/develop/data/src/main/scala/io/prediction/data/webhooks/exampleform/ExampleFormConnector.scala)
+ and how to write [tests for the
+ connector](https://github.com/apache/incubator-predictionio/blob/develop/data/src/test/scala/io/prediction/data/webhooks/exampleform/ExampleFormConnectorSpec.scala).
## 2. Integrate the Connector into Event Server
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0e7563f8/docs/manual/source/customize/dase.html.md.erb
----------------------------------------------------------------------
diff --cc docs/manual/source/customize/dase.html.md.erb
index 85ed22c,456a373..50c223a
--- a/docs/manual/source/customize/dase.html.md.erb
+++ b/docs/manual/source/customize/dase.html.md.erb
@@@ -165,10 -166,10 +166,10 @@@ run `pio train`. Apache PredictionIO (i
predict() is responsible for using this model to make prediction. It is called when you send a JSON query to the engine. Note that predict() is called in real time.
- PredictionIO supports two types of algorithms:
+ Apache PredictionIO (incubating) supports two types of algorithms:
-- **[P2LAlgorithm](https://docs.prediction.io/api/current/#io.prediction.controller.P2LAlgorithm)**: trains a Model which does not contain RDD
-- **[PAlgorithm](https://docs.prediction.io/api/current/#io.prediction.controller.PAlgorithm)**: trains a Model which contains RDD
+- **[P2LAlgorithm](https://docs.prediction.io/api/current/#org.apache.predictionio.controller.P2LAlgorithm)**: trains a Model which does not contain RDD
+- **[PAlgorithm](https://docs.prediction.io/api/current/#org.apache.predictionio.controller.PAlgorithm)**: trains a Model which contains RDD
## P2LAlgorithm
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0e7563f8/docs/manual/source/customize/troubleshooting.html.md
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0e7563f8/docs/manual/source/demo/textclassification.html.md.erb
----------------------------------------------------------------------
diff --cc docs/manual/source/demo/textclassification.html.md.erb
index 0000000,043db5a..f850816
mode 000000,100644..100644
--- a/docs/manual/source/demo/textclassification.html.md.erb
+++ b/docs/manual/source/demo/textclassification.html.md.erb
@@@ -1,0 -1,673 +1,673 @@@
+ ---
+ title: Text Classification Engine Tutorial
+ ---
+
+ (Updated for Text Classification Template version 3.1)
+
+ ## Introduction
+
+ In the real world, there are many applications that collect text as data. For example, spam detectors take email and header content to automatically determine what is or is not spam; applications can gague the general sentiment in a geographical area by analyzing Twitter data; and news articles can be automatically categorized based solely on the text content.There are a wide array of machine learning models you can use to create, or train, a predictive model to assign an incoming article, or query, to an existing category. Before you can use these techniques you must first transform the text data (in this case the set of news articles) into numeric vectors, or feature vectors, that can be used to train your model.
+
+ The purpose of this tutorial is to illustrate how you can go about doing this using PredictionIO's platform. The advantages of using this platform include: a dynamic engine that responds to queries in real-time; [separation of concerns](http://en.wikipedia.org/wiki/Separation_of_concerns), which offers code re-use and maintainability, and distributed computing capabilities for scalability and efficiency. Moreover, it is easy to incorporate non-trivial data modeling tasks into the DASE architecture allowing Data Scientists to focus on tasks related to modeling. This tutorial will exemplify some of these ideas by guiding you through PredictionIO's [text classification template(http://templates.prediction.io/PredictionIO/template-scala-parallel-textclassification/).
+
+
+
+ ## Prerequisites
+
+ Before getting started, please make sure that you have the latest version of
+ Apache PredictionIO (incubating)
+ [installed](http://predictionio.incubator.apache.org/install/). We emphasize
+ here that this is an engine template written in **Scala** and can be more
+ generally thought of as an SBT project containing all the necessary components.
+
+ You should also download the engine template named Text Classification Engine
+ that accompanies this tutorial by cloning the template repository:
+
+ ```
+ pio template get PredictionIO/template-scala-parallel-textclassification < Your new engine directory >
+ ```
+
+
+ ## Engine Overview
+
+ The engine follows the DASE architecture which we briefly review here. As a user, you are tasked with collecting data for your web or application, and importing it into PredictionIO's Event Server. Once the data is in the server, it can be read and processed by the engine via the Data Source and Preparation components, respectively. The Algorithm component uses the processed, or prepared, data to train a set of predictive models. Once you have trained these models, you are ready to deploy your engine and respond to real-time queries via the Serving component which combines the results from different fitted models. The Evaluation component is used to compute an appropriate metric to test the performance of a fitted model, as well as aid in the tuning of model hyper parameters.
+
+ This engine template is meant to handle text classification which means you will be working with text data. This means that a query, or newly observed documents, will be of the form:
+
+ `{text : String}`.
+
+ In the running example, a query would be an incoming news article. Once the engine is deployed it can process the query, and then return a Predicted Result of the form
+
+ `{category : String, confidence : Double}`.
+
+ Here category is the model's class assignment for this new text document (i.e. the best guess for this article's categorization), and confidence, a value between 0 and 1 representing your confidence in the category prediction (0 meaning you have no confidence in the prediction). The Actual Result is of the form
+
+ `{category : String}`.
+
+ This is used in the evaluation stage when estimating the performance of your predictive model (how well does the model predict categories). Please refer to the [following tutorial](https://docs.prediction.io/customize/) for a more detailed explanation of how your engine will interact with your web application, as well as an in depth-overview of DASE.
+
+
+ ## Quick Start
+
+ This is a quick start guide in case you want to start using the engine right away. Sample email data for spam classification will be used. For more detailed information, read the subsequent sections.
+
+
+ ### 1. Create a new application.
+
+ After the application is created, you will be given an access key and application ID for the application.
+
+ ```
+ $ pio app new MyTextApp
+ ```
+
+ ### 2. Import the tutorial data.
+
+ There are three different data sets available, each giving a different use case for this engine. Please refer to the **Data Source: Reading Event Data** section to see how to appropriate modify the `DataSource` class for use with each respective data set. The default data set is an e-mail spam data set.
+
+ These data sets have already been processed and are ready for [batch import](/datacollection/batchimport/). Replace `***` with your actual application ID.
+
+ ```
+ $ pio import --appid *** --input data/stopwords.json
+
+ $ pio import --appid *** --input data/emails.json
+ ```
+
+ ### 3. Set the engine parameters in the file `engine.json`.
+
+ The default settings are shown below. By default, it uses the algorithm name "lr" which is logstic regression. Please see later section for more detailed explanation of engine.json setting.
+
+ Make sure the "appName" is same as the app you created in step1.
+
+
+ ```
+ {
+ "id": "default",
+ "description": "Default settings",
+ "engineFactory": "org.template.textclassification.TextClassificationEngine",
+ "datasource": {
+ "params": {
+ "appName": "MyTextApp"
+ }
+ },
+ "preparator": {
+ "params": {
+ "nGram": 1,
+ "numFeatures": 500,
+ "SPPMI": false
+ }
+ },
+ "algorithms": [
+ {
+ "name": "lr",
+ "params": {
+ "regParam": 0.00000005
+ }
+ }
+ ]
+ }
+
+ ```
+
+ ### 4. Build your engine.
+
+ ```
+ $ pio build --verbose
+ ```
+
+ This command should take few minutes for the first time; all subsequent builds should be less than a minute. You can also run it without `--verbose` if you don't want to see all the log messages.
+
+ Upon successful build, you should see a console message similar to the following
+
+ ```
+ [INFO] [RegisterEngine$] Registering engine 6wxDy2hxLbvaMJra927ahFdQHDIVXeQz 266bae678c570dee58154b2338cef7aa1646e0d3
+ [INFO] [Console$] Your engine is ready for training.
+ ```
+
+ ### 5.a. Train your model and deploy.
+
+ ```
+ $ pio train
+ ```
+
+ When your engine is trained successfully, you should see a console message similar to the following.
+
+ ```
+ [INFO] [CoreWorkflow$] Training completed successfully.
+ ```
+
+ Now your engine is ready to deploy. Run:
+
+ ```
+ $ pio deploy
+ ```
+
+ When the engine is deployed successfully and running, you should see a console message similar to the following:
+
+ ```
+ [INFO] [HttpListener] Bound to /0.0.0.0:8000
+ [INFO] [MasterActor] Engine is deployed and running. Engine API is live at http://0.0.0.0:8000.
+ ```
+
+ Now you can send query to the engine. Open another terminal and send the following http request to the deployed engine:
+
+ ```
+ $ curl -H "Content-Type: application/json" -d '{ "text":"I like speed and fast motorcycles." }' http://localhost:8000/queries.json
+ ```
+
+ you should see following outputs returned by the engine:
+
+ ```
+ {"category":"not spam","confidence":0.852619510921587}
+ ```
+
+ Try another query:
+
+ ```
+ $ curl -H "Content-Type: application/json" -d '{ "text":"Earn extra cash!" }' http://localhost:8000/queries.json
+ ```
+
+ you should see following outputs returned by the engine:
+
+ ```
+ {"category":"spam","confidence":0.5268770133242983}
+ ```
+
+
+ ### 5.b.Evaluate your training model and tune parameters.
+
+ ```
+ $ pio eval org.template.textclassification.AccuracyEvaluation org.template.textclassification.EngineParamsList
+ ```
+
+ **Note:** Training and evaluation stages are generally different stages of engine development. Evaluation is there to help you choose the best [algorithm parameters](/evaluation/paramtuning/) to use for training an engine that is to be deployed as a web service.
+
+ Depending on your needs, in steps (5.x.) above, you can configure your Spark settings by typing a command of the form:
+
+ ```
+ $ pio command command_parameters -- --master url --driver-memory {0}G --executor-memory {1}G --conf spark.akka.framesize={2} --total_executor_cores {3}
+ ```
+
+ Only the latter commands are listed as these are some of the more commonly
+ modified values. See the [Spark
+ documentation](https://spark.apache.org/docs/latest/spark-standalone.html) and
+ the [PredictionIO
+ FAQ's](http://predictionio.incubator.apache.org/resources/faq/) for more
+ information.
+
+ **Note:** We recommend you set your driver memory to `1G` or `2G` as the data size when dealing with text can be very large.
+
+
+ # Detailed Explanation of DASE
+
+ ## Importing Data
+
+ In the quick start, email spam classification is used. This template can easily be modified for other types text classification.
+
+ If you want to import different sets of data, follow the Quick Start instructions to import data from different files. Make sure that the Data Source is modified accordingly to match the `event`, `entityType`, and `properties` fields set for the specific dataset. The following section explains this in more detail.
+
+ ## Data Source: Reading Event Data
+
+ Now that the data has been imported into PredictionIO's Event Server, it needs to be read from storage to be used by the engine. This is precisely what the DataSource engine component is for, which is implemented in the template script `DataSource.scala`. The class `Observation` serves as a wrapper for storing the information about a news document needed to train a model. The attribute label refers to the label of the category a document belongs to, and text, stores the actual document content as a string. The class TrainingData is used to store an RDD of Observation objects along with the set of stop words.
+
+ The class `DataSourceParams` is used to specify the parameters needed to read and prepare the data for processing. This class is initialized with two parameters `appName` and `evalK`. The first parameter specifies your application name (i.e. MyTextApp), which is needed so that the DataSource component knows where to pull the event data from. The second parameter is used for model evaluation and specifies the number of folds to use in [cross-validation](http://en.wikipedia.org/wiki/Cross-validation_%28statistics%29) when estimating a model performance metric.
+
+ The final and most important ingredient is the DataSource class. This is initialized with its corresponding parameter class, and extends `PDataSource`. This **must** implement the method `readTraining` which returns an instance of type TrainingData. This method completely relies on the defined private methods readEventData and readStopWords. Both of these functions read data observations as Event instances, create an RDD containing these events and finally transforms the RDD of events into an object of the appropriate type as seen below:
+
+ ```scala
+ ...
+ private def readEventData(sc: SparkContext) : RDD[Observation] = {
+ //Get RDD of Events.
+ PEventStore.find(
+ appName = dsp.appName,
+ entityType = Some("content"), // specify data entity type
+ eventNames = Some(List("e-mail")) // specify data event name
+
+ // Convert collected RDD of events to and RDD of Observation
+ // objects.
+ )(sc).map(e => {
+ val label : String = e.properties.get[String]("label")
+ Observation(
+ if (label == "spam") 1.0 else 0.0,
+ e.properties.get[String]("text"),
+ label
+ )
+ }).cache
+ }
+
+ // Helper function used to store stop words from
+ // event server.
+ private def readStopWords(sc : SparkContext) : Set[String] = {
+ PEventStore.find(
+ appName = dsp.appName,
+ entityType = Some("resource"),
+ eventNames = Some(List("stopwords"))
+
+ //Convert collected RDD of strings to a string set.
+ )(sc)
+ .map(e => e.properties.get[String]("word"))
+ .collect
+ .toSet
+ }
+ ...
+ ```
+
+ Note that `readEventData` and `readStopWords` use different entity types and event names, but use the same application name. This is because the sample import script imports two different data types, documents and stop words. These field distinctions are required for distinguishing between the two. The method `readEval` is used to prepare the different cross-validation folds needed for evaluating your model and tuning hyper parameters.
+
+ Now, the default dataset used for training is contained in the file `data/emails.json` and contains a set of e-mail spam data. If we want to switch over to one of the other data sets we must make sure that the `eventNames` and `entityType` fields are changed accordingly.
+
+ In the data/ directory, you will find different sets of data files for different types of text classifcaiton application. The following show one observation from each of the provided data files:
+
+ - `emails.json`:
+
+ ```
+ {"eventTime": "2015-06-08T16:45:00.590+0000", "entityId": 1, "properties": {"text": "Subject: dobmeos with hgh my energy level has gone up ! stukm\nintroducing\ndoctor - formulated\nhgh\nhuman growth hormone - also called hgh\nis referred to in medical science as the master hormone . it is very plentiful\nwhen we are young , but near the age of twenty - one our bodies begin to produce\nless of it . by the time we are forty nearly everyone is deficient in hgh ,\nand at eighty our production has normally diminished at least 90 - 95 % .\nadvantages of hgh :\n- increased muscle strength\n- loss in body fat\n- increased bone density\n- lower blood pressure\n- quickens wound healing\n- reduces cellulite\n- improved vision\n- wrinkle disappearance\n- increased skin thickness texture\n- increased energy levels\n- improved sleep and emotional stability\n- improved memory and mental alertness\n- increased sexual potency\n- resistance to common illness\n- strengthened heart muscle\n- controll
ed cholesterol\n- controlled mood swings\n- new hair growth and color restore\nread\nmore at this website\nunsubscribe\n", "label": "spam"}, "event": "e-mail", "entityType": "content"}
+
+ ```
+
+ - `20newsgroups.json`:
+
+ ```
+ {"entityType": "source", "eventTime": "2015-06-08T18:01:55.003+0000", "event": "documents", "entityId": 1, "properties": {"category": "sci.crypt", "text": "From: rj@ri.cadre.com (Rob deFriesse)\nSubject: Can DES code be shipped to Canada?\nArticle-I.D.: fripp.1993Apr22.125402.27561\nReply-To: rj@ri.cadre.com\nOrganization: Cadre Technologies Inc.\nLines: 13\nNntp-Posting-Host: 192.9.200.19\n\nSomeone in Canada asked me to send him some public domain DES file\nencryption code I have. Is it legal for me to send it?\n\nThanx.\n--\nEschew Obfuscation\n\nRob deFriesse Mail: rj@ri.cadre.com\nCadre Technologies Inc. Phone: (401) 351-5950\n222 Richmond St. Fax: (401) 351-7380\nProvidence, RI 02903\n\nI don't speak for my employer.\n", "label": 11.0}}
+ ```
+
+ - `sentimentanalysis.json`:
+
+ ```
+ {"eventTime": "2015-06-08T16:58:14.278+0000", "entityId": 23714, "entityType": "source", "properties": {"phrase": "Tosca 's intoxicating ardor", "sentiment": 3}, "event": "phrases"}
+ ```
+
+ Now, note that the `entityType`, `event`, and `properties` fields for the `20newsgroups.json` dataset differ from the default `emails.json` set. Default DataSource implementation is to read from `email.json` data set. If you want to use others such as newsgroups data set, the engine's Data Source component must be modified accordingly. To do this, you need only modify the method `readEventData` as follows:
+
+ ### Modify DataSource to Read `20newsgroups.json`
+
+ ```scala
+ private def readEventData(sc: SparkContext) : RDD[Observation] = {
+ //Get RDD of Events.
+ PEventStore.find(
+ appName = dsp.appName,
+ entityType = Some("source"), // specify data entity type
+ eventNames = Some(List("documents")) // specify data event name
+
+ // Convert collected RDD of events to and RDD of Observation
+ // objects.
+ )(sc).map(e => {
+
+ Observation(
+ e.properties.get[Double]("label"),
+ e.properties.get[String]("text"),
+ e.properties.get[String]("category")
+ )
+ }).cache
+ }
+ ```
+
+ ### Modify DataSource to Read `sentimentanalysis.json`
+
+ ```scala
+ private def readEventData(sc: SparkContext) : RDD[Observation] = {
+ //Get RDD of Events.
+ PEventStore.find(
+ appName = dsp.appName,
+ entityType = Some("source"), // specify data entity type
+ eventNames = Some(List("phrases")) // specify data event name
+
+ // Convert collected RDD of events to and RDD of Observation
+ // objects.
+ )(sc).map(e => {
+ val label = e.properties.get[Double]("sentiment")
+
+ Observation(
+ label,
+ e.properties.get[String]("phrase"),
+ label.toString
+ )
+ }).cache
+ }
+ ```
+
+ Note that `event` field in the json file refers to the `eventNames` field in the `readEventData` method. When using this engine with a custom data set, you need to make sure that the respective json fields match with the corresponding fields in the DataSource component. We have included a data sanity check with this engine component that lets you know if your data is actually being read in. If you have 0 observations being read, you should see the following output when your training process performs the Training Data sanity check:
+
+ `Data set is empty, make sure event fields match imported data.`
+
+ This data sanity check is a PredictionIO feature available for your `TrainingData` and `PreparedData` classes. The following code block demonstrates how the sanity check is implemented:
+
+ ```scala
+ class TrainingData(
+ val data : RDD[Observation],
+ val stopWords : Set[String]
+ ) extends Serializable with SanityCheck {
+
+ // Sanity check to make sure your data is being fed in correctly.
+
+ def sanityCheck {
+ try {
+ val obs : Array[Double] = data.takeSample(false, 5).map(_.label)
+
+ println()
+ (0 until 5).foreach(
+ k => println("Observation " + (k + 1) +" label: " + obs(k))
+ )
+ println()
+ } catch {
+ case (e : ArrayIndexOutOfBoundsException) => {
+ println()
+ println("Data set is empty, make sure event fields match imported data.")
+ println()
+ }
+ }
+
+ }
+
+ }
+ ```
+
+ ## Preparator : Data Processing With DASE
+
+ Recall that the Preparator stage is used for doing any prior data processing needed to fit a predictive model. In line with the separation of concerns, the Data Model implementation, PreparedData, is built to do the heavy lifting needed for this data processing. The Preparator must simply implement the prepare method which outputs an object of type PreparedData. This requires you to specify two n-gram window components, and two inverse i.d.f. window components (these terms will be defined in the following section). Therefore a custom class of parameters for the Preparator component, PreparatorParams, must be incorporated. The code defining the full Preparator component is given below:
+
+ ```scala
+ // 1. Initialize Preparator parameters. Recall that for our data
+ // representation we are only required to input the n-gram window
+ // components.
+
+ case class PreparatorParams(
+ nGram: Int,
+ numFeatures: Int = 5000,
+ SPPMI: Boolean
+ ) extends Params
+
+
+
+ // 2. Initialize your Preparator class.
+
+ class Preparator(pp: PreparatorParams) extends PPreparator[TrainingData, PreparedData] {
+
+ // Prepare your training data.
+ def prepare(sc : SparkContext, td: TrainingData): PreparedData = {
+ new PreparedData(td, pp.nGram)
+ }
+ }
+
+ ```
+
+ The simplicity of this stage implementation truly exemplifies one of the benefits of using the PredictionIO platform. For developers, it is easy to incorporate different classes and tools into the DASE framework so that the process of creating an engine is greatly simplified which helps increase your productivity. For data scientists, the load of implementation details you need to worry about is minimized so that you can focus on what is important to you: training a good predictive model.
+
+ The following subsection explains the class PreparedData, which actually handles the transformation of text documents to feature vectors.
+
+ ### PreparedData: Text Vectorization and Feature Reduction
+
+ The Scala class PreparedData which takes the parameters td, nGram, where td is an object of class TrainingData. The other parameter specifies the n-gram parametrization which will be described shortly.
+
+ It will be easier to explain the preparation process with an example, so consider the document \\(d\\):
+
+ `"Hello, my name is Marco."`
+
+ The first thing you need to do is break up \\(d\\) into an array of "allowed tokens." You can think of a token as a terminating sequence of characters that exist in a document (think of a word in a sentence). For example, the list of tokens that appear in \\(d\\) is:
+
+ ```scala
+ val A = Array("Hello", ",", "my", "name", "is", "Marco", ".")
+ ```
+
+ Recall that a set of stop words was also imported in the previous sections. This set of stop words contains all the words (or tokens) that you do not want to include once documents are tokenized. Those tokens that appear in \\(d\\) and are not contained in the set of stop words will be called allowed tokens. So, if the set of stop words is `{"my", "is"}`, then the list of allowed tokens appearing in \\(d\\) is:
+
+ ```scala
+ val A = Array("Hello", ",", "name", "Marco", ".")
+ ```
+
+ The next step in the data representation is to take the array of allowed tokens and extract a set of n-grams and a corresponding value indicating the number of times a given n-gram appears. The set of n-grams for n equal to 1 and 2 in the running example is the set of elements of the form `[A(`\\(i\\)`)]` and `[A(`\\(j\\)`), A(`\\(j + 1\\)`)]`, respectively. In the general case, the set of n-grams extracted from an array of allowed tokens `A` will be of the form `[A(`\\(i\\)`), A(`\\(i + 1\\)`), ..., A(`\\(i + n - 1\\)`)]` for \\(i = 0, 1, 2, ...,\\) `A.size` \\(- n\\). You can set `n` with the `nGram` parameter option in your `PreparatorParams`.
+
+ We use MLLib's `HashingTF` class to implement the conversion from text to term frequency vectors, and can be seen in the following method of the class `PreparedData`:
+
+ ```scala
+ ...
+ // 1. Hashing function: Text -> term frequency vector.
+
+ private val hasher = new HashingTF()
+
+ private def hashTF (text : String) : Vector = {
+ val newList : Array[String] = text.split(" ")
+ .sliding(nGram)
+ .map(_.mkString)
+ .toArray
+
+ hasher.transform(newList)
+ }
+
+ // 2. Term frequency vector -> t.f.-i.d.f. vector.
+
+ val idf : IDFModel = new IDF().fit(td.data.map(e => hashTF(e.text)))
+ ...
+ ```
+
+ The next step is, once all of the observations have been hashed, to collect all n-grams and compute their corresponding [t.f.-i.d.f. value](http://en.wikipedia.org/wiki/Tf%E2%80%93idf). The t.f.-i.d.f. transformation is defined for n-grams, and helps to give less weight to those n-grams that appear with high frequency across all documents, and vice versa. This helps to leverage the predictive power of those words that appear rarely, but can make a big difference in the categorization of a given text document. This is implemented using MLLib's `IDF` and `IDFModel` classes:
+
+ ```scala
+ // 2. Term frequency vector -> t.f.-i.d.f. vector.
+
+ val idf : IDFModel = new IDF().fit(td.data.map(e => hashTF(e.text)))
+ ```
+
+
+ The last two functions that will be mentioned are the methods you will actually use for the data transformation. The method transform takes a document and outputs a sparse vector (MLLib implementation). The transformData method simply transforms the TrainingData input (a corpus of documents) into a set of vectors that can now be used for training. The method transform is used both to transform the training data and future queries.
+
+ ```scala
+ ...
+ // 3. Document Transformer: text => tf-idf vector.
+
+ def transform(text : String): Vector = {
+ // Map(n-gram -> document tf)
+ idf.transform(hashTF(text))
+ }
+
+
+ // 4. Data Transformer: RDD[documents] => RDD[LabeledPoints]
+
+ val transformedData: RDD[(LabeledPoint)] = {
+ td.data.map(e => LabeledPoint(e.label, transform(e.text)))
+ }
+ ```
+
+ The last and final object implemented in this class simply creates a Map with keys being class labels and values, the corresponding category.
+
+ ```scala
+ // 5. Finally extract category map, associating label to category.
+ val categoryMap = td.data.map(e => (e.label, e.category)).collectAsMap
+ ```
+
+
+ ## Algorithm Component
+
+ The algorithm components in this engine, `NBAlgorithm` and `LRAlgorithm`, actually follows a very general form. Firstly, a parameter class must again be initialized to feed in the corresponding Algorithm model parameters. For example, NBAlgorithm incorporates NBAlgorithmParams which holds the appropriate additive smoothing parameter lambda for the Naive Bayes model.
+
+
-The main class of interest in this component is the class that extends [P2LAlgorithm](https://docs.prediction.io/api/current/#io.prediction.controller.P2LAlgorithm). This class must implement a method named train which will output your predictive model (as a concrete object, this will be implemented via a Scala class). It must also implement a predict method that transforms a query to an appropriate feature vector, and uses this to predict with the fitted model. The vectorization function is implemented by a PreparedData object, and the categorization (prediction) is handled by an instance of the NBModel implementation. Again, this demonstrates the facility with which different models can be incorporated into PredictionIO's DASE architecture.
++The main class of interest in this component is the class that extends [P2LAlgorithm](https://docs.prediction.io/api/current/#org.apache.predictionio.controller.P2LAlgorithm). This class must implement a method named train which will output your predictive model (as a concrete object, this will be implemented via a Scala class). It must also implement a predict method that transforms a query to an appropriate feature vector, and uses this to predict with the fitted model. The vectorization function is implemented by a PreparedData object, and the categorization (prediction) is handled by an instance of the NBModel implementation. Again, this demonstrates the facility with which different models can be incorporated into PredictionIO's DASE architecture.
+
+ The model class itself will be discussed in the following section, however, turn your attention to the TextManipulationEngine object defined in the script `Engine.scala`. You can see here that the engine is initialized by specifying the DataSource, Preparator, and Serving classes, as well as a Map of algorithm names to Algorithm classes. This tells the engine which algorithms to run. In practice, you can have as many statistical learning models as you'd like, you simply have to implement a new algorithm component to do this. However, this general design form will persist, and the main meat of the work should be in the implementation of your model class.
+
+ The following subsection will go over our Naive Bayes implementation in NBModel.
+
+
+ ### Naive Bayes Classification
+
+ This Training Model class only uses the Multinomial Naive Bayes [implementation](https://spark.apache.org/docs/latest/mllib-naive-bayes.html) found in the Spark MLLib library. However, recall that the predicted results required in the specifications listed in the overview are of the form:
+
+
+ `{category: String, confidence: Double}`.
+
+ The confidence value should really be interpreted as the probability that a document belongs to a category given its vectorized form. Note that MLLib's Naive Bayes model has the class members pi (\\(\pi\\)), and theta (\\(\theta\\)). \\(\pi\\) is a vector of log prior class probabilities, which shows your prior beliefs regarding the probability that an arbitrary document belongs in a category. \\(\theta\\) is a C \\(\times\\) D matrix, where C is the number of classes, and D, the number of features, giving the log probabilities that parametrize the Multinomial likelihood model assumed for each class. The multinomial model is easiest to think about as a problem of randomly throwing balls into bins, where the ball lands in each bin with a certain probability. The model treats each n-gram as a bin, and the corresponding t.f.-i.d.f. value as the number of balls thrown into it. The likelihood is the probability of observing a (vectorized) document given that it comes from a particular c
lass.
+
+ Now, letting \\(\mathbf{x}\\) be a vectorized text document, then it can be shown that the vector
+
+ $$
+ \frac{\exp\left(\pi + \theta\mathbf{x}\right)}{\left|\left|\exp\left(\pi + \theta\mathbf{x}\right)\right|\right|}
+ $$
+
+ is a vector with C components that represent the posterior class membership probabilities for the document given \\(\mathbf{x}\\). That is, the update belief regarding what category this document belongs to after observing its vectorized form. This is the motivation behind defining the class NBModel which uses Spark MLLib's NaiveBayesModel, but implements a separate prediction method.
+
+ The private methods innerProduct and getScores are implemented to do the matrix computation above.
+
+ ```scala
+ ...
+ // 2. Set up linear algebra framework.
+
+ private def innerProduct (x : Array[Double], y : Array[Double]) : Double = {
+ x.zip(y).map(e => e._1 * e._2).sum
+ }
+
+ val normalize = (u: Array[Double]) => {
+ val uSum = u.sum
+
+ u.map(e => e / uSum)
+ }
+
+
+
+ // 3. Given a document string, return a vector of corresponding
+ // class membership probabilities.
+
+ private def getScores(doc: String): Array[Double] = {
+ // Helper function used to normalize probability scores.
+ // Returns an object of type Array[Double]
+
+ // Vectorize query,
+ val x: Vector = pd.transform(doc)
+
+ normalize(
+ nb.pi
+ .zip(nb.theta)
+ .map(
+ e => exp(innerProduct(e._2, x.toArray) + e._1))
+ )
+ }
+ ...
+ ```
+
+
+ Once you have a vector of class probabilities, you can classify the text document to the category with highest posterior probability, and, finally, return both the category as well as the probability of belonging to that category (i.e. the confidence in the prediction) given the observed data. This is implemented in the method predict.
+
+ ```scala
+ ...
+ // 4. Implement predict method for our model using
+ // the prediction rule given in tutorial.
+
+ def predict(doc : String) : PredictedResult = {
+ val x: Array[Double] = getScores(doc)
+ val y: (Double, Double) = (nb.labels zip x).maxBy(_._2)
+ new PredictedResult(pd.categoryMap.getOrElse(y._1, ""), y._2)
+ }
+ ```
+
+ ### Logistic Regression Classification
+
+ To use the alternative multinomial logistic regression algorithm change your `engine.json` as follows:
+
+ ```json
+ {
+ "id": "default",
+ "description": "Default settings",
+ "engineFactory": "org.template.textclassification.TextClassificationEngine",
+ "datasource": {
+ "params": {
+ "appName": "MyTextApp"
+ }
+ },
+ "preparator": {
+ "params": {
+ "nGram": 2
+ }
+ },
+ "algorithms": [
+ {
+ "name": "regParam",
+ "params": {
+ "regParam": 0.1
+ }
+ }
+ ]
+ }
+ ```
+
+
+ ## Serving: Delivering the Final Prediction
+
-The serving component is the final stage in the engine, and in a sense, the most important. This is the final stage in which you combine the results obtained from the different models you choose to run. The Serving class extends the [LServing](https://docs.prediction.io/api/current/#io.prediction.controller.LServing) class which must implement a method called serve. This takes a query and an associated sequence of predicted results, which contains the predicted results from the different algorithms that are implemented in your engine, and combines the results to yield a final prediction. It is this final prediction that you will receive after sending a query.
++The serving component is the final stage in the engine, and in a sense, the most important. This is the final stage in which you combine the results obtained from the different models you choose to run. The Serving class extends the [LServing](https://docs.prediction.io/api/current/#org.apache.predictionio.controller.LServing) class which must implement a method called serve. This takes a query and an associated sequence of predicted results, which contains the predicted results from the different algorithms that are implemented in your engine, and combines the results to yield a final prediction. It is this final prediction that you will receive after sending a query.
+
+ For example, you could choose to slightly modify the implementation to return class probabilities coming from a mixture of model estimates for class probabilities, or any other technique you could conceive for combining your results. The default engine setting has this set to yield the label from the model predicting with greater confidence.
+
+
+
+ ## Evaluation: Model Assessment and Selection
+
+ A predictive model needs to be evaluated to see how it will generalize to future observations. PredictionIO uses cross-validation to perform model performance metric estimates needed to assess your particular choice of model. The script `Evaluation.scala` available with the engine template exemplifies what a usual evaluator setup will look like. First, you must define an appropriate metric. In the engine template, since the topic is text classification, the default metric implemented is category accuracy.
+
- Second you must define an evaluation object (i.e. extends the class [Evaluation](https://docs.prediction.io/api/current/#io.prediction.controller.Evaluation)).
++ Second you must define an evaluation object (i.e. extends the class [Evaluation](https://docs.prediction.io/api/current/#org.apache.predictionio.controller.Evaluation)).
+ Here, you must specify the actual engine and metric components that are to be used for the evaluation. In the engine template, the specified engine is the TextManipulationEngine object, and metric, Accuracy. Lastly, you must specify the parameter values that you want to test in the cross validation. You see in the following block of code:
+
+ ```scala
+ object EngineParamsList extends EngineParamsGenerator {
+
+ // Set data source and preparator parameters.
+ private[this] val baseEP = EngineParams(
+ dataSourceParams = DataSourceParams(appName = "marco-MyTextApp", evalK = Some(5)),
+ preparatorParams = PreparatorParams(nMin = 1, nMax = 2)
+ )
+
+ // Set the algorithm params for which we will assess an accuracy score.
+ engineParamsList = Seq(
+ baseEP.copy(algorithmParamsList = Seq(("nb", NBAlgorithmParams(0.5)))),
+ baseEP.copy(algorithmParamsList = Seq(("nb", NBAlgorithmParams(1.5)))),
+ baseEP.copy(algorithmParamsList = Seq(("nb", NBAlgorithmParams(5))))
+ )
+ ```
+
+
+ ## Engine Deployment
+
+ Once an engine is ready for deployment it can interact with your web application in real-time. This section will cover how to send and receive queries from your engine, gather more data, and re-training your model with the newly gathered data.
+
+ ### Sending Queries
+
+ Recall that one of the greatest advantages of using the PredictionIO platform is that once your engine is deployed, you can respond to queries in real-time. Recall that our queries are of the form
+
+ `{"text" : "..."}`.
+
+ To actually send a query you can use our REST API by typing in the following shell command:
+
+ ```
+ curl -H "Content-Type: application/json" -d '{ "text":"I like speed and fast motorcycles." }' http://localhost:8000/queries.json
+ ```
+
+ There are a number of [SDK's](https://github.com/PredictionIO) you can use to send your queries and obtain a response. Recall that our predicted response is of the form
+
+ ```
+ {"category" : "class", "confidence" : 1.0}
+ ```
+
+ which is what you should see upon inputting the latter command for querying.
+
+ ### Gathering More Data and Retraining Your Model
+
+ The importing data section that is included in this tutorial uses a sample data set for illustration purposes, and uses the PredictionIO Python SDK to import the data. However, there are a variety of ways that you can [import](<%= url_root %>/datacollection/eventapi/) your collected data (via REST or other SDKs).
+
+
+ As you continue to collect your data, it is quite easy to retrain your model once you actually import your data into the Event Server. You simply repeat the steps listed in the Quick Start guide. We re-list them here again:
+
+
+ **1.** Build your engine.
+
+ ```
+ $ pio build
+ ```
+
+ **2.a.** Evaluate your training model and tune parameters.
+
+ ```
+ $ pio eval org.template.textclassification.AccuracyEvaluation org.template.textclassification.EngineParamsList
+ ```
+
+ **2.b.** Train your model and deploy.
+
+ ```
+ $ pio train
+ $ pio deploy
+ ```
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/0e7563f8/docs/manual/source/install/index.html.md.erb
----------------------------------------------------------------------
diff --cc docs/manual/source/install/index.html.md.erb
index a4c3edc,84e9719..4f7d1f6
--- a/docs/manual/source/install/index.html.md.erb
+++ b/docs/manual/source/install/index.html.md.erb
@@@ -24,10 -19,10 +19,10 @@@ The above script will complete the inst
### Prerequisites
It is **very important** to meet the minimum version of the following
- technologies that power PredictionIO.
+ technologies that power Apache PredictionIO (incubating).
* Apache Hadoop 2.4.0 (optional, required only if YARN and HDFS are needed)
-* Apache Spark 1.3.0 for Hadoop 2.4
+* Apache Spark 1.4.0 for Hadoop 2.4
* Java SE Development Kit 7
and one of the following sets:
[14/15] incubator-predictionio git commit: Replace reference to
Google Groups with Apache mailing lists
Posted by do...@apache.org.
Replace reference to Google Groups with Apache mailing lists
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/8fc24eb0
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/8fc24eb0
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/8fc24eb0
Branch: refs/heads/develop
Commit: 8fc24eb0cab34b6a67a837610b2a0c171ec0252a
Parents: 1ed9b15
Author: Donald Szeto <do...@apache.org>
Authored: Mon Jul 25 14:34:13 2016 -0700
Committer: Donald Szeto <do...@apache.org>
Committed: Mon Jul 25 14:34:13 2016 -0700
----------------------------------------------------------------------
docs/manual/data/nav/main.yml | 5 +----
docs/manual/source/partials/_header.html.slim | 4 ----
docs/manual/source/support/index.html.md.erb | 19 +++++++++++--------
3 files changed, 12 insertions(+), 16 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/8fc24eb0/docs/manual/data/nav/main.yml
----------------------------------------------------------------------
diff --git a/docs/manual/data/nav/main.yml b/docs/manual/data/nav/main.yml
index 65b3445..200bbe8 100644
--- a/docs/manual/data/nav/main.yml
+++ b/docs/manual/data/nav/main.yml
@@ -209,11 +209,8 @@ root:
body: 'FAQs'
url: '/resources/faq/'
-
- body: 'Community Support'
+ body: 'Support'
url: '/support/'
- -
- body: 'Enterprise Support'
- url: '/support/#enterprise-support'
-
body: 'Resources'
url: '#'
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/8fc24eb0/docs/manual/source/partials/_header.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_header.html.slim b/docs/manual/source/partials/_header.html.slim
index ed78bed..b43269e 100644
--- a/docs/manual/source/partials/_header.html.slim
+++ b/docs/manual/source/partials/_header.html.slim
@@ -8,10 +8,6 @@ header
= link_to 'http://predictionio.incubator.apache.org/' do
= image_tag 'logos/logo.png', alt: 'PredictionIO', id: 'logo'
#menu-wrapper
- #header-nav-options-wrapper
- ul
- li>: a href="/" Install & Doc
- li>: a href="/support" Support
#pill-wrapper
a.pill.left> href="//templates.prediction.io/" TEMPLATES
a.pill.right href="//github.com/apache/incubator-predictionio/" OPEN SOURCE
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/8fc24eb0/docs/manual/source/support/index.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/support/index.html.md.erb b/docs/manual/source/support/index.html.md.erb
index 6ef45a8..5507bd4 100644
--- a/docs/manual/source/support/index.html.md.erb
+++ b/docs/manual/source/support/index.html.md.erb
@@ -4,18 +4,21 @@ title: Getting Help
## Community Support
-PredictionIO has a welcoming and active community.
-We are here to support you and make sure that you can use PredictionIO successfully.
+Apahce PredictionIO (incubating) has a welcoming and active community. We are
+here to support you and make sure that you can use Apahce PredictionIO
+(incubating) successfully.
-Here are some resources you can use:
+If you are a user, please subscribe to our user mailing list.
-* [User Support Forum](https://groups.google.com/forum/#!forum/predictionio-user)
-* [Product Feature Request](https://predictionio.uservoice.com/forums/219398-general/filters/top)
+* [Subscribe](mailto:user-subscribe@predictionio.incubator.apache.org)
+* [Unsubscribe](mailto:user-unsubscribe@predictionio.incubator.apache.org)
-If you are a contributor, we have a dedicated forum for you:
+If you are a contributor, please subscribe to our development mailing list.
-* [Contributor Support Forum](https://groups.google.com/forum/#!forum/predictionio-dev)
+* [Subscribe](mailto:dev-subscribe@predictionio.incubator.apache.org)
+* [Unsubscribe](mailto:dev-unsubscribe@predictionio.incubator.apache.org)
## Enterprise Support
-For enterprise users who require professional support, you may contact <a href="mailto:contact@actionml.com">contact@actionml.com</a>
+For enterprise users who require professional support, you may contact <a
+href="mailto:contact@actionml.com">contact@actionml.com</a>
[04/15] incubator-predictionio git commit: Update footer
Posted by do...@apache.org.
Update footer
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/9d497546
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/9d497546
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/9d497546
Branch: refs/heads/develop
Commit: 9d497546914e5256bfaef386970f0de9ab9eb421
Parents: 2800083
Author: Donald Szeto <do...@prediction.io>
Authored: Mon Apr 11 10:47:48 2016 -0700
Committer: Donald Szeto <do...@prediction.io>
Committed: Mon Apr 11 10:47:48 2016 -0700
----------------------------------------------------------------------
docs/manual/Gemfile.lock | 6 +++
docs/manual/source/partials/_footer.html.slim | 43 ++++++++--------------
2 files changed, 22 insertions(+), 27 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/9d497546/docs/manual/Gemfile.lock
----------------------------------------------------------------------
diff --git a/docs/manual/Gemfile.lock b/docs/manual/Gemfile.lock
index 495ec1c..947840f 100644
--- a/docs/manual/Gemfile.lock
+++ b/docs/manual/Gemfile.lock
@@ -91,6 +91,7 @@ GEM
kramdown (1.6.0)
launchy (2.4.3)
addressable (~> 2.3)
+ libv8 (3.16.14.13)
listen (2.10.0)
celluloid (~> 0.16.0)
rb-fsevent (>= 0.9.3)
@@ -181,6 +182,7 @@ GEM
rb-inotify (0.9.5)
ffi (>= 0.5.0)
redcarpet (3.3.4)
+ ref (2.0.0)
rouge (1.8.0)
ruby-progressbar (1.7.5)
sass (3.4.22)
@@ -198,6 +200,9 @@ GEM
sprockets (~> 2.0)
tilt (~> 1.1)
temple (0.7.6)
+ therubyracer (0.12.2)
+ libv8 (~> 3.16.14.0)
+ ref
thor (0.19.1)
thread_safe (0.3.5)
tilt (1.4.1)
@@ -242,6 +247,7 @@ DEPENDENCIES
rainbow
redcarpet (>= 3.2.3)
slim
+ therubyracer
travis
tzinfo-data
wdm (~> 0.1.0)
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/9d497546/docs/manual/source/partials/_footer.html.slim
----------------------------------------------------------------------
diff --git a/docs/manual/source/partials/_footer.html.slim b/docs/manual/source/partials/_footer.html.slim
index e40f2f4..7d6f7f2 100644
--- a/docs/manual/source/partials/_footer.html.slim
+++ b/docs/manual/source/partials/_footer.html.slim
@@ -4,46 +4,35 @@ footer
.container
.seperator
.row
- .col-md-4.col-md-push-8.col-xs-12
- = partial 'subscribe_form'
-
- .col-md-2.col-md-pull-4.col-xs-6.footer-link-column
+ .col-md-4.col-xs-6.footer-link-column
.footer-link-column-row
h4 Community
ul
- li: a href="//docs.prediction.io/install/" target="blank" Download
- li: a href="//docs.prediction.io/" target="blank" Docs
- li: a href="//github.com/PredictionIO/PredictionIO" target="blank" GitHub
- li: a href="//groups.google.com/forum/#!forum/predictionio-user" target="blank" Support Forum
- li: a href="//stackoverflow.com/questions/tagged/predictionio" target="blank" Stackoverflow
+ li: a href="//docs.prediction.io/install/" target="_blank" Download
+ li: a href="//docs.prediction.io/" target="_blank" Docs
+ li: a href="//github.com/PredictionIO/PredictionIO" target="_blank" GitHub
+ li: a href="//groups.google.com/forum/#!forum/predictionio-user" target="_blank" Support Forum
+ li: a href="//stackoverflow.com/questions/tagged/predictionio" target="_blank" Stackoverflow
li: a href="mailto:support@prediction.io" target="blank" Contact Us
- .col-md-2.col-md-pull-4.col-xs-6.footer-link-column
+ .col-md-4.col-xs-6.footer-link-column
.footer-link-column-row
h4 Contribute
ul
- li: a href="//docs.prediction.io/community/contribute-code/" target="blank" Contribute
- li: a href="//github.com/PredictionIO/PredictionIO" target="blank" Source Code
- li: a href="//github.com/PredictionIO/PredictionIO/issues" target="blank" Bug Tracker
- li: a href="//groups.google.com/forum/#!forum/predictionio-dev" target="blank" Contributors’ Forum
- li: a href="//prediction.io/cla" Contributor Agreement
- li: a href="//predictionio.uservoice.com/forums/219398-general/filters/top" Request Features
- .col-md-2.col-md-pull-4.col-xs-6.footer-link-column
+ li: a href="//docs.prediction.io/community/contribute-code/" target="_blank" Contribute
+ li: a href="//github.com/PredictionIO/PredictionIO" target="_blank" Source Code
+ li: a href="//github.com/PredictionIO/PredictionIO/issues" target="_blank" Bug Tracker
+ li: a href="//groups.google.com/forum/#!forum/predictionio-dev" target="_blank" Contributors’ Forum
+ li: a href="//prediction.io/cla" target="_blank" Contributor Agreement
+ li: a href="//predictionio.uservoice.com/forums/219398-general/filters/top" target="_blank" Request Features
+ .col-md-4.col-xs-6.footer-link-column
.footer-link-column-row
h4 Enterprise
ul
- li: a href="//docs.prediction.io/support/" target="blank" Support
- li: a href="//prediction.io/enterprise" Enterprise
- li: a href="//prediction.io/products/predictionio-enterprise" Services
+ li: a href="//docs.prediction.io/support/" target="_blank" Support
.footer-link-column-row
h4 Connect
ul
- li: a href="//blog.prediction.io/" target="blank" Blog
- li: a href="//predictionio.theresumator.com/" target="blank" Careers
- .col-md-2.col-md-pull-4.col-xs-6.footer-link-column
- .footer-link-column-row
- h4 Partnership
- ul
- li: a href="//prediction.io/partners/program" Partner Program
+ li: a href="//blog.prediction.io/" target="_blank" Blog
#footer-bottom
.container
[12/15] incubator-predictionio git commit: Merge remote-tracking
branch 'alex9311/docs-monit' into livedoc
Posted by do...@apache.org.
Merge remote-tracking branch 'alex9311/docs-monit' into livedoc
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/6cab260e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/6cab260e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/6cab260e
Branch: refs/heads/develop
Commit: 6cab260e1d368313056129178dca15d6eb3209a8
Parents: 0cebfb7b 28ca061
Author: Donald Szeto <do...@apache.org>
Authored: Mon Jul 25 14:25:14 2016 -0700
Committer: Donald Szeto <do...@apache.org>
Committed: Mon Jul 25 14:25:14 2016 -0700
----------------------------------------------------------------------
docs/manual/Gemfile | 1 -
docs/manual/Gemfile.lock | 4 +-
docs/manual/data/nav/main.yml | 3 +
docs/manual/source/deploy/monitoring.html.md | 154 +++++++++++++++++++
docs/manual/source/images/monit.png | Bin 0 -> 192080 bytes
.../source/partials/_action_call.html.slim | 2 +-
6 files changed, 160 insertions(+), 4 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/6cab260e/docs/manual/data/nav/main.yml
----------------------------------------------------------------------
[10/15] incubator-predictionio git commit: [PIO-6] Fix build error
Posted by do...@apache.org.
[PIO-6] Fix build error
Project: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/commit/e0a73d21
Tree: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/tree/e0a73d21
Diff: http://git-wip-us.apache.org/repos/asf/incubator-predictionio/diff/e0a73d21
Branch: refs/heads/develop
Commit: e0a73d215a1d013dc63fdf3ded44456f253b400c
Parents: 5be3cca
Author: Donald Szeto <do...@apache.org>
Authored: Wed Jul 20 12:08:38 2016 -0700
Committer: Donald Szeto <do...@apache.org>
Committed: Wed Jul 20 12:08:38 2016 -0700
----------------------------------------------------------------------
docs/manual/source/demo/index.html.md.erb | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-predictionio/blob/e0a73d21/docs/manual/source/demo/index.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/demo/index.html.md.erb b/docs/manual/source/demo/index.html.md.erb
index 43da798..4fe11f3 100644
--- a/docs/manual/source/demo/index.html.md.erb
+++ b/docs/manual/source/demo/index.html.md.erb
@@ -10,5 +10,4 @@ PredictionIO (incubating).
## Contribute
-Interested in writing a demo? Contact us at <%= mail_to
-'dev@prediction.incubator.apache.org' %>.
+Interested in writing a demo? Contact us at <%= mail_to 'dev@prediction.incubator.apache.org' %>.