You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@predictionio.apache.org by do...@apache.org on 2018/08/27 00:31:10 UTC
predictionio git commit: [PIO-159] Update IntelliJ documentation
Repository: predictionio
Updated Branches:
refs/heads/livedoc fc481c9c8 -> 1c8ea8afe
[PIO-159] Update IntelliJ documentation
Project: http://git-wip-us.apache.org/repos/asf/predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/predictionio/commit/1c8ea8af
Tree: http://git-wip-us.apache.org/repos/asf/predictionio/tree/1c8ea8af
Diff: http://git-wip-us.apache.org/repos/asf/predictionio/diff/1c8ea8af
Branch: refs/heads/livedoc
Commit: 1c8ea8afecfcd56cc29c18f81c0a4d28bd29e05c
Parents: fc481c9
Author: Donald Szeto <do...@apache.org>
Authored: Sun Aug 26 17:30:02 2018 -0700
Committer: Donald Szeto <do...@apache.org>
Committed: Sun Aug 26 17:30:02 2018 -0700
----------------------------------------------------------------------
.../images/intellij/intelliJ-scala-plugin.png | Bin 89212 -> 187820 bytes
.../images/intellij/intellij-config-2.png | Bin 259619 -> 0 bytes
.../images/intellij/intellij-dependencies.png | Bin 238042 -> 0 bytes
.../intellij/intellij-module-settings.png | Bin 0 -> 471191 bytes
.../images/intellij/intellij-scala-plugin-2.png | Bin 128386 -> 244855 bytes
.../images/intellij/pio-runtime-jar-deps.png | Bin 258163 -> 0 bytes
.../source/images/intellij/pio-runtime-jars.png | Bin 650385 -> 0 bytes
.../images/intellij/pio-train-env-vars.png | Bin 0 -> 212997 bytes
.../manual/source/images/intellij/pio-train.png | Bin 0 -> 274725 bytes
.../source/resources/intellij.html.md.erb | 218 +++++++++++--------
10 files changed, 122 insertions(+), 96 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/predictionio/blob/1c8ea8af/docs/manual/source/images/intellij/intelliJ-scala-plugin.png
----------------------------------------------------------------------
diff --git a/docs/manual/source/images/intellij/intelliJ-scala-plugin.png b/docs/manual/source/images/intellij/intelliJ-scala-plugin.png
index 8725257..7de0023 100644
Binary files a/docs/manual/source/images/intellij/intelliJ-scala-plugin.png and b/docs/manual/source/images/intellij/intelliJ-scala-plugin.png differ
http://git-wip-us.apache.org/repos/asf/predictionio/blob/1c8ea8af/docs/manual/source/images/intellij/intellij-config-2.png
----------------------------------------------------------------------
diff --git a/docs/manual/source/images/intellij/intellij-config-2.png b/docs/manual/source/images/intellij/intellij-config-2.png
deleted file mode 100644
index 5d84ee8..0000000
Binary files a/docs/manual/source/images/intellij/intellij-config-2.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/predictionio/blob/1c8ea8af/docs/manual/source/images/intellij/intellij-dependencies.png
----------------------------------------------------------------------
diff --git a/docs/manual/source/images/intellij/intellij-dependencies.png b/docs/manual/source/images/intellij/intellij-dependencies.png
deleted file mode 100644
index ff6ec0c..0000000
Binary files a/docs/manual/source/images/intellij/intellij-dependencies.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/predictionio/blob/1c8ea8af/docs/manual/source/images/intellij/intellij-module-settings.png
----------------------------------------------------------------------
diff --git a/docs/manual/source/images/intellij/intellij-module-settings.png b/docs/manual/source/images/intellij/intellij-module-settings.png
new file mode 100644
index 0000000..63c3234
Binary files /dev/null and b/docs/manual/source/images/intellij/intellij-module-settings.png differ
http://git-wip-us.apache.org/repos/asf/predictionio/blob/1c8ea8af/docs/manual/source/images/intellij/intellij-scala-plugin-2.png
----------------------------------------------------------------------
diff --git a/docs/manual/source/images/intellij/intellij-scala-plugin-2.png b/docs/manual/source/images/intellij/intellij-scala-plugin-2.png
index 76066ce..d121255 100644
Binary files a/docs/manual/source/images/intellij/intellij-scala-plugin-2.png and b/docs/manual/source/images/intellij/intellij-scala-plugin-2.png differ
http://git-wip-us.apache.org/repos/asf/predictionio/blob/1c8ea8af/docs/manual/source/images/intellij/pio-runtime-jar-deps.png
----------------------------------------------------------------------
diff --git a/docs/manual/source/images/intellij/pio-runtime-jar-deps.png b/docs/manual/source/images/intellij/pio-runtime-jar-deps.png
deleted file mode 100644
index 84410ed..0000000
Binary files a/docs/manual/source/images/intellij/pio-runtime-jar-deps.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/predictionio/blob/1c8ea8af/docs/manual/source/images/intellij/pio-runtime-jars.png
----------------------------------------------------------------------
diff --git a/docs/manual/source/images/intellij/pio-runtime-jars.png b/docs/manual/source/images/intellij/pio-runtime-jars.png
deleted file mode 100644
index 587bf04..0000000
Binary files a/docs/manual/source/images/intellij/pio-runtime-jars.png and /dev/null differ
http://git-wip-us.apache.org/repos/asf/predictionio/blob/1c8ea8af/docs/manual/source/images/intellij/pio-train-env-vars.png
----------------------------------------------------------------------
diff --git a/docs/manual/source/images/intellij/pio-train-env-vars.png b/docs/manual/source/images/intellij/pio-train-env-vars.png
new file mode 100644
index 0000000..5c9675b
Binary files /dev/null and b/docs/manual/source/images/intellij/pio-train-env-vars.png differ
http://git-wip-us.apache.org/repos/asf/predictionio/blob/1c8ea8af/docs/manual/source/images/intellij/pio-train.png
----------------------------------------------------------------------
diff --git a/docs/manual/source/images/intellij/pio-train.png b/docs/manual/source/images/intellij/pio-train.png
new file mode 100644
index 0000000..d5c4647
Binary files /dev/null and b/docs/manual/source/images/intellij/pio-train.png differ
http://git-wip-us.apache.org/repos/asf/predictionio/blob/1c8ea8af/docs/manual/source/resources/intellij.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/resources/intellij.html.md.erb b/docs/manual/source/resources/intellij.html.md.erb
index 137f0fb..aa10743 100644
--- a/docs/manual/source/resources/intellij.html.md.erb
+++ b/docs/manual/source/resources/intellij.html.md.erb
@@ -25,6 +25,9 @@ This documentation assumes that you have a fully functional PredictionIO setup.
If you have not installed PredictionIO yet, please follow [these
instructions](/install/).
+The following instructions have been tested with IntelliJ IDEA 2018.2.2
+Community Edition.
+
## Preparing IntelliJ for Engine Development
@@ -51,29 +54,28 @@ IDEA if asked to do so.
### Setting Up the Engine Directory
-INFO: It is very important to run at least `pio build` once in your engine
-directory so that the project correctly recognizes the version of PredictionIO
-that you are using. If you upgraded your PredictionIO installation later, you
-will need to run `pio build` again in order for the engine to pick up the latest
-version of PredictionIO.
-
-Create an engine directory from a template. This requires that you install a
+Create an engine directory from a template. This requires that you download a
template that you wish to start from or modify.
-Follow template [install](/start/download) and [deploy](/start/deploy) instructions
-or go through the [Quick Start](/templates/recommendation/quickstart/) if you are
-planning to modify a recommender. Make sure to build, train, and deploy the
-engine to make sure all is configured properly.
+
+Follow template [install](/start/download) and [deploy](/start/deploy)
+instructions or go through the [Quick
+Start](/templates/recommendation/quickstart/) if you are planning to modify a
+recommender. Make sure to build, train, and deploy the engine to make sure all
+is configured properly.
From IntelliJ IDEA, choose *File* > *New* > *Project from Existing Sources...*.
-When asked to select a directory to import, browse to the engine directory that you
-downloaded too and proceed. Make sure you pick *Import project from external model* > *SBT*,
-then proceed to finish.
+When asked to select a directory to import, browse to the engine directory that
+you downloaded too and proceed. Make sure you pick *Import project from external
+model* > *SBT*, then proceed to finish.
-You should be able to build the project at this point. To run and debug
-PredictionIO server, continue on to the rest of the steps.
+You should be able to build the project at this point. To run and debug your
+template, continue on to the rest of the steps.
-INFO: If you are running on OS X, you will need to do the following due to
-this [known issue](http://bit.ly/12Abtvn).
+
+### Optional: Issues with Snappy on macOS
+
+If you are running on macOS and run into the following [known
+issue](http://bit.ly/12Abtvn), follow steps in this section.
Edit `build.sbt` and add the following under `libraryDependencies`
@@ -83,35 +85,71 @@ Edit `build.sbt` and add the following under `libraryDependencies`
-When you are done editing, IntelliJ should either refresh the project
-automatically or prompt you to refresh.
+When you are done editing, IntelliJ should prompt you to import new changes,
+unless you have already enabled auto import. Import this change to make it
+effective.
+
+
+### Module Settings
+
+INFO: IntelliJ will recreate module settings whenever it imports changes of your
+project. You will need to repeat this section whenever that happens.
+
+Due to the way how `pio` command sources required classes during runtime, it is
+necessary to add them manually in module settings for *Run/Debug Configurations*
+to work properly.
+
+Right click on the project and click *Open Module Settings*. Hit the **+**
+button right below the list of dependencies, and select *JARs or
+directories...*.
+
+The first JAR that you need to add is the `pio-assembly-<%= data.versions.pio
+%>.jar` that contains all necessary classes. It can be found inside the
+`dist/lib` directory of your PredictionIO source installation directory (if you
+have built from sources) or the `lib` directory of your PredictionIO binary
+installation directory.
+Next, you will need to make sure some configuration files from your PredictionIO
+installation can be found during runtime. Add the `conf` directory of your
+PredictionIO installation directory. When asked about categories of the
+directory, pick *Classes*.
-### Dependencies
+Finally, you will need to add storage classes. The exact list of JARs that you
+will need to add depends on your storage configuration. These JARs can be found
+inside the `dist/lib/spark` directory of your PredictionIO source installation
+directory (if you have built from sources) or the `lib/spark` directory of your
+PredictionIO binary installation directory.
-IntelliJ has the annoying tendency to drop some dependencies when you refresh your build.sbt after any changes.
-To avoid this we put any jars that must be available at runtime into a separate empty module in the project then
-we make the main engine project depend on this dummy module for runtime classes.
+* `pio-data-elasticsearch-assembly-<%= data.versions.pio %>.jar`
-Right click on the project and click *Open Module Settings*. In the second modules column hit **+** and create a
-new Scala module. Name it pio-runtime-jars and add these assemblies under the module dependencies tab and remember to
-change the scope of the jars to **runtime**:
+ Add this JAR if your configuration uses Elasticsearch.
-- `pio-assembly-<%= data.versions.pio %>.jar`
+* `pio-data-hbase-assembly-<%= data.versions.pio %>.jar`
- This JAR can be found inside the `assembly` or `lib` directory of your PredictionIO
- installation directory.
+ Add this JAR if your configuration uses Apache HBase.
-- `spark-assembly-<%= data.versions.spark %>-hadoop2.4.0.jar`
+* `pio-data-hdfs-assembly-<%= data.versions.pio %>.jar`
- This JAR can be found inside the `assembly` or `lib` directory of your Apache Spark
- installation directory.
+ Add this JAR if your configuration uses HDFS.
-
+* `pio-data-jdbc-assembly-<%= data.versions.pio %>.jar`
-Now make your engine module dependent on the pio-runtime-jars module for scope = runtime.
+ Add this JAR if your configuration uses JDBC. Notice that you must also add any
+ additional JDBC driver JARs.
-
+* `pio-data-localfs-assembly-<%= data.versions.pio %>.jar`
+
+ Add this JAR if your configuration uses local filesystem.
+
+* `pio-data-s3-assembly-<%= data.versions.pio %>.jar`
+
+ Add this JAR if your configuration uses Amazon Web Services S3.
+
+Make sure to change all these additions to *Runtime* scope. The following shows
+an example that uses the JDBC storage backend with PostgreSQL driver.
+
+
## Running and Debugging in IntelliJ IDEA
@@ -121,47 +159,41 @@ Now make your engine module dependent on the pio-runtime-jars module for scope =
Create a new *Run/Debug Configuration* by going to *Run* > *Edit
Configurations...*. Click on the **+** button and select *Application*. Name it
-`pio train` and put in the following.
+`pio train` and put in the following:
-```
-Main class: org.apache.predictionio.workflow.CreateWorkflow
-VM options: -Dspark.master=local -Dlog4j.configuration=file:/**replace_with_your_PredictionIO_path**/conf/log4j.properties
-Program arguments: --engine-id dummy --engine-version dummy --engine-variant engine.json
-```
+* Main class:
-Click the **...** button to the right of *Environment variables*, and paste the
-following.
+ ```
+ org.apache.predictionio.workflow.CreateWorkflow
+ ```
-```
-SPARK_HOME=/**reaplce_w_your_spark_binary_path**
-PIO_FS_BASEDIR=/**replace_w_your_path_to**/.pio_store
-PIO_FS_ENGINESDIR=/**replace_w_your_path_to**/.pio_store/engines
-PIO_FS_TMPDIR=/**replace_w_your_path_to*/.pio_store/tmp
-PIO_STORAGE_REPOSITORIES_METADATA_NAME=predictionio_metadata
-PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
-PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_
-PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
-PIO_STORAGE_REPOSITORIES_APPDATA_NAME=predictionio_appdata
-PIO_STORAGE_REPOSITORIES_APPDATA_SOURCE=ELASTICSEARCH
-PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=predictionio_eventdata
-PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
-PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
-PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
-PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
-PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
-PIO_STORAGE_SOURCES_LOCALFS_HOSTS=/**replace_w_your_path_to**/.pio_store/models
-PIO_STORAGE_SOURCES_LOCALFS_PORTS=0
-PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
-PIO_STORAGE_SOURCES_HBASE_HOSTS=0
-PIO_STORAGE_SOURCES_HBASE_PORTS=0
-```
+* VM options:
+
+ ```
+ -Dspark.master=local -Dlog4j.configuration=file:/<your_pio_path>/conf/log4j.properties -Dpio.log.dir=<path_of_log_file>
+ ```
-INFO: Remember to replace all paths that start with `**replace` with actual
-values. The directory `.pio_store` typically locates inside your home directory.
+* Program arguments:
+
+ ```
+ --engine-id dummy --engine-version dummy --engine-variant engine.json --env dummy=dummy
+ ```
+
+Make sure *Working directory* is set to the base directory of the template that
+you are working on.
+
+Click the folder button to the right of *Environment variables*, and paste the
+relevant values from `conf/pio-env.sh` in your PredictionIO installation
+directory. The following shows an example using JDBC and PostgreSQL.
+
+
+
+Make sure *Include dependencies with "Provided" scope* is checked.
The end result should look something similar to this.
-
+
Save and you can run or debug `pio train` with the new configuration!
@@ -171,20 +203,30 @@ Save and you can run or debug `pio train` with the new configuration!
For `pio deploy`, simply duplicate the previous configuration and replace with
the following.
-```
-Main class: org.apache.predictionio.workflow.CreateServer
-Program Arguments: --engineInstanceId **replace_with_the_id_from_pio_train**
-```
+* Main class:
+
+ ```
+ org.apache.predictionio.workflow.CreateServer
+ ```
+
+* Program Arguments:
+
+ ```
+ --engineInstanceId <id_from_pio_train> --engine-variant engine.json
+ ```
+
### Executing a Query
-You can execute a query with the correct SDK. For a recommender that has been
-trained with the sample MovieLens dataset perhaps the easiest query is a curl
-one. Start by running or debuging your `deploy` config so the service is
-waiting for the query. Then go to the "Terminal" tab at the very bottom of the
-IDEA window and enter the curl request:
+You can execute a query with the correct SDK. For a recommender that has been
+trained with the sample MovieLens dataset perhaps the easiest query is a `curl`
+one. Start by running or debuging your `pio deploy` config so the service is
+waiting for the query. Then go to the "Terminal" tab at the very bottom of the
+IntelliJ IDEA window and enter the `curl` request:
-```$ curl -H "Content-Type: application/json" -d '{ "user": "1", "num": 4 }' http://localhost:8000/queries.json```
+```
+curl -H "Content-Type: application/json" -d '{ "user": "1", "num": 4 }' http://localhost:8000/queries.json
+```
This should return something like:
@@ -200,19 +242,3 @@ This should return something like:
INFO: If you hit a breakpoint you are likely to get a connection timeout. To see
the data that would have been returned, just place a breakpoint where the
response is created or run the query with no breakpoints.
-
-## Loading a Template Into Intellij IDEA
-
-To customize an existing [template](/gallery/template-gallery) using Intellij IDEA, first pull it from the template gallery:
-
-```bash
-$ git clone <Template Source> <New Engine Directory>
-```
-
-Now, before opening the template with Intellij, run the following command in the new engine template directory
-
-```bash
-$ pio build
-```
-
-This should update the pioVersion key in SBT to the version of PredictionIO you have installed, so that Intellij loads the correct JARS via its Auto-Import feature. Now, you can go ahead and open the file `build.sbt` with Intellij IDEA. You are now ready to [customize](/customize/) your new engine template.