You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@predictionio.apache.org by em...@apache.org on 2019/07/31 18:16:42 UTC
[predictionio] branch develop updated (d5b5ad5 -> b2ef983)
This is an automated email from the ASF dual-hosted git repository.
emergentorder pushed a change to branch develop
in repository https://gitbox.apache.org/repos/asf/predictionio.git.
from d5b5ad5 Fix a bug that prevents events with targetEntityType and targetEntity… (#522)
add 823c441 Bump patch versions for akka and hadoop
add c0d5682 Use gitbox for developerConnection
add 74b304d Fix LICENSE
add 37a2df6 Bump akka minor version; fix developerConnection
add 0faf78a Fix LICENSE
new 92cab7a Merge
new a9e12de Merge branch 'develop' of https://github.com/apache/predictionio into develop
new 1094de8 Set engine id, version, variant, even in Python
new b2ef983 [PIO-211] Fix pio deploy fail when using HDFS model storage
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.travis.yml | 5 -----
build.sbt | 19 +++++++++++++++++++
.../workflow/EngineServerPluginContext.scala | 5 +++--
.../source/install/install-sourcecode.html.md.erb | 11 ++++++++---
.../org/apache/predictionio/tools/RunWorkflow.scala | 4 ++--
5 files changed, 32 insertions(+), 12 deletions(-)
[predictionio] 02/04: Merge branch 'develop' of
https://github.com/apache/predictionio into develop
Posted by em...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
emergentorder pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/predictionio.git
commit a9e12deae95650228a4b59cc8bd6210f6cffd899
Merge: 92cab7a d5b5ad5
Author: Alex Merritt <le...@gmail.com>
AuthorDate: Mon Jul 22 17:58:19 2019 -0300
Merge branch 'develop' of https://github.com/apache/predictionio into develop
.../predictionio/data/store/python/PPythonEventStore.scala | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
[predictionio] 04/04: [PIO-211] Fix pio deploy fail when using HDFS
model storage
Posted by em...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
emergentorder pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/predictionio.git
commit b2ef98369542ddd6cd8b783574cdb9d62081a8bf
Author: Alex Merritt <le...@gmail.com>
AuthorDate: Wed Jul 31 15:06:20 2019 -0300
[PIO-211] Fix pio deploy fail when using HDFS model storage
---
.../org/apache/predictionio/workflow/EngineServerPluginContext.scala | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/core/src/main/scala/org/apache/predictionio/workflow/EngineServerPluginContext.scala b/core/src/main/scala/org/apache/predictionio/workflow/EngineServerPluginContext.scala
index 011cd95..1d2eaaa 100644
--- a/core/src/main/scala/org/apache/predictionio/workflow/EngineServerPluginContext.scala
+++ b/core/src/main/scala/org/apache/predictionio/workflow/EngineServerPluginContext.scala
@@ -80,8 +80,9 @@ object EngineServerPluginContext extends Logging {
private def stringFromFile(filePath: String): Option[String] = {
try {
- val fs = FileSystem.get(new Configuration())
- val path = new Path(new URI(filePath))
+ val uri = new URI(filePath)
+ val fs = FileSystem.get(uri, new Configuration())
+ val path = new Path(uri)
if (fs.exists(path)) {
Some(new String(ByteStreams.toByteArray(fs.open(path)).map(_.toChar)))
} else {
[predictionio] 01/04: Merge
Posted by em...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
emergentorder pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/predictionio.git
commit 92cab7a5faaa8c39f46e0f107e67e4aacc97dbe2
Merge: 0faf78a 93f5aa6
Author: Alex Merritt <le...@gmail.com>
AuthorDate: Mon Jun 17 12:18:04 2019 -0400
Merge
.travis.yml | 11 +-
PMC.md | 69 ++++-----
README.md | 4 +-
RELEASE.md | 56 +++++++-
bin/pio-shell | 1 -
bin/pio-start-all | 2 -
build.sbt | 63 ++++----
conf/pio-env.sh.template | 6 -
conf/pio-vendors.sh | 14 +-
.../controller/IdentityPreparator.scala | 2 +-
.../predictionio/controller/LAlgorithm.scala | 6 +-
.../predictionio/controller/LAverageServing.scala | 2 +-
.../predictionio/controller/LDataSource.scala | 4 +-
.../predictionio/controller/LFirstServing.scala | 2 +-
.../predictionio/controller/LPreparator.scala | 2 +-
.../apache/predictionio/controller/LServing.scala | 4 +-
.../LocalFileSystemPersistentModel.scala | 4 +-
.../apache/predictionio/controller/Metric.scala | 22 +--
.../predictionio/controller/MetricEvaluator.scala | 2 +-
.../predictionio/controller/P2LAlgorithm.scala | 6 +-
.../predictionio/controller/PAlgorithm.scala | 6 +-
.../predictionio/controller/PDataSource.scala | 4 +-
.../predictionio/controller/PPreparator.scala | 2 +-
.../predictionio/workflow/CleanupFunctions.scala | 4 +-
.../workflow/EngineServerPluginContext.scala | 19 ++-
.../workflow/EngineServerPluginsActor.scala | 2 +-
.../predictionio/workflow/FakeWorkflow.scala | 6 +-
.../predictionio/workflow/JsonExtractor.scala | 9 +-
data/build.sbt | 1 +
.../predictionio/data/store/LEventStore.scala | 138 ++++++++++++++++--
.../data/store/java/LJavaEventStore.scala | 134 ++++++++++++++++-
python/pypio/__init__.py => docker/.ivy2/.keep | 6 -
docker/JUPYTER.md | 160 +++++++++++++++++++++
docker/README.md | 26 +++-
docker/charts/README.md | 59 ++++++++
.../shell.py => docker/charts/postgresql.yaml | 12 +-
.../charts/predictionio/.helmignore | 27 +++-
.../charts/predictionio/Chart.yaml | 15 +-
docker/charts/predictionio/templates/NOTES.txt | 31 ++++
docker/charts/predictionio/templates/_helpers.tpl | 36 +++++
.../predictionio/templates/pio-deployment.yaml | 75 ++++++++++
.../charts/predictionio/templates/pio-service.yaml | 26 +++-
docker/charts/predictionio/values.yaml | 53 +++++++
docker/charts/predictionio_postgresql.yaml | 41 ++++++
.../shell.py => docker/charts/spark/.helmignore | 27 +++-
.../__init__.py => docker/charts/spark/Chart.yaml | 18 ++-
docker/charts/spark/README.md | 99 +++++++++++++
docker/charts/spark/templates/NOTES.txt | 24 ++++
docker/charts/spark/templates/_helpers.tpl | 43 ++++++
.../spark/templates/spark-master-deployment.yaml | 95 ++++++++++++
.../charts/spark/templates/spark-sql-test.yaml | 21 ++-
.../spark/templates/spark-worker-deployment.yaml | 57 ++++++++
.../charts/spark/templates/spark-worker-hpa.yaml | 30 +++-
docker/charts/spark/values.yaml | 63 ++++++++
.../docker-compose.jupyter.yml | 22 ++-
docker/jupyter/Dockerfile | 100 +++++++++++++
.../shell.py => docker/jupyter/fix-permissions | 23 ++-
docker/jupyter/jupyter_notebook_config.py | 18 +++
docker/jupyter/requirements.txt | 14 ++
.../utils.py => docker/jupyter/start-jupyter.sh | 30 ++--
docker/jupyter/start.sh | 81 +++++++++++
docker/pio/Dockerfile | 6 +-
docs/manual/data/versions.yml | 6 +-
.../{install => archived}/launch-aws.html.md.erb | 0
.../community/contribute-documentation.html.md | 2 +-
docs/manual/source/datacollection/plugin.html.md | 4 +-
docs/manual/source/deploy/plugin.html.md | 4 +-
docs/manual/source/index.html.md.erb | 4 +-
.../manual/source/install/config-datastore.html.md | 59 --------
docs/manual/source/install/index.html.md.erb | 11 +-
.../source/install/install-docker.html.md.erb | 127 ++++++++++++++++
.../source/install/install-sourcecode.html.md.erb | 21 ++-
.../shared/install/_elasticsearch.html.erb | 2 +-
.../source/partials/shared/install/_spark.html.erb | 2 +-
docs/manual/source/resources/upgrade.html.md | 106 ++++++++++++--
docs/manual/source/start/index.html.md | 2 +-
.../recommendation/quickstart.html.md.erb | 20 +--
.../predictionio/e2/engine/PythonEngine.scala | 96 +++++++++++++
examples/scala-parallel-classification/README.md | 4 +-
.../add-algorithm/build.sbt | 6 +-
.../add-algorithm/project/assembly.sbt | 2 +-
.../add-algorithm/project/build.properties | 2 +-
.../src/main/scala/NaiveBayesAlgorithm.scala | 2 +
.../src/main/scala/PrecisionEvaluation.scala | 1 +
.../add-algorithm/src/main/scala/Preparator.scala | 1 +
.../src/main/scala/RandomForestAlgorithm.scala | 2 +
.../reading-custom-properties/build.sbt | 6 +-
.../reading-custom-properties/project/assembly.sbt | 2 +-
.../project/build.properties | 2 +-
.../src/main/scala/NaiveBayesAlgorithm.scala | 2 +
.../src/main/scala/PrecisionEvaluation.scala | 1 +
.../src/main/scala/Preparator.scala | 1 +
.../README.md | 4 +-
.../adjust-score/build.sbt | 6 +-
.../adjust-score/project/assembly.sbt | 2 +-
.../adjust-score/project/build.properties | 2 +-
.../src/main/scala/ECommAlgorithm.scala | 2 +
.../adjust-score/src/main/scala/Preparator.scala | 1 +
.../train-with-rate-event/build.sbt | 6 +-
.../train-with-rate-event/project/assembly.sbt | 2 +-
.../train-with-rate-event/project/build.properties | 2 +-
examples/scala-parallel-recommendation/README.md | 4 +-
.../blacklist-items/build.sbt | 6 +-
.../blacklist-items/project/assembly.sbt | 2 +-
.../blacklist-items/project/build.properties | 2 +-
.../src/main/scala/ALSAlgorithm.scala | 2 +
.../blacklist-items/src/main/scala/ALSModel.scala | 1 +
.../src/main/scala/Evaluation.scala | 2 +
.../src/main/scala/Preparator.scala | 1 +
.../customize-data-prep/build.sbt | 6 +-
.../customize-data-prep/project/assembly.sbt | 2 +-
.../customize-data-prep/project/build.properties | 2 +-
.../src/main/scala/ALSAlgorithm.scala | 2 +
.../src/main/scala/ALSModel.scala | 1 +
.../src/main/scala/Evaluation.scala | 2 +
.../src/main/scala/Preparator.scala | 1 +
.../customize-serving/build.sbt | 6 +-
.../customize-serving/project/assembly.sbt | 2 +-
.../customize-serving/project/build.properties | 2 +-
.../src/main/scala/ALSAlgorithm.scala | 2 +
.../src/main/scala/ALSModel.scala | 1 +
.../src/main/scala/Evaluation.scala | 2 +
.../src/main/scala/Preparator.scala | 1 +
.../reading-custom-events/build.sbt | 6 +-
.../reading-custom-events/project/assembly.sbt | 2 +-
.../reading-custom-events/project/build.properties | 2 +-
.../src/main/scala/ALSAlgorithm.scala | 2 +
.../src/main/scala/ALSModel.scala | 1 +
.../src/main/scala/Evaluation.scala | 2 +
.../src/main/scala/Preparator.scala | 1 +
.../train-with-view-event/build.sbt | 6 +-
.../train-with-view-event/project/assembly.sbt | 2 +-
.../train-with-view-event/project/build.properties | 2 +-
.../src/main/scala/ALSAlgorithm.scala | 2 +
.../src/main/scala/ALSModel.scala | 1 +
.../src/main/scala/Evaluation.scala | 2 +
.../src/main/scala/Preparator.scala | 1 +
examples/scala-parallel-similarproduct/README.md | 4 +-
.../multi-events-multi-algos/build.sbt | 6 +-
.../multi-events-multi-algos/project/assembly.sbt | 2 +-
.../project/build.properties | 2 +-
.../src/main/scala/ALSAlgorithm.scala | 2 +
.../src/main/scala/CooccurrenceAlgorithm.scala | 2 +
.../src/main/scala/Preparator.scala | 1 +
.../recommended-user/build.sbt | 6 +-
.../recommended-user/project/assembly.sbt | 2 +-
.../recommended-user/project/build.properties | 2 +-
.../src/main/scala/ALSAlgorithm.scala | 2 +
.../src/main/scala/Preparator.scala | 1 +
.../return-item-properties/build.sbt | 6 +-
.../return-item-properties/project/assembly.sbt | 2 +-
.../project/build.properties | 2 +-
.../src/main/scala/ALSAlgorithm.scala | 2 +
.../src/main/scala/CooccurrenceAlgorithm.scala | 2 +
.../src/main/scala/Preparator.scala | 1 +
.../rid-user-set-event/build.sbt | 6 +-
.../rid-user-set-event/project/assembly.sbt | 2 +-
.../rid-user-set-event/project/build.properties | 2 +-
.../src/main/scala/ALSAlgorithm.scala | 2 +
.../src/main/scala/CooccurrenceAlgorithm.scala | 2 +
.../src/main/scala/Preparator.scala | 1 +
.../train-with-rate-event/build.sbt | 6 +-
.../train-with-rate-event/project/assembly.sbt | 2 +-
.../train-with-rate-event/project/build.properties | 2 +-
.../src/main/scala/ALSAlgorithm.scala | 2 +
.../src/main/scala/CooccurrenceAlgorithm.scala | 2 +
.../src/main/scala/Preparator.scala | 1 +
project/PIOBuild.scala | 1 -
project/plugins.sbt | 2 +-
python/pypio/__init__.py | 9 ++
python/pypio/data/eventstore.py | 4 +-
python/pypio/pypio.py | 111 ++++++++++++++
python/pypio/utils.py | 40 +++++-
storage/elasticsearch/build.sbt | 6 +-
.../data/storage/elasticsearch/ESAccessKeys.scala | 5 +-
.../data/storage/elasticsearch/ESApps.scala | 8 +-
.../data/storage/elasticsearch/ESChannels.scala | 7 +-
.../storage/elasticsearch/ESEngineInstances.scala | 42 +++---
.../elasticsearch/ESEvaluationInstances.scala | 8 +-
.../data/storage/elasticsearch/ESLEvents.scala | 5 +-
.../data/storage/elasticsearch/ESSequences.scala | 6 +-
.../data/storage/elasticsearch/ESUtils.scala | 20 +--
storage/elasticsearch1/.gitignore | 1 -
storage/elasticsearch1/build.sbt | 38 -----
.../data/storage/elasticsearch/ESAccessKeys.scala | 119 ---------------
.../data/storage/elasticsearch/ESApps.scala | 130 -----------------
.../data/storage/elasticsearch/ESChannels.scala | 117 ---------------
.../storage/elasticsearch/ESEngineInstances.scala | 158 --------------------
.../elasticsearch/ESEvaluationInstances.scala | 136 ------------------
.../data/storage/elasticsearch/ESSequences.scala | 64 ---------
.../data/storage/elasticsearch/ESUtils.scala | 48 -------
.../data/storage/elasticsearch/StorageClient.scala | 50 -------
.../data/storage/elasticsearch/package.scala | 25 ----
.../src/test/resources/application.conf | 28 ----
.../data/storage/s3/StorageClient.scala | 8 +-
tests/docker-files/env-conf/pio-env.sh | 11 +-
.../engines/recommendation-engine/build.sbt | 4 +-
.../apache/predictionio/tools/RunWorkflow.scala | 32 ++---
.../predictionio/tools/commands/Engine.scala | 27 ++--
199 files changed, 2418 insertions(+), 1485 deletions(-)
diff --cc .travis.yml
index 42662de,c807165..13e6ec3
--- a/.travis.yml
+++ b/.travis.yml
@@@ -99,7 -96,12 +96,7 @@@ env
- BUILD_TYPE=Integration
METADATA_REP=PGSQL EVENTDATA_REP=PGSQL MODELDATA_REP=HDFS
PIO_SCALA_VERSION=2.11.12
- PIO_SPARK_VERSION=2.3.2
- PIO_SPARK_VERSION=2.3.3
- PIO_HADOOP_VERSION=2.7.7
- - BUILD_TYPE=Integration
- METADATA_REP=PGSQL EVENTDATA_REP=PGSQL MODELDATA_REP=HDFS
- PIO_SCALA_VERSION=2.11.12
+ PIO_SPARK_VERSION=2.4.0
PIO_HADOOP_VERSION=2.7.7
- BUILD_TYPE=LicenseCheck
diff --cc build.sbt
index 9efed21,082521b..3e063bf
--- a/build.sbt
+++ b/build.sbt
@@@ -16,28 -16,9 +16,28 @@@
*/
import PIOBuild._
+lazy val scalaSparkDepsVersion = Map(
+ "2.11" -> Map(
+ "2.0" -> Map(
+ "akka" -> "2.5.16",
+ "hadoop" -> "2.7.7",
+ "json4s" -> "3.2.11"),
+ "2.1" -> Map(
+ "akka" -> "2.5.17",
+ "hadoop" -> "2.7.7",
+ "json4s" -> "3.2.11"),
+ "2.2" -> Map(
+ "akka" -> "2.5.17",
+ "hadoop" -> "2.7.7",
+ "json4s" -> "3.2.11"),
+ "2.3" -> Map(
+ "akka" -> "2.5.17",
+ "hadoop" -> "2.7.7",
+ "json4s" -> "3.2.11")))
+
name := "apache-predictionio-parent"
- version in ThisBuild := "0.14.0-SNAPSHOT"
+ version in ThisBuild := "0.15.0-SNAPSHOT"
organization in ThisBuild := "org.apache.predictionio"
diff --cc docs/manual/source/install/install-sourcecode.html.md.erb
index 9a1e456,60f8772..b3e4149
--- a/docs/manual/source/install/install-sourcecode.html.md.erb
+++ b/docs/manual/source/install/install-sourcecode.html.md.erb
@@@ -100,13 -100,10 +100,15 @@@ and complete information
### Building
Run the following at the directory where you downloaded the source code to build
-Apache PredictionIO®.
-As an example, if you want to build PredictionIO to support Scala 2.11.12,
-Spark 2.4.0, and Elasticsearch 6.4.2, you can do
++
+Apache PredictionIO®. By default, the build will be against
+
+* Scala 2.11.8
+* Spark 2.1.1
+* Hadoop 2.7.7
+* Elasticsearch 5.6.9
+
+
```
$ tar zxvf apache-predictionio-<%= data.versions.pio %>.tar.gz
$ cd apache-predictionio-<%= data.versions.pio %>
[predictionio] 03/04: Set engine id, version, variant,
even in Python
Posted by em...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
emergentorder pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/predictionio.git
commit 1094de8322285a572b2d2e8ae49073d1066242d2
Author: Alex Merritt <le...@gmail.com>
AuthorDate: Thu Jul 25 19:19:01 2019 -0300
Set engine id, version, variant, even in Python
---
tools/src/main/scala/org/apache/predictionio/tools/RunWorkflow.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/src/main/scala/org/apache/predictionio/tools/RunWorkflow.scala b/tools/src/main/scala/org/apache/predictionio/tools/RunWorkflow.scala
index 50b9337..701a3ee 100644
--- a/tools/src/main/scala/org/apache/predictionio/tools/RunWorkflow.scala
+++ b/tools/src/main/scala/org/apache/predictionio/tools/RunWorkflow.scala
@@ -52,14 +52,14 @@ object RunWorkflow extends Logging {
val jarFiles = jarFilesForScala(engineDirPath).map(_.toURI)
val args =
- (if (wa.mainPyFile.isEmpty) {
+ {
val variantJson = wa.variantJson.getOrElse(new File(engineDirPath, "engine.json"))
val ei = Console.getEngineInfo(variantJson, engineDirPath)
Seq(
"--engine-id", ei.engineId,
"--engine-version", ei.engineVersion,
"--engine-variant", variantJson.toURI.toString)
- } else Nil) ++
+ } ++
wa.engineFactory.map(
x => Seq("--engine-factory", x)).getOrElse(Nil) ++
wa.engineParamsKey.map(