You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@predictionio.apache.org by ch...@apache.org on 2018/03/01 21:50:36 UTC
[1/5] predictionio git commit: [PIO-151] Add S3 storage docs [Forced
Update!]
Repository: predictionio
Updated Branches:
refs/heads/develop 8999cdef5 -> 161bc0ef5 (forced update)
[PIO-151] Add S3 storage docs
This closes #449
Project: http://git-wip-us.apache.org/repos/asf/predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/predictionio/commit/c1270e23
Tree: http://git-wip-us.apache.org/repos/asf/predictionio/tree/c1270e23
Diff: http://git-wip-us.apache.org/repos/asf/predictionio/diff/c1270e23
Branch: refs/heads/develop
Commit: c1270e23b00ff459e76a4e9dde589bb2e1d0d0db
Parents: e156cb4
Author: James Ward <ja...@jamesward.org>
Authored: Thu Mar 1 13:26:35 2018 -0800
Committer: Chan Lee <ch...@gmail.com>
Committed: Thu Mar 1 13:48:59 2018 -0800
----------------------------------------------------------------------
.../source/system/anotherdatastore.html.md | 43 ++++++++++++++++++++
1 file changed, 43 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/predictionio/blob/c1270e23/docs/manual/source/system/anotherdatastore.html.md
----------------------------------------------------------------------
diff --git a/docs/manual/source/system/anotherdatastore.html.md b/docs/manual/source/system/anotherdatastore.html.md
index b3a3709..071a197 100644
--- a/docs/manual/source/system/anotherdatastore.html.md
+++ b/docs/manual/source/system/anotherdatastore.html.md
@@ -114,6 +114,10 @@ PredictionIO comes with the following sources:
* Type name is **hdfs**.
* Can be used for *Model Data* repository
+- **S3**:
+ * Type name is **s3**.
+ * Can be used for *Model Data* repository
+
Each repository can be configured to use different sources as shown above.
Each source has its own set of configuration parameters. Configuration variables will be explained in more details in later sections below (see Data Store Configuration).
@@ -293,6 +297,45 @@ supported.
`PIO_STORAGE_SOURCES_HDFS_PATH=/mymodels`
+#### S3 Configuration
+
+Variable Format: `PIO_STORAGE_SOURCES_[NAME]_TYPE=s3`
+
+Supported Repositories: **model**
+
+To provide authentication information, you can set the `AWS_ACCESS_KEY_ID`
+and `AWS_SECRET_ACCESS_KEY` environment variables or use one of the other
+methods in the [AWS Setup Docs](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#config-settings-and-precedence)
+
+When `TYPE` is set to `s3`, the following configuration keys are
+supported.
+
+- REGION (mandatory)
+
+ AWS Region to use, e.g.
+ `PIO_STORAGE_SOURCES_S3_REGION=us-east-1`
+
+- BUCKET_NAME (mandatory)
+
+ S3 Bucket where models are stored, e.g.
+ `PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket`
+
+- BASE_PATH (optional)
+
+ S3 base path where models are stored, e.g.
+ `PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model`
+
+- DISABLE_CHUNKED_ENCODING (optional)
+
+ Disable the use of Chunked Encoding when transferring files to/from S3, e.g.
+ `PIO_STORAGE_SOURCES_S3_DISABLE_CHUNKED_ENCODING=true`
+
+- ENDPOINT (optional)
+
+ S3 Endpoint to use, e.g.
+ `PIO_STORAGE_SOURCES_S3_ENDPOINT=http://localstack:4572`
+
+
## Adding Support of Other Backends
It is quite straightforward to implement support of other backends. A good
[3/5] predictionio git commit: Disable gzip to avoid large commits on
site repo
Posted by ch...@apache.org.
Disable gzip to avoid large commits on site repo
Project: http://git-wip-us.apache.org/repos/asf/predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/predictionio/commit/a59ef970
Tree: http://git-wip-us.apache.org/repos/asf/predictionio/tree/a59ef970
Diff: http://git-wip-us.apache.org/repos/asf/predictionio/diff/a59ef970
Branch: refs/heads/develop
Commit: a59ef970763d49112e0888739cc8b9576857e034
Parents: c2528cb
Author: Donald Szeto <do...@apache.org>
Authored: Thu Mar 1 10:43:05 2018 -0800
Committer: Chan Lee <ch...@gmail.com>
Committed: Thu Mar 1 13:49:09 2018 -0800
----------------------------------------------------------------------
docs/manual/config.rb | 1 -
1 file changed, 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/predictionio/blob/a59ef970/docs/manual/config.rb
----------------------------------------------------------------------
diff --git a/docs/manual/config.rb b/docs/manual/config.rb
index 26d4b65..d836133 100644
--- a/docs/manual/config.rb
+++ b/docs/manual/config.rb
@@ -25,7 +25,6 @@ set :images_dir, 'images'
set :partials_dir, 'partials'
activate :directory_indexes
-activate :gzip
activate :syntax, line_numbers: true
activate :autoprefixer
[2/5] predictionio git commit: [PIO-146] Change TM to (R) on text
marks
Posted by ch...@apache.org.
[PIO-146] Change TM to (R) on text marks
This is a requirement of ASF
Project: http://git-wip-us.apache.org/repos/asf/predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/predictionio/commit/c2528cb3
Tree: http://git-wip-us.apache.org/repos/asf/predictionio/tree/c2528cb3
Diff: http://git-wip-us.apache.org/repos/asf/predictionio/diff/c2528cb3
Branch: refs/heads/develop
Commit: c2528cb30128830f063499191ae6f49c5ebba44c
Parents: f1f8723
Author: Donald Szeto <do...@apache.org>
Authored: Thu Mar 1 10:33:12 2018 -0800
Committer: Chan Lee <ch...@gmail.com>
Committed: Thu Mar 1 13:49:09 2018 -0800
----------------------------------------------------------------------
docs/manual/data/nav/main.yml | 6 +++---
docs/manual/source/index.html.md.erb | 12 ++++++------
2 files changed, 9 insertions(+), 9 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/predictionio/blob/c2528cb3/docs/manual/data/nav/main.yml
----------------------------------------------------------------------
diff --git a/docs/manual/data/nav/main.yml b/docs/manual/data/nav/main.yml
index 6973e39..24ad8d6 100644
--- a/docs/manual/data/nav/main.yml
+++ b/docs/manual/data/nav/main.yml
@@ -1,10 +1,10 @@
root:
-
- body: 'Apache PredictionIO™ Documentation'
+ body: 'Apache PredictionIO® Documentation'
url: '/'
children:
-
- body: 'Welcome to Apache PredictionIO'
+ body: 'Welcome to Apache PredictionIO®'
url: '/'
-
body: 'Getting Started'
@@ -163,7 +163,7 @@ root:
body: 'Using Another Data Store'
url: '/system/anotherdatastore/'
-
- body: 'PredictionIO Official Templates'
+ body: 'PredictionIO® Official Templates'
url: '#'
children:
-
http://git-wip-us.apache.org/repos/asf/predictionio/blob/c2528cb3/docs/manual/source/index.html.md.erb
----------------------------------------------------------------------
diff --git a/docs/manual/source/index.html.md.erb b/docs/manual/source/index.html.md.erb
index cef8e7a..09eaed8 100644
--- a/docs/manual/source/index.html.md.erb
+++ b/docs/manual/source/index.html.md.erb
@@ -1,6 +1,6 @@
---
-title: Welcome to Apache PredictionIO™!
-description: Apache PredictionIO™ Open Source Machine Learning Server
+title: Welcome to Apache PredictionIO®!
+description: Apache PredictionIO® Open Source Machine Learning Server
---
<!--
@@ -20,9 +20,9 @@ See the License for the specific language governing permissions and
limitations under the License.
-->
-## What is Apache PredictionIO?
+## What is Apache PredictionIO®?
-Apache PredictionIO is an **open source Machine Learning Server**
+Apache PredictionIO® is an **open source Machine Learning Server**
built on top of a state-of-the-art open source stack for developers and data
scientists to create predictive engines for any machine learning task. It lets you:
@@ -40,7 +40,7 @@ scientists to create predictive engines for any machine learning task. It lets y
into your engine;
* simplify data infrastructure management.
-Apache PredictionIO can be [installed](/install/) as a full machine
+Apache PredictionIO® can be [installed](/install/) as a full machine
learning stack, bundled with **Apache Spark**, **MLlib**, **HBase**, **Spray**
and **Elasticsearch**, which simplifies and accelerates scalable machine
learning infrastructure management.
@@ -65,7 +65,7 @@ Foundation are on GitHub
## Licensing
-Apache PredictionIO is licensed under the Apache License, Version
+Apache PredictionIO® is licensed under the Apache License, Version
2.0. See
[LICENSE](https://github.com/apache/predictionio/blob/master/LICENSE.txt)
for the full license text.
[5/5] predictionio git commit: [PIO-147] Fix broken Scala API
documentation
Posted by ch...@apache.org.
[PIO-147] Fix broken Scala API documentation
Project: http://git-wip-us.apache.org/repos/asf/predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/predictionio/commit/f1f8723f
Tree: http://git-wip-us.apache.org/repos/asf/predictionio/tree/f1f8723f
Diff: http://git-wip-us.apache.org/repos/asf/predictionio/diff/f1f8723f
Branch: refs/heads/develop
Commit: f1f8723f0389391ad9dc06bab21bc38b9705c137
Parents: c1270e2
Author: Donald Szeto <do...@apache.org>
Authored: Mon Feb 12 15:56:53 2018 -0800
Committer: Chan Lee <ch...@gmail.com>
Committed: Thu Mar 1 13:49:09 2018 -0800
----------------------------------------------------------------------
build.sbt | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/predictionio/blob/f1f8723f/build.sbt
----------------------------------------------------------------------
diff --git a/build.sbt b/build.sbt
index 6eee3f8..7a81249 100644
--- a/build.sbt
+++ b/build.sbt
@@ -199,8 +199,8 @@ val root = (project in file(".")).
settings(commonSettings: _*).
enablePlugins(ScalaUnidocPlugin).
settings(
- unidocProjectFilter in (ScalaUnidoc, unidoc) := inAnyProject -- inProjects(dataElasticsearch),
- unidocProjectFilter in (JavaUnidoc, unidoc) := inAnyProject -- inProjects(dataElasticsearch),
+ unidocProjectFilter in (ScalaUnidoc, unidoc) := inAnyProject -- inProjects(dataElasticsearch, dataElasticsearch1),
+ unidocProjectFilter in (JavaUnidoc, unidoc) := inAnyProject -- inProjects(dataElasticsearch, dataElasticsearch1),
scalacOptions in (ScalaUnidoc, unidoc) ++= Seq(
"-groups",
"-skip-packages",
@@ -212,8 +212,16 @@ val root = (project in file(".")).
"org.apache.predictionio.controller.html",
"org.apache.predictionio.controller.java",
"org.apache.predictionio.data.api",
+ "org.apache.predictionio.data.storage.*",
+ "org.apache.predictionio.data.storage.hdfs",
+ "org.apache.predictionio.data.storage.jdbc",
+ "org.apache.predictionio.data.storage.localfs",
+ "org.apache.predictionio.data.storage.s3",
+ "org.apache.predictionio.data.storage.hbase",
"org.apache.predictionio.data.view",
+ "org.apache.predictionio.data.webhooks",
"org.apache.predictionio.tools",
+ "org.apache.predictionio.workflow.html",
"scalikejdbc").mkString(":"),
"-doc-title",
"PredictionIO Scala API",
[4/5] predictionio git commit: [PIO-136] Add CleanupFunctions for
Python
Posted by ch...@apache.org.
[PIO-136] Add CleanupFunctions for Python
This closes #444
Project: http://git-wip-us.apache.org/repos/asf/predictionio/repo
Commit: http://git-wip-us.apache.org/repos/asf/predictionio/commit/161bc0ef
Tree: http://git-wip-us.apache.org/repos/asf/predictionio/tree/161bc0ef
Diff: http://git-wip-us.apache.org/repos/asf/predictionio/diff/161bc0ef
Branch: refs/heads/develop
Commit: 161bc0ef54421d7aa7eab4da98794e14793bf3f4
Parents: a59ef97
Author: Shinsuke Sugaya <sh...@yahoo.co.jp>
Authored: Thu Mar 1 13:44:43 2018 -0800
Committer: Chan Lee <ch...@gmail.com>
Committed: Thu Mar 1 13:49:09 2018 -0800
----------------------------------------------------------------------
python/pypio/shell.py | 2 ++
python/pypio/workflow/__init__.py | 25 +++++++++++++++++++
python/pypio/workflow/cleanup_functions.py | 32 +++++++++++++++++++++++++
3 files changed, 59 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/predictionio/blob/161bc0ef/python/pypio/shell.py
----------------------------------------------------------------------
diff --git a/python/pypio/shell.py b/python/pypio/shell.py
index 94c1e1a..b0295d3 100644
--- a/python/pypio/shell.py
+++ b/python/pypio/shell.py
@@ -16,6 +16,8 @@
#
from pypio.data import PEventStore
+from pypio.workflow import CleanupFunctions
p_event_store = PEventStore(spark._jsparkSession, sqlContext)
+cleanup_functions = CleanupFunctions(sqlContext)
http://git-wip-us.apache.org/repos/asf/predictionio/blob/161bc0ef/python/pypio/workflow/__init__.py
----------------------------------------------------------------------
diff --git a/python/pypio/workflow/__init__.py b/python/pypio/workflow/__init__.py
new file mode 100644
index 0000000..3504e3a
--- /dev/null
+++ b/python/pypio/workflow/__init__.py
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+from pypio.workflow.cleanup_functions import CleanupFunctions
+
+
+__all__ = [
+ 'CleanupFunctions'
+]
http://git-wip-us.apache.org/repos/asf/predictionio/blob/161bc0ef/python/pypio/workflow/cleanup_functions.py
----------------------------------------------------------------------
diff --git a/python/pypio/workflow/cleanup_functions.py b/python/pypio/workflow/cleanup_functions.py
new file mode 100644
index 0000000..e4eeb5a
--- /dev/null
+++ b/python/pypio/workflow/cleanup_functions.py
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+
+__all__ = ["CleanupFunctions"]
+
+
+class CleanupFunctions(object):
+
+ def __init__(self, sql_ctx):
+ self.sql_ctx = sql_ctx
+ self._sc = sql_ctx and sql_ctx._sc
+
+ def run(self):
+ cf = self._sc._jvm.org.apache.predictionio.workflow.CleanupFunctions
+ cf.run()
+