You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@predictionio.apache.org by sh...@apache.org on 2018/10/10 07:31:04 UTC
[predictionio] branch develop updated: [PIO-86] Docker support
(#462)
This is an automated email from the ASF dual-hosted git repository.
shinsuke pushed a commit to branch develop
in repository https://gitbox.apache.org/repos/asf/predictionio.git
The following commit(s) were added to refs/heads/develop by this push:
new dc08628 [PIO-86] Docker support (#462)
dc08628 is described below
commit dc08628eb6489e594af369310bb1c9d36c81d15a
Author: Shinsuke Sugaya <sh...@gmail.com>
AuthorDate: Wed Oct 10 16:30:57 2018 +0900
[PIO-86] Docker support (#462)
---
docker/README.md | 249 ++++++++++++++++++++++++++
docker/bin/pio-docker | 56 ++++++
docker/deploy/run.sh | 21 +++
docker/docker-compose.deploy.yml | 23 +++
docker/docker-compose.spark.yml | 34 ++++
docker/docker-compose.yml | 25 +++
docker/elasticsearch/docker-compose.base.yml | 41 +++++
docker/elasticsearch/docker-compose.event.yml | 22 +++
docker/elasticsearch/docker-compose.meta.yml | 22 +++
docker/localfs/docker-compose.model.yml | 27 +++
docker/mysql/docker-compose.base.yml | 34 ++++
docker/mysql/docker-compose.event.yml | 22 +++
docker/mysql/docker-compose.meta.yml | 22 +++
docker/mysql/docker-compose.model.yml | 22 +++
docker/pgsql/docker-compose.base.yml | 32 ++++
docker/pgsql/docker-compose.event.yml | 22 +++
docker/pgsql/docker-compose.meta.yml | 22 +++
docker/pgsql/docker-compose.model.yml | 22 +++
docker/pio/Dockerfile | 69 +++++++
docker/pio/pio_run | 64 +++++++
docker/templates/.keep | 14 ++
21 files changed, 865 insertions(+)
diff --git a/docker/README.md b/docker/README.md
new file mode 100644
index 0000000..f1bc5a0
--- /dev/null
+++ b/docker/README.md
@@ -0,0 +1,249 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+Apache PredictionIO Docker
+==========================
+
+## Overview
+
+PredictionIO Docker provides Docker image for use in development and production environment.
+
+
+## Usage
+
+### Run PredictionIO with Selectable docker-compose Files
+
+You can choose storages for event/meta/model to select docker-compose.yml.
+
+```
+docker-compose -f docker-compose.yml -f ... up
+```
+
+Supported storages are as below:
+
+| Type | Storage |
+|:-----:|:---------------------------------|
+| Event | Postgresql, MySQL, Elasticsearch |
+| Meta | Postgresql, MySQL, Elasticsearch |
+| Model | Postgresql, MySQL, LocalFS |
+
+If you run PredictionIO with Postgresql, run as below:
+
+```
+docker-compose -f docker-compose.yml \
+ -f pgsql/docker-compose.base.yml \
+ -f pgsql/docker-compose.meta.yml \
+ -f pgsql/docker-compose.event.yml \
+ -f pgsql/docker-compose.model.yml \
+ up
+```
+
+To use localfs as model storage, change as below:
+
+```
+docker-compose -f docker-compose.yml \
+ -f pgsql/docker-compose.base.yml \
+ -f pgsql/docker-compose.meta.yml \
+ -f pgsql/docker-compose.event.yml \
+ -f localfs/docker-compose.model.yml \
+ up
+```
+
+## Tutorial
+
+In this demo, we will show you how to build a recommendation template.
+
+### Run PredictionIO environment
+
+The following command starts PredictionIO with an event server.
+PredictionIO docker image mounts ./templates directory to /templates.
+
+```
+$ docker-compose -f docker-compose.yml \
+ -f pgsql/docker-compose.base.yml \
+ -f pgsql/docker-compose.meta.yml \
+ -f pgsql/docker-compose.event.yml \
+ -f pgsql/docker-compose.model.yml \
+ up
+```
+
+We provide `pio-docker` command as an utility for `pio` command.
+`pio-docker` invokes `pio` command in PredictionIO container.
+
+```
+$ export PATH=`pwd`/bin:$PATH
+$ pio-docker status
+...
+[INFO] [Management$] Your system is all ready to go.
+```
+
+### Download Recommendation Template
+
+This demo uses [predictionio-template-recommender](https://github.com/apache/predictionio-template-recommender).
+
+```
+$ cd templates
+$ git clone https://github.com/apache/predictionio-template-recommender.git MyRecommendation
+$ cd MyRecommendation
+```
+
+### Register Application
+
+You need to register this application to PredictionIO:
+
+```
+$ pio-docker app new MyApp1
+[INFO] [App$] Initialized Event Store for this app ID: 1.
+[INFO] [Pio$] Created a new app:
+[INFO] [Pio$] Name: MyApp1
+[INFO] [Pio$] ID: 1
+[INFO] [Pio$] Access Key: i-zc4EleEM577EJhx3CzQhZZ0NnjBKKdSbp3MiR5JDb2zdTKKzH9nF6KLqjlMnvl
+```
+
+Since an access key is required in subsequent steps, set it to ACCESS_KEY.
+
+```
+$ ACCESS_KEY=i-zc4EleEM577EJhx3CzQhZZ0NnjBKKdSbp3MiR5JDb2zdTKKzH9nF6KLqjlMnvl
+```
+
+`engine.json` contains an application name, so replace `INVALID_APP_NAME` with `MyApp1`.
+
+```
+...
+"datasource": {
+ "params" : {
+ "appName": "MyApp1"
+ }
+},
+...
+```
+
+### Import Data
+
+To import training data to Event server for PredictionIO, this template provides an import tool.
+The tool depends on PredictionIO Python SDK and install as below:
+
+```
+$ pip install predictionio
+```
+and then import data:
+```
+$ curl https://raw.githubusercontent.com/apache/spark/master/data/mllib/sample_movielens_data.txt --create-dirs -o data/sample_movielens_data.txt
+$ python data/import_eventserver.py --access_key $ACCESS_KEY
+```
+
+### Build Template
+
+This is Scala based template.
+So, you need to build this template by `pio` command.
+
+```
+$ pio-docker build --verbose
+```
+
+### Train and Create Model
+
+To train a recommendation model, run `train` sub-command:
+
+```
+$ pio-docker train
+```
+
+### Deploy Model
+
+If a recommendation model is created successfully, deploy it to Prediction server for PredictionIO.
+
+```
+$ pio-docker deploy
+
+```
+You can check predictions as below:
+```
+$ curl -H "Content-Type: application/json" \
+-d '{ "user": "1", "num": 4 }' http://localhost:8000/queries.json
+```
+
+## Advanced Topics
+
+### Run with Elasticsearch
+
+For Elasticsearch, Meta and Event storage are available.
+To start PredictionIO with Elasticsearch,
+
+```
+docker-compose -f docker-compose.yml \
+ -f elasticsearch/docker-compose.base.yml \
+ -f elasticsearch/docker-compose.meta.yml \
+ -f elasticsearch/docker-compose.event.yml \
+ -f localfs/docker-compose.model.yml \
+ up
+```
+
+### Run with Spark Cluster
+
+Adding `docker-compose.spark.yml`, you can use Spark cluster on `pio train`.
+
+```
+docker-compose -f docker-compose.yml \
+ -f docker-compose.spark.yml \
+ -f elasticsearch/docker-compose.base.yml \
+ -f elasticsearch/docker-compose.meta.yml \
+ -f elasticsearch/docker-compose.event.yml \
+ -f localfs/docker-compose.model.yml \
+ up
+```
+
+To submit a training task to Spark Cluster, run `pio-deploy train` with `--master` option:
+
+```
+pio-docker train -- --master spark://spark-master:7077
+```
+
+See `docker-compose.spark.yml` if changing settings for Spark Cluster.
+
+### Run Engine Server
+
+To deploy your engine and start an engine server, run Docker with `docker-compose.deploy.yml`.
+
+```
+docker-compose -f docker-compose.yml \
+ -f pgsql/docker-compose.base.yml \
+ -f pgsql/docker-compose.meta.yml \
+ -f pgsql/docker-compose.event.yml \
+ -f pgsql/docker-compose.model.yml \
+ -f docker-compose.deploy.yml \
+ up
+```
+
+See `deploy/run.sh` and `docker-compose.deploy.yml` if changing a deployment.
+
+
+## Development
+
+### Build Docker Image
+
+```
+docker build -t predictionio/pio pio
+```
+
+### Push Docker Image
+
+```
+docker push predictionio/pio:latest
+docker tag predictionio/pio:latest predictionio/pio:$PIO\_VERSION
+docker push predictionio/pio:$PIO\_VERSION
+```
diff --git a/docker/bin/pio-docker b/docker/bin/pio-docker
new file mode 100755
index 0000000..8046060
--- /dev/null
+++ b/docker/bin/pio-docker
@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+BASE_WORK_DIR=/templates
+CURRENT_DIR=`pwd`
+
+get_container_id() {
+ if [ x"$PIO_CONTAINER_ID" != "x" ] ; then
+ echo $PIO_CONTAINER_ID
+ return
+ fi
+ for i in `docker ps -f "name=pio" -q` ; do
+ echo $i
+ return
+ done
+}
+
+get_current_dir() {
+ if [ x"$PIO_CURRENT_DIR" != "x" ] ; then
+ echo $PIO_CURRENT_DIR
+ return
+ fi
+ D=`echo $CURRENT_DIR | sed -e "s,.*$BASE_WORK_DIR,$BASE_WORK_DIR,"`
+ if [[ $D = $BASE_WORK_DIR* ]] ; then
+ echo $D
+ else
+ echo $BASE_WORK_DIR
+ fi
+}
+
+cid=`get_container_id`
+if [ x"$cid" = "x" ] ; then
+ echo "Docker Container is not found."
+ exit 1
+fi
+
+wdir=`get_current_dir`
+
+docker exec -w $wdir -it $cid pio $@
+
diff --git a/docker/deploy/run.sh b/docker/deploy/run.sh
new file mode 100644
index 0000000..f225a62
--- /dev/null
+++ b/docker/deploy/run.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+cd /templates/$PIO_TEMPLATE_NAME
+pio deploy
diff --git a/docker/docker-compose.deploy.yml b/docker/docker-compose.deploy.yml
new file mode 100644
index 0000000..145a960
--- /dev/null
+++ b/docker/docker-compose.deploy.yml
@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ pio:
+ environment:
+ - "PIO_TEMPLATE_NAME=MyRecommendation"
+ - "PIO_RUN_FILE=/deploy/run.sh"
+ volumes:
+ - ./deploy:/deploy
diff --git a/docker/docker-compose.spark.yml b/docker/docker-compose.spark.yml
new file mode 100644
index 0000000..8f3c43e
--- /dev/null
+++ b/docker/docker-compose.spark.yml
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ spark-master:
+ image: bde2020/spark-master:2.2.2-hadoop2.7
+ container_name: spark-master
+ ports:
+ - "8080:8080"
+ - "7077:7077"
+ environment:
+ - INIT_DAEMON_STEP=setup_spark
+ spark-worker-1:
+ image: bde2020/spark-worker:2.2.2-hadoop2.7
+ container_name: spark-worker-1
+ depends_on:
+ - spark-master
+ ports:
+ - "8081:8081"
+ environment:
+ - "SPARK_MASTER=spark://spark-master:7077"
diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml
new file mode 100644
index 0000000..e42c907
--- /dev/null
+++ b/docker/docker-compose.yml
@@ -0,0 +1,25 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ pio:
+ image: predictionio/pio:latest
+ ports:
+ - 7070:7070
+ - 8000:8000
+ volumes:
+ - ./templates:/templates
+ dns: 8.8.8.8
diff --git a/docker/elasticsearch/docker-compose.base.yml b/docker/elasticsearch/docker-compose.base.yml
new file mode 100644
index 0000000..4784f4a
--- /dev/null
+++ b/docker/elasticsearch/docker-compose.base.yml
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ elasticsearch:
+ image: docker.elastic.co/elasticsearch/elasticsearch:5.6.4
+ environment:
+ - xpack.graph.enabled=false
+ - xpack.ml.enabled=false
+ - xpack.monitoring.enabled=false
+ - xpack.security.enabled=false
+ - xpack.watcher.enabled=false
+ - cluster.name=predictionio
+ - bootstrap.memory_lock=true
+ - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
+ ulimits:
+ memlock:
+ soft: -1
+ hard: -1
+ pio:
+ depends_on:
+ - elasticsearch
+ environment:
+ PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE: elasticsearch
+ PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS: elasticsearch
+ PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS: 9200
+ PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES: http
+
diff --git a/docker/elasticsearch/docker-compose.event.yml b/docker/elasticsearch/docker-compose.event.yml
new file mode 100644
index 0000000..5a77b6a
--- /dev/null
+++ b/docker/elasticsearch/docker-compose.event.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ pio:
+ environment:
+ PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME: pio_event
+ PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE: ELASTICSEARCH
+
diff --git a/docker/elasticsearch/docker-compose.meta.yml b/docker/elasticsearch/docker-compose.meta.yml
new file mode 100644
index 0000000..0ce31dd
--- /dev/null
+++ b/docker/elasticsearch/docker-compose.meta.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ pio:
+ environment:
+ PIO_STORAGE_REPOSITORIES_METADATA_NAME: pio_meta
+ PIO_STORAGE_REPOSITORIES_METADATA_SOURCE: ELASTICSEARCH
+
diff --git a/docker/localfs/docker-compose.model.yml b/docker/localfs/docker-compose.model.yml
new file mode 100644
index 0000000..f38ff53
--- /dev/null
+++ b/docker/localfs/docker-compose.model.yml
@@ -0,0 +1,27 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ pio:
+ environment:
+ PIO_STORAGE_REPOSITORIES_MODELDATA_NAME: pio_model
+ PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE: LOCALFS
+ PIO_FS_BASEDIR: /work/pio_store
+ PIO_FS_ENGINESDIR: /work/pio_store/engines
+ PIO_FS_TMPDIR: /work/pio_store/tmp
+ PIO_STORAGE_SOURCES_LOCALFS_TYPE: localfs
+ PIO_STORAGE_SOURCES_LOCALFS_PATH: /work/pio_store/models
+
diff --git a/docker/mysql/docker-compose.base.yml b/docker/mysql/docker-compose.base.yml
new file mode 100644
index 0000000..fad5309
--- /dev/null
+++ b/docker/mysql/docker-compose.base.yml
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ mysql:
+ image: mysql:8
+ command: mysqld --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
+ environment:
+ MYSQL_ROOT_PASSWORD: root
+ MYSQL_USER: pio
+ MYSQL_PASSWORD: pio
+ MYSQL_DATABASE: pio
+ pio:
+ depends_on:
+ - mysql
+ environment:
+ PIO_STORAGE_SOURCES_MYSQL_TYPE: jdbc
+ PIO_STORAGE_SOURCES_MYSQL_URL: "jdbc:mysql://mysql/pio"
+ PIO_STORAGE_SOURCES_MYSQL_USERNAME: pio
+ PIO_STORAGE_SOURCES_MYSQL_PASSWORD: pio
+
diff --git a/docker/mysql/docker-compose.event.yml b/docker/mysql/docker-compose.event.yml
new file mode 100644
index 0000000..f5f4035
--- /dev/null
+++ b/docker/mysql/docker-compose.event.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ pio:
+ environment:
+ PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME: pio_event
+ PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE: MYSQL
+
diff --git a/docker/mysql/docker-compose.meta.yml b/docker/mysql/docker-compose.meta.yml
new file mode 100644
index 0000000..f7a5ece
--- /dev/null
+++ b/docker/mysql/docker-compose.meta.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ pio:
+ environment:
+ PIO_STORAGE_REPOSITORIES_METADATA_NAME: pio_meta
+ PIO_STORAGE_REPOSITORIES_METADATA_SOURCE: MYSQL
+
diff --git a/docker/mysql/docker-compose.model.yml b/docker/mysql/docker-compose.model.yml
new file mode 100644
index 0000000..4a48684
--- /dev/null
+++ b/docker/mysql/docker-compose.model.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ pio:
+ environment:
+ PIO_STORAGE_REPOSITORIES_MODELDATA_NAME: pio_model
+ PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE: MYSQL
+
diff --git a/docker/pgsql/docker-compose.base.yml b/docker/pgsql/docker-compose.base.yml
new file mode 100644
index 0000000..297d6a8
--- /dev/null
+++ b/docker/pgsql/docker-compose.base.yml
@@ -0,0 +1,32 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ postgres:
+ image: postgres:9
+ environment:
+ POSTGRES_USER: pio
+ POSTGRES_PASSWORD: pio
+ POSTGRES_INITDB_ARGS: --encoding=UTF8
+ pio:
+ depends_on:
+ - postgres
+ environment:
+ PIO_STORAGE_SOURCES_PGSQL_TYPE: jdbc
+ PIO_STORAGE_SOURCES_PGSQL_URL: "jdbc:postgresql://postgres/pio"
+ PIO_STORAGE_SOURCES_PGSQL_USERNAME: pio
+ PIO_STORAGE_SOURCES_PGSQL_PASSWORD: pio
+
diff --git a/docker/pgsql/docker-compose.event.yml b/docker/pgsql/docker-compose.event.yml
new file mode 100644
index 0000000..2579141
--- /dev/null
+++ b/docker/pgsql/docker-compose.event.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ pio:
+ environment:
+ PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME: pio_event
+ PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE: PGSQL
+
diff --git a/docker/pgsql/docker-compose.meta.yml b/docker/pgsql/docker-compose.meta.yml
new file mode 100644
index 0000000..345bfbe
--- /dev/null
+++ b/docker/pgsql/docker-compose.meta.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ pio:
+ environment:
+ PIO_STORAGE_REPOSITORIES_METADATA_NAME: pio_meta
+ PIO_STORAGE_REPOSITORIES_METADATA_SOURCE: PGSQL
+
diff --git a/docker/pgsql/docker-compose.model.yml b/docker/pgsql/docker-compose.model.yml
new file mode 100644
index 0000000..329649f
--- /dev/null
+++ b/docker/pgsql/docker-compose.model.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
+services:
+ pio:
+ environment:
+ PIO_STORAGE_REPOSITORIES_MODELDATA_NAME: pio_model
+ PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE: PGSQL
+
diff --git a/docker/pio/Dockerfile b/docker/pio/Dockerfile
new file mode 100644
index 0000000..88930bd
--- /dev/null
+++ b/docker/pio/Dockerfile
@@ -0,0 +1,69 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8
+
+ARG PIO_GIT_URL=https://github.com/apache/predictionio.git
+ARG PIO_TAG=v0.13.0
+ENV SCALA_VERSION=2.11.12
+ENV SPARK_VERSION=2.2.2
+ENV HADOOP_VERSION=2.7.7
+ENV ELASTICSEARCH_VERSION=5.5.3
+ENV PGSQL_VERSION=42.2.4
+ENV MYSQL_VERSION=8.0.12
+ENV PIO_HOME=/usr/share/predictionio
+
+RUN apt-get update && \
+ apt-get install -y dpkg-dev fakeroot && \
+ apt-get clean && \
+ rm -rf /var/lib/apt/lists/*
+
+WORKDIR /opt/src
+RUN git clone -b $PIO_TAG $PIO_GIT_URL
+WORKDIR /opt/src/predictionio
+RUN bash ./make-distribution.sh \
+ -Dscala.version=$SCALA_VERSION \
+ -Dspark.version=$SPARK_VERSION \
+ -Dhadoop.version=$HADOOP_VERSION \
+ -Delasticsearch.version=$ELASTICSEARCH_VERSION \
+ --with-deb && \
+ dpkg -i ./assembly/target/predictionio_*.deb && \
+ rm -rf /opt/src/predictionio/*
+
+
+RUN cp /etc/predictionio/pio-env.sh /etc/predictionio/pio-env.sh.orig && \
+ echo "#!/usr/bin/env bash" > /etc/predictionio/pio-env.sh
+RUN curl -o $PIO_HOME/lib/postgresql-$PGSQL_VERSION.jar \
+ http://central.maven.org/maven2/org/postgresql/postgresql/$PGSQL_VERSION/postgresql-$PGSQL_VERSION.jar && \
+ echo "POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-$PGSQL_VERSION.jar" >> /etc/predictionio/pio-env.sh && \
+ echo "MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-$MYSQL_VERSION.jar" >> /etc/predictionio/pio-env.sh
+
+WORKDIR /usr/share
+RUN curl -o /opt/src/spark-$SPARK_VERSION.tgz \
+ http://www-us.apache.org/dist/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop2.7.tgz && \
+ tar zxvf /opt/src/spark-$SPARK_VERSION.tgz && \
+ echo "SPARK_HOME="`pwd`/`ls -d spark*` >> /etc/predictionio/pio-env.sh && \
+ rm -rf /opt/src
+
+WORKDIR /templates
+ADD pio_run /usr/bin/pio_run
+
+EXPOSE 7070
+EXPOSE 8000
+
+CMD ["sh", "/usr/bin/pio_run"]
+
diff --git a/docker/pio/pio_run b/docker/pio/pio_run
new file mode 100644
index 0000000..83ac6cc
--- /dev/null
+++ b/docker/pio/pio_run
@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+. /etc/predictionio/pio-env.sh
+
+# check elasticsearch status
+if [ x"$PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE" != "x" ] ; then
+ RET=-1
+ COUNT=0
+ ES_HOST=`echo $PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS | sed -e "s/,.*//"`
+ ES_PORT=`echo $PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS | sed -e "s/,.*//"`
+ # Wait for elasticsearch startup
+ while [ $RET != 0 -a $COUNT -lt 10 ] ; do
+ echo "Waiting for ${ES_HOST}..."
+ curl --connect-timeout 60 --retry 10 -s "$ES_HOST:$ES_PORT/_cluster/health?wait_for_status=green&timeout=1m"
+ RET=$?
+ COUNT=`expr $COUNT + 1`
+ sleep 1
+ done
+fi
+
+# check mysql jar file
+if [ x"$PIO_STORAGE_SOURCES_MYSQL_TYPE" != "x" ] ; then
+ MYSQL_JAR_FILE=$PIO_HOME/lib/mysql-connector-java-$MYSQL_VERSION.jar
+ if [ ! -f $MYSQL_JAR_FILE ] ; then
+ curl -o $MYSQL_JAR_FILE http://central.maven.org/maven2/mysql/mysql-connector-java/$MYSQL_VERSION/mysql-connector-java-$MYSQL_VERSION.jar
+ fi
+fi
+
+# Check PIO status
+RET=-1
+COUNT=0
+while [ $RET != 0 -a $COUNT -lt 10 ] ; do
+ echo "Waiting for PredictionIO..."
+ $PIO_HOME/bin/pio status
+ RET=$?
+ COUNT=`expr $COUNT + 1`
+ sleep 1
+done
+
+
+if [ x"$PIO_RUN_FILE" != "x" ] ; then
+ sh $PIO_RUN_FILE
+else
+ # Start PIO Event Server
+ $PIO_HOME/bin/pio eventserver
+fi
+
diff --git a/docker/templates/.keep b/docker/templates/.keep
new file mode 100644
index 0000000..ec20143
--- /dev/null
+++ b/docker/templates/.keep
@@ -0,0 +1,14 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.