You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@predictionio.apache.org by Jim Miller <je...@gmail.com> on 2017/09/20 02:32:35 UTC
Unable to connect to all storage backends successfully
Hi,
I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
All starts with no errors but with pio status I get:
[INFO] [Management$] Inspecting PredictionIO...
[INFO] [Management$] PredictionIO 0.12.0-incubating is installed at /home/vagrant/pio/PredictionIO-0.12.0-incubating
[INFO] [Management$] Inspecting Apache Spark...
[INFO] [Management$] Apache Spark is installed at /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
[INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum requirement of 1.3.0)
[INFO] [Management$] Inspecting storage backend connections...
[INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
[ERROR] [Management$] Unable to connect to all storage backends successfully.
The following shows the error message from the storage backend.
Connection closed (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedException)
Dumping configuration of initialized storage backend sources.
Please make sure they are correct.
Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME -> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4, HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE -> elasticsearch
Can anyone give me an idea of what I need to fix this issue? Here is
# PredictionIO Main Configuration
#
# This section controls core behavior of PredictionIO. It is very likely that
# you need to change these to fit your site.
# SPARK_HOME: Apache Spark is a hard dependency and must be configured.
# SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
# ES_CONF_DIR: You must configure this if you have advanced configuration for
# your Elasticsearch setup.
ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
# HADOOP_CONF_DIR: You must configure this if you intend to run PredictionIO
# with Hadoop 2.
HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
# HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO
# with HBase on a remote cluster.
HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
# Filesystem paths where PredictionIO uses as block storage.
PIO_FS_BASEDIR=$HOME/.pio_store
PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
# PredictionIO Storage Configuration
#
# This section controls programs that make use of PredictionIO's built-in
# storage facilities. Default values are shown below.
#
# For more information on storage configuration please refer to
# http://predictionio.incubator.apache.org/system/anotherdatastore/
# Storage Repositories
# Default is to use PostgreSQL
PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
# Storage Data Sources
# PostgreSQL Default Settings
# Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL
# Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
# PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
# PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
# PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
# PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
# PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
# MySQL Example
# PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
# PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
# PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
# PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
# Elasticsearch Example
# PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
# PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
# PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
# Optional basic HTTP auth
# PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
# PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
# Elasticsearch 1.x Example
PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4
# Local File System Example
PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
# HBase Example
PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
# AWS S3 Example
# PIO_STORAGE_SOURCES_S3_TYPE=s3
# PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
# PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
Thanks,
Jim
--
Jim Miller
Re: Unable to connect to all storage backends successfully
Posted by Jim Miller <je...@gmail.com>.
Hi Guys,
The template that I am using is*
incubator-predictionio-template-recommender.*
Thanks,
Jim
On Wed, Sep 27, 2017 at 3:49 PM, Jim Miller <je...@gmail.com> wrote:
> incubator-predictionio-template-recommender
>
>
> On Wed, Sep 27, 2017 at 2:25 PM, Donald Szeto <do...@apache.org> wrote:
>
>> What template and version are you using?
>>
>> On Tue, Sep 26, 2017 at 9:28 PM Jim Miller <je...@gmail.com>
>> wrote:
>>
>>> Hi guys,
>>>
>>> I have another error that I cannot get past:
>>>
>>> *vagrant:MyRecommendation/ (develop*) $ pio train
>>>
>>>
>>> [21:47:39]*
>>> *[ERROR] [Console$] unable to read engine factory from
>>> /home/vagrant/MyRecommendation/engine.json. Aborting.*
>>>
>>>
>>> Does anyone have any advice?
>>>
>>> Thanks,
>>>
>>> Jim
>>>
>>> --
>>> Jim Miller
>>>
>>> On September 26, 2017 at 7:42:02 PM, Jim Miller (jemiller1963@gmail.com)
>>> wrote:
>>>
>>> Shinsuke, you rock! That was it.
>>>
>>> Thank you so much,
>>>
>>> Jim
>>>
>>> --
>>> Jim Miller
>>>
>>> On September 24, 2017 at 7:29:28 PM, Shinsuke Sugaya (
>>> shinsuke.sugaya@gmail.com) wrote:
>>>
>>> Hi,
>>>
>>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>>>
>>> Elasticsearch 5 uses 9200 port.
>>> Could you try 9200 port?
>>>
>>> Thanks,
>>> shinsuke
>>>
>>>
>>> 2017-09-24 3:02 GMT+09:00 Jim Miller <je...@gmail.com>:
>>> > Hi Donald,
>>> >
>>> > Tried just now and received the following error:
>>> >
>>> > vagrant:~/ $ pio status
>>> > [13:34:52]
>>> > [INFO] [Management$] Inspecting PredictionIO...
>>> > [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
>>> > /home/vagrant/pio/PredictionIO-0.12.0-incubating
>>> > [INFO] [Management$] Inspecting Apache Spark...
>>> > [INFO] [Management$] Apache Spark is installed at
>>> > /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/
>>> spark-2.1.1-bin-hadoop2.7
>>> > [INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum
>>> requirement
>>> > of 1.3.0)
>>> > [INFO] [Management$] Inspecting storage backend connections...
>>> > [INFO] [Storage$] Verifying Meta Data Backend (Source:
>>> ELASTICSEARCH)...
>>> > [ERROR] [Management$] Unable to connect to all storage backends
>>> > successfully.
>>> > The following shows the error message from the storage backend.
>>> >
>>> > error while performing request (java.lang.RuntimeException)
>>> >
>>> > Dumping configuration of initialized storage backend sources.
>>> > Please make sure they are correct.
>>> >
>>> > Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
>>> > /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/ela
>>> sticsearch-5.5.2,
>>> > HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE ->
>>> > elastic search
>>> >
>>> >
>>> > HERE IS MY PIO-ENV.SH
>>> >
>>> > # PredictionIO Main Configuration
>>> > #
>>> > # This section controls core behavior of PredictionIO. It is very
>>> likely
>>> > that
>>> > # you need to change these to fit your site.
>>> >
>>> > # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
>>> > # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
>>> > SPARK_HOME=$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7
>>> >
>>> > POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
>>> > MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>>> >
>>> > # ES_CONF_DIR: You must configure this if you have advanced
>>> configuration
>>> > for
>>> > # your Elasticsearch setup.
>>> > ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-5.5.2
>>> > # HADOOP_CONF_DIR=/opt/hadoop
>>> >
>>> > # HBASE_CONF_DIR: You must configure this if you intend to run
>>> PredictionIO
>>> > # with HBase on a remote cluster.
>>> > HBASE_CONF_DIR=$PIO_HOME/hbase-1.3.1/conf
>>> >
>>> > # Filesystem paths where PredictionIO uses as block storage.
>>> > PIO_FS_BASEDIR=$HOME/.pio_store
>>> > PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
>>> > PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>>> >
>>> > # PredictionIO Storage Configuration
>>> > #
>>> > # This section controls programs that make use of PredictionIO's
>>> built-in
>>> > # storage facilities. Default values are shown below.
>>> > #
>>> > # For more information on storage configuration please refer to
>>> > # http://predictionio.incubator.apache.org/system/anotherdatastore/
>>> >
>>> > # Storage Repositories
>>> >
>>> > # Default is to use PostgreSQL
>>> > PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
>>> > PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>>> >
>>> > PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
>>> > PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>>> >
>>> > PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
>>> > PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>>> >
>>> > # Storage Data Sources
>>> >
>>> > # PostgreSQL Default Settings
>>> > # Please change "pio" to your database name in
>>> PIO_STORAGE_SOURCES_PGSQL_URL
>>> > # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
>>> > # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
>>> > # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
>>> > # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
>>> > # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
>>> > # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>>> >
>>> > # MySQL Example
>>> > # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
>>> > # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
>>> > # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
>>> > # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>>> >
>>> > # Elasticsearch Example
>>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
>>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
>>> > #
>>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/
>>> elasticsearch-5.5.2
>>> > # Optional basic HTTP auth
>>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
>>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
>>> > # Elasticsearch 1.x Example
>>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstCluster
>>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/
>>> elasticsearch-5.5.2
>>> >
>>> > # Local File System Example
>>> > PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>> > PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>>> >
>>> > # HBase Example
>>> > PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
>>> > PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.3.1
>>> >
>>> > # AWS S3 Example
>>> > # PIO_STORAGE_SOURCES_S3_TYPE=s3
>>> > # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
>>> > # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>>> >
>>> >
>>> > ELASTICSEARCH.YML
>>> >
>>> > #======================== Elasticsearch Configuration
>>> > =========================
>>> > #
>>> > # NOTE: Elasticsearch comes with reasonable defaults for most settings.
>>> > # Before you set out to tweak and tune the configuration, make sure
>>> > you
>>> > # understand what are you trying to accomplish and the consequences.
>>> > #
>>> > # The primary way of configuring a node is via this file. This template
>>> > lists
>>> > # the most important settings you may want to configure for a
>>> production
>>> > cluster.
>>> > #
>>> > # Please consult the documentation for further information on
>>> configuration
>>> > options:
>>> > # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
>>> > #
>>> > # ---------------------------------- Cluster
>>> > -----------------------------------
>>> > #
>>> > # Use a descriptive name for your cluster:
>>> > #
>>> > cluster.name: firstCluster
>>> > #
>>> > # ------------------------------------ Node
>>> > ------------------------------------
>>> > #
>>> > # Use a descriptive name for the node:
>>> > #
>>> > #node.name: node-1
>>> > #
>>> > # Add custom attributes to the node:
>>> > #
>>> > #node.attr.rack: r1
>>> > #
>>> > # ----------------------------------- Paths
>>> > ------------------------------------
>>> > #
>>> > # Path to directory where to store the data (separate multiple
>>> locations by
>>> > comma):
>>> > #
>>> > #path.data: /path/to/data
>>> > #
>>> > # Path to log files:
>>> > #
>>> > #path.logs: /path/to/logs
>>> > #
>>> > # ----------------------------------- Memory
>>> > -----------------------------------
>>> > #
>>> > # Lock the memory on startup:
>>> > #
>>> > #bootstrap.memory_lock: true
>>> > #
>>> > # Make sure that the heap size is set to about half the memory
>>> available
>>> > # on the system and that the owner of the process is allowed to use
>>> this
>>> > # limit.
>>> > #
>>> > # Elasticsearch performs poorly when the system is swapping the memory.
>>> > #
>>> > # ---------------------------------- Network
>>> > -----------------------------------
>>> > #
>>> > # Set the bind address to a specific IP (IPv4 or IPv6):
>>> > #
>>> > #network.host: 192.168.0.1
>>> > #
>>> > # Set a custom port for HTTP:
>>> > #
>>> > #http.port: 9200
>>> > #
>>> > # For more information, consult the network module documentation.
>>> > #
>>> > # --------------------------------- Discovery
>>> > ----------------------------------
>>> > #
>>> > # Pass an initial list of hosts to perform discovery when new node is
>>> > started:
>>> > # The default list of hosts is ["127.0.0.1", "[::1]"]
>>> > #
>>> > #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
>>> > #
>>> > # Prevent the "split brain" by configuring the majority of nodes (total
>>> > number of master-eligible nodes / 2 + 1):
>>> > #
>>> > #discovery.zen.minimum_master_nodes: 3
>>> > #
>>> > # For more information, consult the zen discovery module documentation.
>>> > #
>>> > # ---------------------------------- Gateway
>>> > -----------------------------------
>>> > #
>>> > # Block initial recovery after a full cluster restart until N nodes are
>>> > started:
>>> > #
>>> > #gateway.recover_after_nodes: 3
>>> > #
>>> > # For more information, consult the gateway module documentation.
>>> > #
>>> > # ---------------------------------- Various
>>> > -----------------------------------
>>> > #
>>> > # Require explicit names when deleting indices:
>>> > #
>>> > #action.destructive_requires_name: true
>>> >
>>> >
>>> > Thanks for all of your help!
>>> >
>>> > Jim
>>> > --
>>> > Jim Miller
>>> >
>>> > On September 20, 2017 at 9:14:37 PM, Jim Miller (
>>> jemiller1963@gmail.com)
>>> > wrote:
>>> >
>>> > Hi Donald,
>>> >
>>> > I did not. I will read the release notes and update accordingly.
>>> >
>>> > Thanks!
>>> >
>>> > Jim
>>> >
>>> > --
>>> > Jim Miller
>>> >
>>> > On September 20, 2017 at 1:01:53 PM, Donald Szeto (donald@apache.org)
>>> wrote:
>>> >
>>> > Hey Jim,
>>> >
>>> > Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated
>>> in
>>> > 0.12 so the default build will use ES 5.x support.
>>> >
>>> > See the upcoming release notes:
>>> > https://github.com/apache/incubator-predictionio/blob/releas
>>> e/0.12.0/RELEASE.md#behavior-changes
>>> >
>>> > Regards,
>>> > Donald
>>> >
>>> > On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <je...@gmail.com>
>>> wrote:
>>> >>
>>> >> Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0:
>>> >>
>>> >> https://medium.freecodecamp.org/building-an-recommendation-
>>> engine-with-apache-prediction-io-ml-server-aed0319e0d8
>>> >>
>>> >> --
>>> >> Jim Miller
>>> >>
>>> >> On September 20, 2017 at 10:51:39 AM, Pat Ferrel (
>>> pat@occamsmachete.com)
>>> >> wrote:
>>> >>
>>> >> meaning is “firstcluster” the cluster name in your Elasticsearch
>>> >> configuration?
>>> >>
>>> >>
>>> >> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <va...@gmail.com>
>>> wrote:
>>> >>
>>> >> I think the problem is with Elasticsearch, are you sure the cluster
>>> exists
>>> >> in elasticsearch configuration?
>>> >>
>>> >> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <je...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi,
>>> >>>
>>> >>> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and
>>> Hbase:
>>> >>>
>>> >>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
>>> >>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
>>> >>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>>> >>>
>>> >>>
>>> >>> All starts with no errors but with pio status I get:
>>> >>>
>>> >>> [INFO] [Management$] Inspecting PredictionIO...
>>> >>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
>>> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating
>>> >>> [INFO] [Management$] Inspecting Apache Spark...
>>> >>> [INFO] [Management$] Apache Spark is installed at
>>> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/
>>> spark-1.5.1-bin-hadoop2.6
>>> >>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum
>>> >>> requirement of 1.3.0)
>>> >>> [INFO] [Management$] Inspecting storage backend connections...
>>> >>> [INFO] [Storage$] Verifying Meta Data Backend (Source:
>>> ELASTICSEARCH)...
>>> >>> [ERROR] [Management$] Unable to connect to all storage backends
>>> >>> successfully.
>>> >>> The following shows the error message from the storage backend.
>>> >>>
>>> >>> Connection closed
>>> >>> (org.apache.predictionio.shaded.org.apache.http.ConnectionCl
>>> osedException)
>>> >>>
>>> >>> Dumping configuration of initialized storage backend sources.
>>> >>> Please make sure they are correct.
>>> >>>
>>> >>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME
>>> ->
>>> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/ela
>>> sticsearch-1.4.4,
>>> >>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE
>>> ->
>>> >>> elasticsearch
>>> >>>
>>> >>>
>>> >>> Can anyone give me an idea of what I need to fix this issue? Here is
>>> >>>
>>> >>>
>>> >>> # PredictionIO Main Configuration
>>> >>> #
>>> >>> # This section controls core behavior of PredictionIO. It is very
>>> likely
>>> >>> that
>>> >>> # you need to change these to fit your site.
>>> >>>
>>> >>> # SPARK_HOME: Apache Spark is a hard dependency and must be
>>> configured.
>>> >>> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
>>> >>> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
>>> >>>
>>> >>> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
>>> >>> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>>> >>>
>>> >>> # ES_CONF_DIR: You must configure this if you have advanced
>>> configuration
>>> >>> for
>>> >>> # your Elasticsearch setup.
>>> >>> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
>>> >>>
>>> >>> # HADOOP_CONF_DIR: You must configure this if you intend to run
>>> >>> PredictionIO
>>> >>> # with Hadoop 2.
>>> >>> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
>>> >>>
>>> >>> # HBASE_CONF_DIR: You must configure this if you intend to run
>>> >>> PredictionIO
>>> >>> # with HBase on a remote cluster.
>>> >>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
>>> >>>
>>> >>> # Filesystem paths where PredictionIO uses as block storage.
>>> >>> PIO_FS_BASEDIR=$HOME/.pio_store
>>> >>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
>>> >>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>>> >>>
>>> >>> # PredictionIO Storage Configuration
>>> >>> #
>>> >>> # This section controls programs that make use of PredictionIO's
>>> built-in
>>> >>> # storage facilities. Default values are shown below.
>>> >>> #
>>> >>> # For more information on storage configuration please refer to
>>> >>> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>>> >>>
>>> >>> # Storage Repositories
>>> >>>
>>> >>> # Default is to use PostgreSQL
>>> >>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
>>> >>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>>> >>>
>>> >>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
>>> >>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>>> >>>
>>> >>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
>>> >>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>>> >>>
>>> >>> # Storage Data Sources
>>> >>>
>>> >>> # PostgreSQL Default Settings
>>> >>> # Please change "pio" to your database name in
>>> >>> PIO_STORAGE_SOURCES_PGSQL_URL
>>> >>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
>>> >>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
>>> >>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
>>> >>> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
>>> >>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
>>> >>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>>> >>>
>>> >>> # MySQL Example
>>> >>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
>>> >>> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
>>> >>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
>>> >>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>>> >>>
>>> >>> # Elasticsearch Example
>>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
>>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
>>> >>> #
>>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/
>>> elasticsearch-5.5.2
>>> >>> # Optional basic HTTP auth
>>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
>>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
>>> >>> # Elasticsearch 1.x Example
>>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
>>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>>> >>>
>>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/
>>> elasticsearch-1.4.4
>>> >>>
>>> >>> # Local File System Example
>>> >>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>> >>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>>> >>>
>>> >>> # HBase Example
>>> >>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
>>> >>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
>>> >>>
>>> >>> # AWS S3 Example
>>> >>> # PIO_STORAGE_SOURCES_S3_TYPE=s3
>>> >>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
>>> >>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>>> >>>
>>> >>>
>>> >>> Thanks,
>>> >>>
>>> >>> Jim
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Jim Miller
>>> >>
>>> >>
>>> >>
>>> >
>>>
>>>
>
Re: Unable to connect to all storage backends successfully
Posted by Jim Miller <je...@gmail.com>.
incubator-predictionio-template-recommender
On Wed, Sep 27, 2017 at 2:25 PM, Donald Szeto <do...@apache.org> wrote:
> What template and version are you using?
>
> On Tue, Sep 26, 2017 at 9:28 PM Jim Miller <je...@gmail.com> wrote:
>
>> Hi guys,
>>
>> I have another error that I cannot get past:
>>
>> *vagrant:MyRecommendation/ (develop*) $ pio train
>>
>> [21:47:39]*
>> *[ERROR] [Console$] unable to read engine factory from
>> /home/vagrant/MyRecommendation/engine.json. Aborting.*
>>
>>
>> Does anyone have any advice?
>>
>> Thanks,
>>
>> Jim
>>
>> --
>> Jim Miller
>>
>> On September 26, 2017 at 7:42:02 PM, Jim Miller (jemiller1963@gmail.com)
>> wrote:
>>
>> Shinsuke, you rock! That was it.
>>
>> Thank you so much,
>>
>> Jim
>>
>> --
>> Jim Miller
>>
>> On September 24, 2017 at 7:29:28 PM, Shinsuke Sugaya (
>> shinsuke.sugaya@gmail.com) wrote:
>>
>> Hi,
>>
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>>
>> Elasticsearch 5 uses 9200 port.
>> Could you try 9200 port?
>>
>> Thanks,
>> shinsuke
>>
>>
>> 2017-09-24 3:02 GMT+09:00 Jim Miller <je...@gmail.com>:
>> > Hi Donald,
>> >
>> > Tried just now and received the following error:
>> >
>> > vagrant:~/ $ pio status
>> > [13:34:52]
>> > [INFO] [Management$] Inspecting PredictionIO...
>> > [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
>> > /home/vagrant/pio/PredictionIO-0.12.0-incubating
>> > [INFO] [Management$] Inspecting Apache Spark...
>> > [INFO] [Management$] Apache Spark is installed at
>> > /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-2.1.
>> 1-bin-hadoop2.7
>> > [INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum
>> requirement
>> > of 1.3.0)
>> > [INFO] [Management$] Inspecting storage backend connections...
>> > [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
>> > [ERROR] [Management$] Unable to connect to all storage backends
>> > successfully.
>> > The following shows the error message from the storage backend.
>> >
>> > error while performing request (java.lang.RuntimeException)
>> >
>> > Dumping configuration of initialized storage backend sources.
>> > Please make sure they are correct.
>> >
>> > Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
>> > /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/
>> elasticsearch-5.5.2,
>> > HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE ->
>> > elastic search
>> >
>> >
>> > HERE IS MY PIO-ENV.SH
>> >
>> > # PredictionIO Main Configuration
>> > #
>> > # This section controls core behavior of PredictionIO. It is very likely
>> > that
>> > # you need to change these to fit your site.
>> >
>> > # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
>> > # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
>> > SPARK_HOME=$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7
>> >
>> > POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
>> > MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>> >
>> > # ES_CONF_DIR: You must configure this if you have advanced
>> configuration
>> > for
>> > # your Elasticsearch setup.
>> > ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-5.5.2
>> > # HADOOP_CONF_DIR=/opt/hadoop
>> >
>> > # HBASE_CONF_DIR: You must configure this if you intend to run
>> PredictionIO
>> > # with HBase on a remote cluster.
>> > HBASE_CONF_DIR=$PIO_HOME/hbase-1.3.1/conf
>> >
>> > # Filesystem paths where PredictionIO uses as block storage.
>> > PIO_FS_BASEDIR=$HOME/.pio_store
>> > PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
>> > PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>> >
>> > # PredictionIO Storage Configuration
>> > #
>> > # This section controls programs that make use of PredictionIO's
>> built-in
>> > # storage facilities. Default values are shown below.
>> > #
>> > # For more information on storage configuration please refer to
>> > # http://predictionio.incubator.apache.org/system/anotherdatastore/
>> >
>> > # Storage Repositories
>> >
>> > # Default is to use PostgreSQL
>> > PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
>> > PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>> >
>> > PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
>> > PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>> >
>> > PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
>> > PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>> >
>> > # Storage Data Sources
>> >
>> > # PostgreSQL Default Settings
>> > # Please change "pio" to your database name in
>> PIO_STORAGE_SOURCES_PGSQL_URL
>> > # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
>> > # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
>> > # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
>> > # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
>> > # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
>> > # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>> >
>> > # MySQL Example
>> > # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
>> > # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
>> > # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
>> > # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>> >
>> > # Elasticsearch Example
>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
>> > #
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/
>> vendors/elasticsearch-5.5.2
>> > # Optional basic HTTP auth
>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
>> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
>> > # Elasticsearch 1.x Example
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstCluster
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/
>> vendors/elasticsearch-5.5.2
>> >
>> > # Local File System Example
>> > PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>> > PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>> >
>> > # HBase Example
>> > PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
>> > PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.3.1
>> >
>> > # AWS S3 Example
>> > # PIO_STORAGE_SOURCES_S3_TYPE=s3
>> > # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
>> > # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>> >
>> >
>> > ELASTICSEARCH.YML
>> >
>> > #======================== Elasticsearch Configuration
>> > =========================
>> > #
>> > # NOTE: Elasticsearch comes with reasonable defaults for most settings.
>> > # Before you set out to tweak and tune the configuration, make sure
>> > you
>> > # understand what are you trying to accomplish and the consequences.
>> > #
>> > # The primary way of configuring a node is via this file. This template
>> > lists
>> > # the most important settings you may want to configure for a production
>> > cluster.
>> > #
>> > # Please consult the documentation for further information on
>> configuration
>> > options:
>> > # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
>> > #
>> > # ---------------------------------- Cluster
>> > -----------------------------------
>> > #
>> > # Use a descriptive name for your cluster:
>> > #
>> > cluster.name: firstCluster
>> > #
>> > # ------------------------------------ Node
>> > ------------------------------------
>> > #
>> > # Use a descriptive name for the node:
>> > #
>> > #node.name: node-1
>> > #
>> > # Add custom attributes to the node:
>> > #
>> > #node.attr.rack: r1
>> > #
>> > # ----------------------------------- Paths
>> > ------------------------------------
>> > #
>> > # Path to directory where to store the data (separate multiple
>> locations by
>> > comma):
>> > #
>> > #path.data: /path/to/data
>> > #
>> > # Path to log files:
>> > #
>> > #path.logs: /path/to/logs
>> > #
>> > # ----------------------------------- Memory
>> > -----------------------------------
>> > #
>> > # Lock the memory on startup:
>> > #
>> > #bootstrap.memory_lock: true
>> > #
>> > # Make sure that the heap size is set to about half the memory available
>> > # on the system and that the owner of the process is allowed to use this
>> > # limit.
>> > #
>> > # Elasticsearch performs poorly when the system is swapping the memory.
>> > #
>> > # ---------------------------------- Network
>> > -----------------------------------
>> > #
>> > # Set the bind address to a specific IP (IPv4 or IPv6):
>> > #
>> > #network.host: 192.168.0.1
>> > #
>> > # Set a custom port for HTTP:
>> > #
>> > #http.port: 9200
>> > #
>> > # For more information, consult the network module documentation.
>> > #
>> > # --------------------------------- Discovery
>> > ----------------------------------
>> > #
>> > # Pass an initial list of hosts to perform discovery when new node is
>> > started:
>> > # The default list of hosts is ["127.0.0.1", "[::1]"]
>> > #
>> > #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
>> > #
>> > # Prevent the "split brain" by configuring the majority of nodes (total
>> > number of master-eligible nodes / 2 + 1):
>> > #
>> > #discovery.zen.minimum_master_nodes: 3
>> > #
>> > # For more information, consult the zen discovery module documentation.
>> > #
>> > # ---------------------------------- Gateway
>> > -----------------------------------
>> > #
>> > # Block initial recovery after a full cluster restart until N nodes are
>> > started:
>> > #
>> > #gateway.recover_after_nodes: 3
>> > #
>> > # For more information, consult the gateway module documentation.
>> > #
>> > # ---------------------------------- Various
>> > -----------------------------------
>> > #
>> > # Require explicit names when deleting indices:
>> > #
>> > #action.destructive_requires_name: true
>> >
>> >
>> > Thanks for all of your help!
>> >
>> > Jim
>> > --
>> > Jim Miller
>> >
>> > On September 20, 2017 at 9:14:37 PM, Jim Miller (jemiller1963@gmail.com
>> )
>> > wrote:
>> >
>> > Hi Donald,
>> >
>> > I did not. I will read the release notes and update accordingly.
>> >
>> > Thanks!
>> >
>> > Jim
>> >
>> > --
>> > Jim Miller
>> >
>> > On September 20, 2017 at 1:01:53 PM, Donald Szeto (donald@apache.org)
>> wrote:
>> >
>> > Hey Jim,
>> >
>> > Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated
>> in
>> > 0.12 so the default build will use ES 5.x support.
>> >
>> > See the upcoming release notes:
>> > https://github.com/apache/incubator-predictionio/blob/
>> release/0.12.0/RELEASE.md#behavior-changes
>> >
>> > Regards,
>> > Donald
>> >
>> > On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <je...@gmail.com>
>> wrote:
>> >>
>> >> Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0:
>> >>
>> >> https://medium.freecodecamp.org/building-an-
>> recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8
>> >>
>> >> --
>> >> Jim Miller
>> >>
>> >> On September 20, 2017 at 10:51:39 AM, Pat Ferrel (
>> pat@occamsmachete.com)
>> >> wrote:
>> >>
>> >> meaning is “firstcluster” the cluster name in your Elasticsearch
>> >> configuration?
>> >>
>> >>
>> >> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <va...@gmail.com>
>> wrote:
>> >>
>> >> I think the problem is with Elasticsearch, are you sure the cluster
>> exists
>> >> in elasticsearch configuration?
>> >>
>> >> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <je...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
>> >>>
>> >>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
>> >>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
>> >>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>> >>>
>> >>>
>> >>> All starts with no errors but with pio status I get:
>> >>>
>> >>> [INFO] [Management$] Inspecting PredictionIO...
>> >>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
>> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating
>> >>> [INFO] [Management$] Inspecting Apache Spark...
>> >>> [INFO] [Management$] Apache Spark is installed at
>> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.
>> 1-bin-hadoop2.6
>> >>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum
>> >>> requirement of 1.3.0)
>> >>> [INFO] [Management$] Inspecting storage backend connections...
>> >>> [INFO] [Storage$] Verifying Meta Data Backend (Source:
>> ELASTICSEARCH)...
>> >>> [ERROR] [Management$] Unable to connect to all storage backends
>> >>> successfully.
>> >>> The following shows the error message from the storage backend.
>> >>>
>> >>> Connection closed
>> >>> (org.apache.predictionio.shaded.org.apache.http.
>> ConnectionClosedException)
>> >>>
>> >>> Dumping configuration of initialized storage backend sources.
>> >>> Please make sure they are correct.
>> >>>
>> >>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME
>> ->
>> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/
>> elasticsearch-1.4.4,
>> >>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE
>> ->
>> >>> elasticsearch
>> >>>
>> >>>
>> >>> Can anyone give me an idea of what I need to fix this issue? Here is
>> >>>
>> >>>
>> >>> # PredictionIO Main Configuration
>> >>> #
>> >>> # This section controls core behavior of PredictionIO. It is very
>> likely
>> >>> that
>> >>> # you need to change these to fit your site.
>> >>>
>> >>> # SPARK_HOME: Apache Spark is a hard dependency and must be
>> configured.
>> >>> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
>> >>> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
>> >>>
>> >>> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
>> >>> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>> >>>
>> >>> # ES_CONF_DIR: You must configure this if you have advanced
>> configuration
>> >>> for
>> >>> # your Elasticsearch setup.
>> >>> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
>> >>>
>> >>> # HADOOP_CONF_DIR: You must configure this if you intend to run
>> >>> PredictionIO
>> >>> # with Hadoop 2.
>> >>> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
>> >>>
>> >>> # HBASE_CONF_DIR: You must configure this if you intend to run
>> >>> PredictionIO
>> >>> # with HBase on a remote cluster.
>> >>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
>> >>>
>> >>> # Filesystem paths where PredictionIO uses as block storage.
>> >>> PIO_FS_BASEDIR=$HOME/.pio_store
>> >>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
>> >>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>> >>>
>> >>> # PredictionIO Storage Configuration
>> >>> #
>> >>> # This section controls programs that make use of PredictionIO's
>> built-in
>> >>> # storage facilities. Default values are shown below.
>> >>> #
>> >>> # For more information on storage configuration please refer to
>> >>> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>> >>>
>> >>> # Storage Repositories
>> >>>
>> >>> # Default is to use PostgreSQL
>> >>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
>> >>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>> >>>
>> >>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
>> >>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>> >>>
>> >>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
>> >>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>> >>>
>> >>> # Storage Data Sources
>> >>>
>> >>> # PostgreSQL Default Settings
>> >>> # Please change "pio" to your database name in
>> >>> PIO_STORAGE_SOURCES_PGSQL_URL
>> >>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
>> >>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
>> >>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
>> >>> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
>> >>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
>> >>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>> >>>
>> >>> # MySQL Example
>> >>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
>> >>> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
>> >>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
>> >>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>> >>>
>> >>> # Elasticsearch Example
>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
>> >>> #
>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/
>> vendors/elasticsearch-5.5.2
>> >>> # Optional basic HTTP auth
>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
>> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
>> >>> # Elasticsearch 1.x Example
>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>> >>>
>> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/
>> vendors/elasticsearch-1.4.4
>> >>>
>> >>> # Local File System Example
>> >>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>> >>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>> >>>
>> >>> # HBase Example
>> >>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
>> >>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
>> >>>
>> >>> # AWS S3 Example
>> >>> # PIO_STORAGE_SOURCES_S3_TYPE=s3
>> >>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
>> >>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>> >>>
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Jim
>> >>>
>> >>>
>> >>> --
>> >>> Jim Miller
>> >>
>> >>
>> >>
>> >
>>
>>
Re: Unable to connect to all storage backends successfully
Posted by Donald Szeto <do...@apache.org>.
What template and version are you using?
On Tue, Sep 26, 2017 at 9:28 PM Jim Miller <je...@gmail.com> wrote:
> Hi guys,
>
> I have another error that I cannot get past:
>
> *vagrant:MyRecommendation/ (develop*) $ pio train
>
> [21:47:39]*
> *[ERROR] [Console$] unable to read engine factory from
> /home/vagrant/MyRecommendation/engine.json. Aborting.*
>
>
> Does anyone have any advice?
>
> Thanks,
>
> Jim
>
> --
> Jim Miller
>
> On September 26, 2017 at 7:42:02 PM, Jim Miller (jemiller1963@gmail.com)
> wrote:
>
> Shinsuke, you rock! That was it.
>
> Thank you so much,
>
> Jim
>
> --
> Jim Miller
>
> On September 24, 2017 at 7:29:28 PM, Shinsuke Sugaya (
> shinsuke.sugaya@gmail.com) wrote:
>
> Hi,
>
> > PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>
> Elasticsearch 5 uses 9200 port.
> Could you try 9200 port?
>
> Thanks,
> shinsuke
>
>
> 2017-09-24 3:02 GMT+09:00 Jim Miller <je...@gmail.com>:
> > Hi Donald,
> >
> > Tried just now and received the following error:
> >
> > vagrant:~/ $ pio status
> > [13:34:52]
> > [INFO] [Management$] Inspecting PredictionIO...
> > [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
> > /home/vagrant/pio/PredictionIO-0.12.0-incubating
> > [INFO] [Management$] Inspecting Apache Spark...
> > [INFO] [Management$] Apache Spark is installed at
> >
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.7
> > [INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum
> requirement
> > of 1.3.0)
> > [INFO] [Management$] Inspecting storage backend connections...
> > [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
> > [ERROR] [Management$] Unable to connect to all storage backends
> > successfully.
> > The following shows the error message from the storage backend.
> >
> > error while performing request (java.lang.RuntimeException)
> >
> > Dumping configuration of initialized storage backend sources.
> > Please make sure they are correct.
> >
> > Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
> >
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-5.5.2,
> > HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE ->
> > elastic search
> >
> >
> > HERE IS MY PIO-ENV.SH
> >
> > # PredictionIO Main Configuration
> > #
> > # This section controls core behavior of PredictionIO. It is very likely
> > that
> > # you need to change these to fit your site.
> >
> > # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
> > # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
> > SPARK_HOME=$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7
> >
> > POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
> > MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
> >
> > # ES_CONF_DIR: You must configure this if you have advanced configuration
> > for
> > # your Elasticsearch setup.
> > ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-5.5.2
> > # HADOOP_CONF_DIR=/opt/hadoop
> >
> > # HBASE_CONF_DIR: You must configure this if you intend to run
> PredictionIO
> > # with HBase on a remote cluster.
> > HBASE_CONF_DIR=$PIO_HOME/hbase-1.3.1/conf
> >
> > # Filesystem paths where PredictionIO uses as block storage.
> > PIO_FS_BASEDIR=$HOME/.pio_store
> > PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
> > PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
> >
> > # PredictionIO Storage Configuration
> > #
> > # This section controls programs that make use of PredictionIO's built-in
> > # storage facilities. Default values are shown below.
> > #
> > # For more information on storage configuration please refer to
> > # http://predictionio.incubator.apache.org/system/anotherdatastore/
> >
> > # Storage Repositories
> >
> > # Default is to use PostgreSQL
> > PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
> > PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
> >
> > PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
> > PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
> >
> > PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
> > PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
> >
> > # Storage Data Sources
> >
> > # PostgreSQL Default Settings
> > # Please change "pio" to your database name in
> PIO_STORAGE_SOURCES_PGSQL_URL
> > # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
> > # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
> > # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
> > # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
> > # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
> > # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
> >
> > # MySQL Example
> > # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
> > # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
> > # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
> > # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
> >
> > # Elasticsearch Example
> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
> > #
> >
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
> > # Optional basic HTTP auth
> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
> > # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
> > # Elasticsearch 1.x Example
> > PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> > PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstCluster
> > PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> > PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
> >
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
> >
> > # Local File System Example
> > PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
> > PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
> >
> > # HBase Example
> > PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
> > PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.3.1
> >
> > # AWS S3 Example
> > # PIO_STORAGE_SOURCES_S3_TYPE=s3
> > # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
> > # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
> >
> >
> > ELASTICSEARCH.YML
> >
> > #======================== Elasticsearch Configuration
> > =========================
> > #
> > # NOTE: Elasticsearch comes with reasonable defaults for most settings.
> > # Before you set out to tweak and tune the configuration, make sure
> > you
> > # understand what are you trying to accomplish and the consequences.
> > #
> > # The primary way of configuring a node is via this file. This template
> > lists
> > # the most important settings you may want to configure for a production
> > cluster.
> > #
> > # Please consult the documentation for further information on
> configuration
> > options:
> > # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
> > #
> > # ---------------------------------- Cluster
> > -----------------------------------
> > #
> > # Use a descriptive name for your cluster:
> > #
> > cluster.name: firstCluster
> > #
> > # ------------------------------------ Node
> > ------------------------------------
> > #
> > # Use a descriptive name for the node:
> > #
> > #node.name: node-1
> > #
> > # Add custom attributes to the node:
> > #
> > #node.attr.rack: r1
> > #
> > # ----------------------------------- Paths
> > ------------------------------------
> > #
> > # Path to directory where to store the data (separate multiple locations
> by
> > comma):
> > #
> > #path.data: /path/to/data
> > #
> > # Path to log files:
> > #
> > #path.logs: /path/to/logs
> > #
> > # ----------------------------------- Memory
> > -----------------------------------
> > #
> > # Lock the memory on startup:
> > #
> > #bootstrap.memory_lock: true
> > #
> > # Make sure that the heap size is set to about half the memory available
> > # on the system and that the owner of the process is allowed to use this
> > # limit.
> > #
> > # Elasticsearch performs poorly when the system is swapping the memory.
> > #
> > # ---------------------------------- Network
> > -----------------------------------
> > #
> > # Set the bind address to a specific IP (IPv4 or IPv6):
> > #
> > #network.host: 192.168.0.1
> > #
> > # Set a custom port for HTTP:
> > #
> > #http.port: 9200
> > #
> > # For more information, consult the network module documentation.
> > #
> > # --------------------------------- Discovery
> > ----------------------------------
> > #
> > # Pass an initial list of hosts to perform discovery when new node is
> > started:
> > # The default list of hosts is ["127.0.0.1", "[::1]"]
> > #
> > #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
> > #
> > # Prevent the "split brain" by configuring the majority of nodes (total
> > number of master-eligible nodes / 2 + 1):
> > #
> > #discovery.zen.minimum_master_nodes: 3
> > #
> > # For more information, consult the zen discovery module documentation.
> > #
> > # ---------------------------------- Gateway
> > -----------------------------------
> > #
> > # Block initial recovery after a full cluster restart until N nodes are
> > started:
> > #
> > #gateway.recover_after_nodes: 3
> > #
> > # For more information, consult the gateway module documentation.
> > #
> > # ---------------------------------- Various
> > -----------------------------------
> > #
> > # Require explicit names when deleting indices:
> > #
> > #action.destructive_requires_name: true
> >
> >
> > Thanks for all of your help!
> >
> > Jim
> > --
> > Jim Miller
> >
> > On September 20, 2017 at 9:14:37 PM, Jim Miller (jemiller1963@gmail.com)
> > wrote:
> >
> > Hi Donald,
> >
> > I did not. I will read the release notes and update accordingly.
> >
> > Thanks!
> >
> > Jim
> >
> > --
> > Jim Miller
> >
> > On September 20, 2017 at 1:01:53 PM, Donald Szeto (donald@apache.org)
> wrote:
> >
> > Hey Jim,
> >
> > Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated in
> > 0.12 so the default build will use ES 5.x support.
> >
> > See the upcoming release notes:
> >
> https://github.com/apache/incubator-predictionio/blob/release/0.12.0/RELEASE.md#behavior-changes
> >
> > Regards,
> > Donald
> >
> > On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <je...@gmail.com>
> wrote:
> >>
> >> Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0:
> >>
> >>
> https://medium.freecodecamp.org/building-an-recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8
> >>
> >> --
> >> Jim Miller
> >>
> >> On September 20, 2017 at 10:51:39 AM, Pat Ferrel (pat@occamsmachete.com
> )
> >> wrote:
> >>
> >> meaning is “firstcluster” the cluster name in your Elasticsearch
> >> configuration?
> >>
> >>
> >> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <va...@gmail.com>
> wrote:
> >>
> >> I think the problem is with Elasticsearch, are you sure the cluster
> exists
> >> in elasticsearch configuration?
> >>
> >> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <je...@gmail.com>
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
> >>>
> >>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
> >>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
> >>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
> >>>
> >>>
> >>> All starts with no errors but with pio status I get:
> >>>
> >>> [INFO] [Management$] Inspecting PredictionIO...
> >>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
> >>> /home/vagrant/pio/PredictionIO-0.12.0-incubating
> >>> [INFO] [Management$] Inspecting Apache Spark...
> >>> [INFO] [Management$] Apache Spark is installed at
> >>>
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
> >>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum
> >>> requirement of 1.3.0)
> >>> [INFO] [Management$] Inspecting storage backend connections...
> >>> [INFO] [Storage$] Verifying Meta Data Backend (Source:
> ELASTICSEARCH)...
> >>> [ERROR] [Management$] Unable to connect to all storage backends
> >>> successfully.
> >>> The following shows the error message from the storage backend.
> >>>
> >>> Connection closed
> >>>
> (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedException)
> >>>
> >>> Dumping configuration of initialized storage backend sources.
> >>> Please make sure they are correct.
> >>>
> >>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
> >>>
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4,
> >>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE ->
> >>> elasticsearch
> >>>
> >>>
> >>> Can anyone give me an idea of what I need to fix this issue? Here is
> >>>
> >>>
> >>> # PredictionIO Main Configuration
> >>> #
> >>> # This section controls core behavior of PredictionIO. It is very
> likely
> >>> that
> >>> # you need to change these to fit your site.
> >>>
> >>> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
> >>> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
> >>> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
> >>>
> >>> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
> >>> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
> >>>
> >>> # ES_CONF_DIR: You must configure this if you have advanced
> configuration
> >>> for
> >>> # your Elasticsearch setup.
> >>> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
> >>>
> >>> # HADOOP_CONF_DIR: You must configure this if you intend to run
> >>> PredictionIO
> >>> # with Hadoop 2.
> >>> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
> >>>
> >>> # HBASE_CONF_DIR: You must configure this if you intend to run
> >>> PredictionIO
> >>> # with HBase on a remote cluster.
> >>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
> >>>
> >>> # Filesystem paths where PredictionIO uses as block storage.
> >>> PIO_FS_BASEDIR=$HOME/.pio_store
> >>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
> >>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
> >>>
> >>> # PredictionIO Storage Configuration
> >>> #
> >>> # This section controls programs that make use of PredictionIO's
> built-in
> >>> # storage facilities. Default values are shown below.
> >>> #
> >>> # For more information on storage configuration please refer to
> >>> # http://predictionio.incubator.apache.org/system/anotherdatastore/
> >>>
> >>> # Storage Repositories
> >>>
> >>> # Default is to use PostgreSQL
> >>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
> >>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
> >>>
> >>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
> >>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
> >>>
> >>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
> >>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
> >>>
> >>> # Storage Data Sources
> >>>
> >>> # PostgreSQL Default Settings
> >>> # Please change "pio" to your database name in
> >>> PIO_STORAGE_SOURCES_PGSQL_URL
> >>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
> >>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
> >>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
> >>> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
> >>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
> >>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
> >>>
> >>> # MySQL Example
> >>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
> >>> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
> >>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
> >>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
> >>>
> >>> # Elasticsearch Example
> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
> >>> #
> >>>
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
> >>> # Optional basic HTTP auth
> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
> >>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
> >>> # Elasticsearch 1.x Example
> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> >>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
> >>>
> >>>
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4
> >>>
> >>> # Local File System Example
> >>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
> >>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
> >>>
> >>> # HBase Example
> >>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
> >>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
> >>>
> >>> # AWS S3 Example
> >>> # PIO_STORAGE_SOURCES_S3_TYPE=s3
> >>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
> >>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Jim
> >>>
> >>>
> >>> --
> >>> Jim Miller
> >>
> >>
> >>
> >
>
>
Re: Unable to connect to all storage backends successfully
Posted by Jim Miller <je...@gmail.com>.
Hi guys,
I have another error that I cannot get past:
vagrant:MyRecommendation/ (develop*) $ pio train [21:47:39]
[ERROR] [Console$] unable to read engine factory from /home/vagrant/MyRecommendation/engine.json. Aborting.
Does anyone have any advice?
Thanks,
Jim
--
Jim Miller
On September 26, 2017 at 7:42:02 PM, Jim Miller (jemiller1963@gmail.com) wrote:
Shinsuke, you rock! That was it.
Thank you so much,
Jim
--
Jim Miller
On September 24, 2017 at 7:29:28 PM, Shinsuke Sugaya (shinsuke.sugaya@gmail.com) wrote:
Hi,
> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
Elasticsearch 5 uses 9200 port.
Could you try 9200 port?
Thanks,
shinsuke
2017-09-24 3:02 GMT+09:00 Jim Miller <je...@gmail.com>:
> Hi Donald,
>
> Tried just now and received the following error:
>
> vagrant:~/ $ pio status
> [13:34:52]
> [INFO] [Management$] Inspecting PredictionIO...
> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
> /home/vagrant/pio/PredictionIO-0.12.0-incubating
> [INFO] [Management$] Inspecting Apache Spark...
> [INFO] [Management$] Apache Spark is installed at
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.7
> [INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum requirement
> of 1.3.0)
> [INFO] [Management$] Inspecting storage backend connections...
> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
> [ERROR] [Management$] Unable to connect to all storage backends
> successfully.
> The following shows the error message from the storage backend.
>
> error while performing request (java.lang.RuntimeException)
>
> Dumping configuration of initialized storage backend sources.
> Please make sure they are correct.
>
> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-5.5.2,
> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE ->
> elastic search
>
>
> HERE IS MY PIO-ENV.SH
>
> # PredictionIO Main Configuration
> #
> # This section controls core behavior of PredictionIO. It is very likely
> that
> # you need to change these to fit your site.
>
> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
> SPARK_HOME=$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7
>
> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>
> # ES_CONF_DIR: You must configure this if you have advanced configuration
> for
> # your Elasticsearch setup.
> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-5.5.2
> # HADOOP_CONF_DIR=/opt/hadoop
>
> # HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO
> # with HBase on a remote cluster.
> HBASE_CONF_DIR=$PIO_HOME/hbase-1.3.1/conf
>
> # Filesystem paths where PredictionIO uses as block storage.
> PIO_FS_BASEDIR=$HOME/.pio_store
> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>
> # PredictionIO Storage Configuration
> #
> # This section controls programs that make use of PredictionIO's built-in
> # storage facilities. Default values are shown below.
> #
> # For more information on storage configuration please refer to
> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>
> # Storage Repositories
>
> # Default is to use PostgreSQL
> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>
> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>
> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>
> # Storage Data Sources
>
> # PostgreSQL Default Settings
> # Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL
> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>
> # MySQL Example
> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>
> # Elasticsearch Example
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
> #
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
> # Optional basic HTTP auth
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
> # Elasticsearch 1.x Example
> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstCluster
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
>
> # Local File System Example
> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>
> # HBase Example
> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.3.1
>
> # AWS S3 Example
> # PIO_STORAGE_SOURCES_S3_TYPE=s3
> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>
>
> ELASTICSEARCH.YML
>
> #======================== Elasticsearch Configuration
> =========================
> #
> # NOTE: Elasticsearch comes with reasonable defaults for most settings.
> # Before you set out to tweak and tune the configuration, make sure
> you
> # understand what are you trying to accomplish and the consequences.
> #
> # The primary way of configuring a node is via this file. This template
> lists
> # the most important settings you may want to configure for a production
> cluster.
> #
> # Please consult the documentation for further information on configuration
> options:
> # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
> #
> # ---------------------------------- Cluster
> -----------------------------------
> #
> # Use a descriptive name for your cluster:
> #
> cluster.name: firstCluster
> #
> # ------------------------------------ Node
> ------------------------------------
> #
> # Use a descriptive name for the node:
> #
> #node.name: node-1
> #
> # Add custom attributes to the node:
> #
> #node.attr.rack: r1
> #
> # ----------------------------------- Paths
> ------------------------------------
> #
> # Path to directory where to store the data (separate multiple locations by
> comma):
> #
> #path.data: /path/to/data
> #
> # Path to log files:
> #
> #path.logs: /path/to/logs
> #
> # ----------------------------------- Memory
> -----------------------------------
> #
> # Lock the memory on startup:
> #
> #bootstrap.memory_lock: true
> #
> # Make sure that the heap size is set to about half the memory available
> # on the system and that the owner of the process is allowed to use this
> # limit.
> #
> # Elasticsearch performs poorly when the system is swapping the memory.
> #
> # ---------------------------------- Network
> -----------------------------------
> #
> # Set the bind address to a specific IP (IPv4 or IPv6):
> #
> #network.host: 192.168.0.1
> #
> # Set a custom port for HTTP:
> #
> #http.port: 9200
> #
> # For more information, consult the network module documentation.
> #
> # --------------------------------- Discovery
> ----------------------------------
> #
> # Pass an initial list of hosts to perform discovery when new node is
> started:
> # The default list of hosts is ["127.0.0.1", "[::1]"]
> #
> #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
> #
> # Prevent the "split brain" by configuring the majority of nodes (total
> number of master-eligible nodes / 2 + 1):
> #
> #discovery.zen.minimum_master_nodes: 3
> #
> # For more information, consult the zen discovery module documentation.
> #
> # ---------------------------------- Gateway
> -----------------------------------
> #
> # Block initial recovery after a full cluster restart until N nodes are
> started:
> #
> #gateway.recover_after_nodes: 3
> #
> # For more information, consult the gateway module documentation.
> #
> # ---------------------------------- Various
> -----------------------------------
> #
> # Require explicit names when deleting indices:
> #
> #action.destructive_requires_name: true
>
>
> Thanks for all of your help!
>
> Jim
> --
> Jim Miller
>
> On September 20, 2017 at 9:14:37 PM, Jim Miller (jemiller1963@gmail.com)
> wrote:
>
> Hi Donald,
>
> I did not. I will read the release notes and update accordingly.
>
> Thanks!
>
> Jim
>
> --
> Jim Miller
>
> On September 20, 2017 at 1:01:53 PM, Donald Szeto (donald@apache.org) wrote:
>
> Hey Jim,
>
> Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated in
> 0.12 so the default build will use ES 5.x support.
>
> See the upcoming release notes:
> https://github.com/apache/incubator-predictionio/blob/release/0.12.0/RELEASE.md#behavior-changes
>
> Regards,
> Donald
>
> On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <je...@gmail.com> wrote:
>>
>> Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0:
>>
>> https://medium.freecodecamp.org/building-an-recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8
>>
>> --
>> Jim Miller
>>
>> On September 20, 2017 at 10:51:39 AM, Pat Ferrel (pat@occamsmachete.com)
>> wrote:
>>
>> meaning is “firstcluster” the cluster name in your Elasticsearch
>> configuration?
>>
>>
>> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <va...@gmail.com> wrote:
>>
>> I think the problem is with Elasticsearch, are you sure the cluster exists
>> in elasticsearch configuration?
>>
>> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <je...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
>>>
>>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
>>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
>>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>>>
>>>
>>> All starts with no errors but with pio status I get:
>>>
>>> [INFO] [Management$] Inspecting PredictionIO...
>>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating
>>> [INFO] [Management$] Inspecting Apache Spark...
>>> [INFO] [Management$] Apache Spark is installed at
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum
>>> requirement of 1.3.0)
>>> [INFO] [Management$] Inspecting storage backend connections...
>>> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
>>> [ERROR] [Management$] Unable to connect to all storage backends
>>> successfully.
>>> The following shows the error message from the storage backend.
>>>
>>> Connection closed
>>> (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedException)
>>>
>>> Dumping configuration of initialized storage backend sources.
>>> Please make sure they are correct.
>>>
>>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4,
>>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE ->
>>> elasticsearch
>>>
>>>
>>> Can anyone give me an idea of what I need to fix this issue? Here is
>>>
>>>
>>> # PredictionIO Main Configuration
>>> #
>>> # This section controls core behavior of PredictionIO. It is very likely
>>> that
>>> # you need to change these to fit your site.
>>>
>>> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
>>> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
>>> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
>>>
>>> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
>>> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>>>
>>> # ES_CONF_DIR: You must configure this if you have advanced configuration
>>> for
>>> # your Elasticsearch setup.
>>> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
>>>
>>> # HADOOP_CONF_DIR: You must configure this if you intend to run
>>> PredictionIO
>>> # with Hadoop 2.
>>> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
>>>
>>> # HBASE_CONF_DIR: You must configure this if you intend to run
>>> PredictionIO
>>> # with HBase on a remote cluster.
>>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
>>>
>>> # Filesystem paths where PredictionIO uses as block storage.
>>> PIO_FS_BASEDIR=$HOME/.pio_store
>>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
>>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>>>
>>> # PredictionIO Storage Configuration
>>> #
>>> # This section controls programs that make use of PredictionIO's built-in
>>> # storage facilities. Default values are shown below.
>>> #
>>> # For more information on storage configuration please refer to
>>> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>>>
>>> # Storage Repositories
>>>
>>> # Default is to use PostgreSQL
>>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
>>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>>>
>>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
>>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>>>
>>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
>>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>>>
>>> # Storage Data Sources
>>>
>>> # PostgreSQL Default Settings
>>> # Please change "pio" to your database name in
>>> PIO_STORAGE_SOURCES_PGSQL_URL
>>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
>>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
>>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
>>> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
>>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
>>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>>>
>>> # MySQL Example
>>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
>>> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
>>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
>>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>>>
>>> # Elasticsearch Example
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
>>> #
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
>>> # Optional basic HTTP auth
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
>>> # Elasticsearch 1.x Example
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>>>
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4
>>>
>>> # Local File System Example
>>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>>>
>>> # HBase Example
>>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
>>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
>>>
>>> # AWS S3 Example
>>> # PIO_STORAGE_SOURCES_S3_TYPE=s3
>>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
>>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>>>
>>>
>>> Thanks,
>>>
>>> Jim
>>>
>>>
>>> --
>>> Jim Miller
>>
>>
>>
>
Re: Unable to connect to all storage backends successfully
Posted by Jim Miller <je...@gmail.com>.
Shinsuke, you rock! That was it.
Thank you so much,
Jim
--
Jim Miller
On September 24, 2017 at 7:29:28 PM, Shinsuke Sugaya (shinsuke.sugaya@gmail.com) wrote:
Hi,
> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
Elasticsearch 5 uses 9200 port.
Could you try 9200 port?
Thanks,
shinsuke
2017-09-24 3:02 GMT+09:00 Jim Miller <je...@gmail.com>:
> Hi Donald,
>
> Tried just now and received the following error:
>
> vagrant:~/ $ pio status
> [13:34:52]
> [INFO] [Management$] Inspecting PredictionIO...
> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
> /home/vagrant/pio/PredictionIO-0.12.0-incubating
> [INFO] [Management$] Inspecting Apache Spark...
> [INFO] [Management$] Apache Spark is installed at
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.7
> [INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum requirement
> of 1.3.0)
> [INFO] [Management$] Inspecting storage backend connections...
> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
> [ERROR] [Management$] Unable to connect to all storage backends
> successfully.
> The following shows the error message from the storage backend.
>
> error while performing request (java.lang.RuntimeException)
>
> Dumping configuration of initialized storage backend sources.
> Please make sure they are correct.
>
> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-5.5.2,
> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE ->
> elastic search
>
>
> HERE IS MY PIO-ENV.SH
>
> # PredictionIO Main Configuration
> #
> # This section controls core behavior of PredictionIO. It is very likely
> that
> # you need to change these to fit your site.
>
> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
> SPARK_HOME=$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7
>
> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>
> # ES_CONF_DIR: You must configure this if you have advanced configuration
> for
> # your Elasticsearch setup.
> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-5.5.2
> # HADOOP_CONF_DIR=/opt/hadoop
>
> # HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO
> # with HBase on a remote cluster.
> HBASE_CONF_DIR=$PIO_HOME/hbase-1.3.1/conf
>
> # Filesystem paths where PredictionIO uses as block storage.
> PIO_FS_BASEDIR=$HOME/.pio_store
> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>
> # PredictionIO Storage Configuration
> #
> # This section controls programs that make use of PredictionIO's built-in
> # storage facilities. Default values are shown below.
> #
> # For more information on storage configuration please refer to
> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>
> # Storage Repositories
>
> # Default is to use PostgreSQL
> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>
> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>
> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>
> # Storage Data Sources
>
> # PostgreSQL Default Settings
> # Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL
> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>
> # MySQL Example
> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>
> # Elasticsearch Example
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
> #
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
> # Optional basic HTTP auth
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
> # Elasticsearch 1.x Example
> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstCluster
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
>
> # Local File System Example
> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>
> # HBase Example
> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.3.1
>
> # AWS S3 Example
> # PIO_STORAGE_SOURCES_S3_TYPE=s3
> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>
>
> ELASTICSEARCH.YML
>
> #======================== Elasticsearch Configuration
> =========================
> #
> # NOTE: Elasticsearch comes with reasonable defaults for most settings.
> # Before you set out to tweak and tune the configuration, make sure
> you
> # understand what are you trying to accomplish and the consequences.
> #
> # The primary way of configuring a node is via this file. This template
> lists
> # the most important settings you may want to configure for a production
> cluster.
> #
> # Please consult the documentation for further information on configuration
> options:
> # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
> #
> # ---------------------------------- Cluster
> -----------------------------------
> #
> # Use a descriptive name for your cluster:
> #
> cluster.name: firstCluster
> #
> # ------------------------------------ Node
> ------------------------------------
> #
> # Use a descriptive name for the node:
> #
> #node.name: node-1
> #
> # Add custom attributes to the node:
> #
> #node.attr.rack: r1
> #
> # ----------------------------------- Paths
> ------------------------------------
> #
> # Path to directory where to store the data (separate multiple locations by
> comma):
> #
> #path.data: /path/to/data
> #
> # Path to log files:
> #
> #path.logs: /path/to/logs
> #
> # ----------------------------------- Memory
> -----------------------------------
> #
> # Lock the memory on startup:
> #
> #bootstrap.memory_lock: true
> #
> # Make sure that the heap size is set to about half the memory available
> # on the system and that the owner of the process is allowed to use this
> # limit.
> #
> # Elasticsearch performs poorly when the system is swapping the memory.
> #
> # ---------------------------------- Network
> -----------------------------------
> #
> # Set the bind address to a specific IP (IPv4 or IPv6):
> #
> #network.host: 192.168.0.1
> #
> # Set a custom port for HTTP:
> #
> #http.port: 9200
> #
> # For more information, consult the network module documentation.
> #
> # --------------------------------- Discovery
> ----------------------------------
> #
> # Pass an initial list of hosts to perform discovery when new node is
> started:
> # The default list of hosts is ["127.0.0.1", "[::1]"]
> #
> #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
> #
> # Prevent the "split brain" by configuring the majority of nodes (total
> number of master-eligible nodes / 2 + 1):
> #
> #discovery.zen.minimum_master_nodes: 3
> #
> # For more information, consult the zen discovery module documentation.
> #
> # ---------------------------------- Gateway
> -----------------------------------
> #
> # Block initial recovery after a full cluster restart until N nodes are
> started:
> #
> #gateway.recover_after_nodes: 3
> #
> # For more information, consult the gateway module documentation.
> #
> # ---------------------------------- Various
> -----------------------------------
> #
> # Require explicit names when deleting indices:
> #
> #action.destructive_requires_name: true
>
>
> Thanks for all of your help!
>
> Jim
> --
> Jim Miller
>
> On September 20, 2017 at 9:14:37 PM, Jim Miller (jemiller1963@gmail.com)
> wrote:
>
> Hi Donald,
>
> I did not. I will read the release notes and update accordingly.
>
> Thanks!
>
> Jim
>
> --
> Jim Miller
>
> On September 20, 2017 at 1:01:53 PM, Donald Szeto (donald@apache.org) wrote:
>
> Hey Jim,
>
> Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated in
> 0.12 so the default build will use ES 5.x support.
>
> See the upcoming release notes:
> https://github.com/apache/incubator-predictionio/blob/release/0.12.0/RELEASE.md#behavior-changes
>
> Regards,
> Donald
>
> On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <je...@gmail.com> wrote:
>>
>> Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0:
>>
>> https://medium.freecodecamp.org/building-an-recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8
>>
>> --
>> Jim Miller
>>
>> On September 20, 2017 at 10:51:39 AM, Pat Ferrel (pat@occamsmachete.com)
>> wrote:
>>
>> meaning is “firstcluster” the cluster name in your Elasticsearch
>> configuration?
>>
>>
>> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <va...@gmail.com> wrote:
>>
>> I think the problem is with Elasticsearch, are you sure the cluster exists
>> in elasticsearch configuration?
>>
>> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <je...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
>>>
>>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
>>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
>>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>>>
>>>
>>> All starts with no errors but with pio status I get:
>>>
>>> [INFO] [Management$] Inspecting PredictionIO...
>>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating
>>> [INFO] [Management$] Inspecting Apache Spark...
>>> [INFO] [Management$] Apache Spark is installed at
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum
>>> requirement of 1.3.0)
>>> [INFO] [Management$] Inspecting storage backend connections...
>>> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
>>> [ERROR] [Management$] Unable to connect to all storage backends
>>> successfully.
>>> The following shows the error message from the storage backend.
>>>
>>> Connection closed
>>> (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedException)
>>>
>>> Dumping configuration of initialized storage backend sources.
>>> Please make sure they are correct.
>>>
>>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4,
>>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE ->
>>> elasticsearch
>>>
>>>
>>> Can anyone give me an idea of what I need to fix this issue? Here is
>>>
>>>
>>> # PredictionIO Main Configuration
>>> #
>>> # This section controls core behavior of PredictionIO. It is very likely
>>> that
>>> # you need to change these to fit your site.
>>>
>>> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
>>> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
>>> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
>>>
>>> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
>>> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>>>
>>> # ES_CONF_DIR: You must configure this if you have advanced configuration
>>> for
>>> # your Elasticsearch setup.
>>> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
>>>
>>> # HADOOP_CONF_DIR: You must configure this if you intend to run
>>> PredictionIO
>>> # with Hadoop 2.
>>> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
>>>
>>> # HBASE_CONF_DIR: You must configure this if you intend to run
>>> PredictionIO
>>> # with HBase on a remote cluster.
>>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
>>>
>>> # Filesystem paths where PredictionIO uses as block storage.
>>> PIO_FS_BASEDIR=$HOME/.pio_store
>>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
>>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>>>
>>> # PredictionIO Storage Configuration
>>> #
>>> # This section controls programs that make use of PredictionIO's built-in
>>> # storage facilities. Default values are shown below.
>>> #
>>> # For more information on storage configuration please refer to
>>> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>>>
>>> # Storage Repositories
>>>
>>> # Default is to use PostgreSQL
>>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
>>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>>>
>>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
>>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>>>
>>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
>>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>>>
>>> # Storage Data Sources
>>>
>>> # PostgreSQL Default Settings
>>> # Please change "pio" to your database name in
>>> PIO_STORAGE_SOURCES_PGSQL_URL
>>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
>>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
>>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
>>> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
>>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
>>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>>>
>>> # MySQL Example
>>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
>>> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
>>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
>>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>>>
>>> # Elasticsearch Example
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
>>> #
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
>>> # Optional basic HTTP auth
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
>>> # Elasticsearch 1.x Example
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>>>
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4
>>>
>>> # Local File System Example
>>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>>>
>>> # HBase Example
>>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
>>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
>>>
>>> # AWS S3 Example
>>> # PIO_STORAGE_SOURCES_S3_TYPE=s3
>>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
>>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>>>
>>>
>>> Thanks,
>>>
>>> Jim
>>>
>>>
>>> --
>>> Jim Miller
>>
>>
>>
>
Re: Unable to connect to all storage backends successfully
Posted by Shinsuke Sugaya <sh...@gmail.com>.
Hi,
> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
Elasticsearch 5 uses 9200 port.
Could you try 9200 port?
Thanks,
shinsuke
2017-09-24 3:02 GMT+09:00 Jim Miller <je...@gmail.com>:
> Hi Donald,
>
> Tried just now and received the following error:
>
> vagrant:~/ $ pio status
> [13:34:52]
> [INFO] [Management$] Inspecting PredictionIO...
> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
> /home/vagrant/pio/PredictionIO-0.12.0-incubating
> [INFO] [Management$] Inspecting Apache Spark...
> [INFO] [Management$] Apache Spark is installed at
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.7
> [INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum requirement
> of 1.3.0)
> [INFO] [Management$] Inspecting storage backend connections...
> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
> [ERROR] [Management$] Unable to connect to all storage backends
> successfully.
> The following shows the error message from the storage backend.
>
> error while performing request (java.lang.RuntimeException)
>
> Dumping configuration of initialized storage backend sources.
> Please make sure they are correct.
>
> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-5.5.2,
> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE ->
> elastic search
>
>
> HERE IS MY PIO-ENV.SH
>
> # PredictionIO Main Configuration
> #
> # This section controls core behavior of PredictionIO. It is very likely
> that
> # you need to change these to fit your site.
>
> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
> SPARK_HOME=$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7
>
> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>
> # ES_CONF_DIR: You must configure this if you have advanced configuration
> for
> # your Elasticsearch setup.
> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-5.5.2
> # HADOOP_CONF_DIR=/opt/hadoop
>
> # HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO
> # with HBase on a remote cluster.
> HBASE_CONF_DIR=$PIO_HOME/hbase-1.3.1/conf
>
> # Filesystem paths where PredictionIO uses as block storage.
> PIO_FS_BASEDIR=$HOME/.pio_store
> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>
> # PredictionIO Storage Configuration
> #
> # This section controls programs that make use of PredictionIO's built-in
> # storage facilities. Default values are shown below.
> #
> # For more information on storage configuration please refer to
> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>
> # Storage Repositories
>
> # Default is to use PostgreSQL
> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>
> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>
> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>
> # Storage Data Sources
>
> # PostgreSQL Default Settings
> # Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL
> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>
> # MySQL Example
> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>
> # Elasticsearch Example
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
> #
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
> # Optional basic HTTP auth
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
> # Elasticsearch 1.x Example
> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstCluster
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
>
> # Local File System Example
> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>
> # HBase Example
> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.3.1
>
> # AWS S3 Example
> # PIO_STORAGE_SOURCES_S3_TYPE=s3
> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>
>
> ELASTICSEARCH.YML
>
> #======================== Elasticsearch Configuration
> =========================
> #
> # NOTE: Elasticsearch comes with reasonable defaults for most settings.
> # Before you set out to tweak and tune the configuration, make sure
> you
> # understand what are you trying to accomplish and the consequences.
> #
> # The primary way of configuring a node is via this file. This template
> lists
> # the most important settings you may want to configure for a production
> cluster.
> #
> # Please consult the documentation for further information on configuration
> options:
> # https://www.elastic.co/guide/en/elasticsearch/reference/index.html
> #
> # ---------------------------------- Cluster
> -----------------------------------
> #
> # Use a descriptive name for your cluster:
> #
> cluster.name: firstCluster
> #
> # ------------------------------------ Node
> ------------------------------------
> #
> # Use a descriptive name for the node:
> #
> #node.name: node-1
> #
> # Add custom attributes to the node:
> #
> #node.attr.rack: r1
> #
> # ----------------------------------- Paths
> ------------------------------------
> #
> # Path to directory where to store the data (separate multiple locations by
> comma):
> #
> #path.data: /path/to/data
> #
> # Path to log files:
> #
> #path.logs: /path/to/logs
> #
> # ----------------------------------- Memory
> -----------------------------------
> #
> # Lock the memory on startup:
> #
> #bootstrap.memory_lock: true
> #
> # Make sure that the heap size is set to about half the memory available
> # on the system and that the owner of the process is allowed to use this
> # limit.
> #
> # Elasticsearch performs poorly when the system is swapping the memory.
> #
> # ---------------------------------- Network
> -----------------------------------
> #
> # Set the bind address to a specific IP (IPv4 or IPv6):
> #
> #network.host: 192.168.0.1
> #
> # Set a custom port for HTTP:
> #
> #http.port: 9200
> #
> # For more information, consult the network module documentation.
> #
> # --------------------------------- Discovery
> ----------------------------------
> #
> # Pass an initial list of hosts to perform discovery when new node is
> started:
> # The default list of hosts is ["127.0.0.1", "[::1]"]
> #
> #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
> #
> # Prevent the "split brain" by configuring the majority of nodes (total
> number of master-eligible nodes / 2 + 1):
> #
> #discovery.zen.minimum_master_nodes: 3
> #
> # For more information, consult the zen discovery module documentation.
> #
> # ---------------------------------- Gateway
> -----------------------------------
> #
> # Block initial recovery after a full cluster restart until N nodes are
> started:
> #
> #gateway.recover_after_nodes: 3
> #
> # For more information, consult the gateway module documentation.
> #
> # ---------------------------------- Various
> -----------------------------------
> #
> # Require explicit names when deleting indices:
> #
> #action.destructive_requires_name: true
>
>
> Thanks for all of your help!
>
> Jim
> --
> Jim Miller
>
> On September 20, 2017 at 9:14:37 PM, Jim Miller (jemiller1963@gmail.com)
> wrote:
>
> Hi Donald,
>
> I did not. I will read the release notes and update accordingly.
>
> Thanks!
>
> Jim
>
> --
> Jim Miller
>
> On September 20, 2017 at 1:01:53 PM, Donald Szeto (donald@apache.org) wrote:
>
> Hey Jim,
>
> Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated in
> 0.12 so the default build will use ES 5.x support.
>
> See the upcoming release notes:
> https://github.com/apache/incubator-predictionio/blob/release/0.12.0/RELEASE.md#behavior-changes
>
> Regards,
> Donald
>
> On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <je...@gmail.com> wrote:
>>
>> Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0:
>>
>> https://medium.freecodecamp.org/building-an-recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8
>>
>> --
>> Jim Miller
>>
>> On September 20, 2017 at 10:51:39 AM, Pat Ferrel (pat@occamsmachete.com)
>> wrote:
>>
>> meaning is “firstcluster” the cluster name in your Elasticsearch
>> configuration?
>>
>>
>> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <va...@gmail.com> wrote:
>>
>> I think the problem is with Elasticsearch, are you sure the cluster exists
>> in elasticsearch configuration?
>>
>> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <je...@gmail.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
>>>
>>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
>>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
>>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>>>
>>>
>>> All starts with no errors but with pio status I get:
>>>
>>> [INFO] [Management$] Inspecting PredictionIO...
>>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating
>>> [INFO] [Management$] Inspecting Apache Spark...
>>> [INFO] [Management$] Apache Spark is installed at
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum
>>> requirement of 1.3.0)
>>> [INFO] [Management$] Inspecting storage backend connections...
>>> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
>>> [ERROR] [Management$] Unable to connect to all storage backends
>>> successfully.
>>> The following shows the error message from the storage backend.
>>>
>>> Connection closed
>>> (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedException)
>>>
>>> Dumping configuration of initialized storage backend sources.
>>> Please make sure they are correct.
>>>
>>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
>>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4,
>>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE ->
>>> elasticsearch
>>>
>>>
>>> Can anyone give me an idea of what I need to fix this issue? Here is
>>>
>>>
>>> # PredictionIO Main Configuration
>>> #
>>> # This section controls core behavior of PredictionIO. It is very likely
>>> that
>>> # you need to change these to fit your site.
>>>
>>> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
>>> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
>>> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
>>>
>>> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
>>> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>>>
>>> # ES_CONF_DIR: You must configure this if you have advanced configuration
>>> for
>>> # your Elasticsearch setup.
>>> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
>>>
>>> # HADOOP_CONF_DIR: You must configure this if you intend to run
>>> PredictionIO
>>> # with Hadoop 2.
>>> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
>>>
>>> # HBASE_CONF_DIR: You must configure this if you intend to run
>>> PredictionIO
>>> # with HBase on a remote cluster.
>>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
>>>
>>> # Filesystem paths where PredictionIO uses as block storage.
>>> PIO_FS_BASEDIR=$HOME/.pio_store
>>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
>>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>>>
>>> # PredictionIO Storage Configuration
>>> #
>>> # This section controls programs that make use of PredictionIO's built-in
>>> # storage facilities. Default values are shown below.
>>> #
>>> # For more information on storage configuration please refer to
>>> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>>>
>>> # Storage Repositories
>>>
>>> # Default is to use PostgreSQL
>>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
>>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>>>
>>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
>>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>>>
>>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
>>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>>>
>>> # Storage Data Sources
>>>
>>> # PostgreSQL Default Settings
>>> # Please change "pio" to your database name in
>>> PIO_STORAGE_SOURCES_PGSQL_URL
>>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
>>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
>>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
>>> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
>>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
>>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>>>
>>> # MySQL Example
>>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
>>> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
>>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
>>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>>>
>>> # Elasticsearch Example
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
>>> #
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
>>> # Optional basic HTTP auth
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
>>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
>>> # Elasticsearch 1.x Example
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>>>
>>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4
>>>
>>> # Local File System Example
>>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>>>
>>> # HBase Example
>>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
>>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
>>>
>>> # AWS S3 Example
>>> # PIO_STORAGE_SOURCES_S3_TYPE=s3
>>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
>>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>>>
>>>
>>> Thanks,
>>>
>>> Jim
>>>
>>>
>>> --
>>> Jim Miller
>>
>>
>>
>
Re: Unable to connect to all storage backends successfully
Posted by Jim Miller <je...@gmail.com>.
Hi Donald,
Tried just now and received the following error:
vagrant:~/ $ pio status [13:34:52]
[INFO] [Management$] Inspecting PredictionIO...
[INFO] [Management$] PredictionIO 0.12.0-incubating is installed at /home/vagrant/pio/PredictionIO-0.12.0-incubating
[INFO] [Management$] Inspecting Apache Spark...
[INFO] [Management$] Apache Spark is installed at /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.7
[INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum requirement of 1.3.0)
[INFO] [Management$] Inspecting storage backend connections...
[INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
[ERROR] [Management$] Unable to connect to all storage backends successfully.
The following shows the error message from the storage backend.
error while performing request (java.lang.RuntimeException)
Dumping configuration of initialized storage backend sources.
Please make sure they are correct.
Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME -> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-5.5.2, HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstCluster, TYPE -> elastic search
HERE IS MY PIO-ENV.SH
# PredictionIO Main Configuration
#
# This section controls core behavior of PredictionIO. It is very likely that
# you need to change these to fit your site.
# SPARK_HOME: Apache Spark is a hard dependency and must be configured.
# SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
SPARK_HOME=$PIO_HOME/vendors/spark-2.1.1-bin-hadoop2.7
POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
# ES_CONF_DIR: You must configure this if you have advanced configuration for
# your Elasticsearch setup.
ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-5.5.2
# HADOOP_CONF_DIR=/opt/hadoop
# HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO
# with HBase on a remote cluster.
HBASE_CONF_DIR=$PIO_HOME/hbase-1.3.1/conf
# Filesystem paths where PredictionIO uses as block storage.
PIO_FS_BASEDIR=$HOME/.pio_store
PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
# PredictionIO Storage Configuration
#
# This section controls programs that make use of PredictionIO's built-in
# storage facilities. Default values are shown below.
#
# For more information on storage configuration please refer to
# http://predictionio.incubator.apache.org/system/anotherdatastore/
# Storage Repositories
# Default is to use PostgreSQL
PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
# Storage Data Sources
# PostgreSQL Default Settings
# Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL
# Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
# PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
# PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
# PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
# PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
# PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
# MySQL Example
# PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
# PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
# PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
# PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
# Elasticsearch Example
# PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
# PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
# PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
# Optional basic HTTP auth
# PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
# PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
# Elasticsearch 1.x Example
PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstCluster
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
# Local File System Example
PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
# HBase Example
PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.3.1
# AWS S3 Example
# PIO_STORAGE_SOURCES_S3_TYPE=s3
# PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
# PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
ELASTICSEARCH.YML
#======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: firstCluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: 192.168.0.1
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
Thanks for all of your help!
Jim
--
Jim Miller
On September 20, 2017 at 9:14:37 PM, Jim Miller (jemiller1963@gmail.com) wrote:
Hi Donald,
I did not. I will read the release notes and update accordingly.
Thanks!
Jim
--
Jim Miller
On September 20, 2017 at 1:01:53 PM, Donald Szeto (donald@apache.org) wrote:
Hey Jim,
Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated in 0.12 so the default build will use ES 5.x support.
See the upcoming release notes: https://github.com/apache/incubator-predictionio/blob/release/0.12.0/RELEASE.md#behavior-changes
Regards,
Donald
On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <je...@gmail.com> wrote:
Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0:
https://medium.freecodecamp.org/building-an-recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8
--
Jim Miller
On September 20, 2017 at 10:51:39 AM, Pat Ferrel (pat@occamsmachete.com) wrote:
meaning is “firstcluster” the cluster name in your Elasticsearch configuration?
On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <va...@gmail.com> wrote:
I think the problem is with Elasticsearch, are you sure the cluster exists in elasticsearch configuration?
On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <je...@gmail.com> wrote:
Hi,
I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
All starts with no errors but with pio status I get:
[INFO] [Management$] Inspecting PredictionIO...
[INFO] [Management$] PredictionIO 0.12.0-incubating is installed at /home/vagrant/pio/PredictionIO-0.12.0-incubating
[INFO] [Management$] Inspecting Apache Spark...
[INFO] [Management$] Apache Spark is installed at /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
[INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum requirement of 1.3.0)
[INFO] [Management$] Inspecting storage backend connections...
[INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
[ERROR] [Management$] Unable to connect to all storage backends successfully.
The following shows the error message from the storage backend.
Connection closed (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedException)
Dumping configuration of initialized storage backend sources.
Please make sure they are correct.
Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME -> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4, HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE -> elasticsearch
Can anyone give me an idea of what I need to fix this issue? Here is
# PredictionIO Main Configuration
#
# This section controls core behavior of PredictionIO. It is very likely that
# you need to change these to fit your site.
# SPARK_HOME: Apache Spark is a hard dependency and must be configured.
# SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
# ES_CONF_DIR: You must configure this if you have advanced configuration for
# your Elasticsearch setup.
ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
# HADOOP_CONF_DIR: You must configure this if you intend to run PredictionIO
# with Hadoop 2.
HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
# HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO
# with HBase on a remote cluster.
HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
# Filesystem paths where PredictionIO uses as block storage.
PIO_FS_BASEDIR=$HOME/.pio_store
PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
# PredictionIO Storage Configuration
#
# This section controls programs that make use of PredictionIO's built-in
# storage facilities. Default values are shown below.
#
# For more information on storage configuration please refer to
# http://predictionio.incubator.apache.org/system/anotherdatastore/
# Storage Repositories
# Default is to use PostgreSQL
PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
# Storage Data Sources
# PostgreSQL Default Settings
# Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL
# Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
# PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
# PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
# PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
# PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
# PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
# MySQL Example
# PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
# PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
# PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
# PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
# Elasticsearch Example
# PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
# PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
# PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
# Optional basic HTTP auth
# PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
# PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
# Elasticsearch 1.x Example
PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4
# Local File System Example
PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
# HBase Example
PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
# AWS S3 Example
# PIO_STORAGE_SOURCES_S3_TYPE=s3
# PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
# PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
Thanks,
Jim
--
Jim Miller
Re: Unable to connect to all storage backends successfully
Posted by Jim Miller <je...@gmail.com>.
Hi Donald,
I did not. I will read the release notes and update accordingly.
Thanks!
Jim
--
Jim Miller
On September 20, 2017 at 1:01:53 PM, Donald Szeto (donald@apache.org) wrote:
Hey Jim,
Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated in 0.12 so the default build will use ES 5.x support.
See the upcoming release notes: https://github.com/apache/incubator-predictionio/blob/release/0.12.0/RELEASE.md#behavior-changes
Regards,
Donald
On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <je...@gmail.com> wrote:
Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0:
https://medium.freecodecamp.org/building-an-recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8
--
Jim Miller
On September 20, 2017 at 10:51:39 AM, Pat Ferrel (pat@occamsmachete.com) wrote:
meaning is “firstcluster” the cluster name in your Elasticsearch configuration?
On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <va...@gmail.com> wrote:
I think the problem is with Elasticsearch, are you sure the cluster exists in elasticsearch configuration?
On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <je...@gmail.com> wrote:
Hi,
I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
All starts with no errors but with pio status I get:
[INFO] [Management$] Inspecting PredictionIO...
[INFO] [Management$] PredictionIO 0.12.0-incubating is installed at /home/vagrant/pio/PredictionIO-0.12.0-incubating
[INFO] [Management$] Inspecting Apache Spark...
[INFO] [Management$] Apache Spark is installed at /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
[INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum requirement of 1.3.0)
[INFO] [Management$] Inspecting storage backend connections...
[INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
[ERROR] [Management$] Unable to connect to all storage backends successfully.
The following shows the error message from the storage backend.
Connection closed (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedException)
Dumping configuration of initialized storage backend sources.
Please make sure they are correct.
Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME -> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4, HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE -> elasticsearch
Can anyone give me an idea of what I need to fix this issue? Here is
# PredictionIO Main Configuration
#
# This section controls core behavior of PredictionIO. It is very likely that
# you need to change these to fit your site.
# SPARK_HOME: Apache Spark is a hard dependency and must be configured.
# SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
# ES_CONF_DIR: You must configure this if you have advanced configuration for
# your Elasticsearch setup.
ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
# HADOOP_CONF_DIR: You must configure this if you intend to run PredictionIO
# with Hadoop 2.
HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
# HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO
# with HBase on a remote cluster.
HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
# Filesystem paths where PredictionIO uses as block storage.
PIO_FS_BASEDIR=$HOME/.pio_store
PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
# PredictionIO Storage Configuration
#
# This section controls programs that make use of PredictionIO's built-in
# storage facilities. Default values are shown below.
#
# For more information on storage configuration please refer to
# http://predictionio.incubator.apache.org/system/anotherdatastore/
# Storage Repositories
# Default is to use PostgreSQL
PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
# Storage Data Sources
# PostgreSQL Default Settings
# Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL
# Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
# PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
# PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
# PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
# PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
# PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
# MySQL Example
# PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
# PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
# PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
# PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
# Elasticsearch Example
# PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
# PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
# PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
# Optional basic HTTP auth
# PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
# PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
# Elasticsearch 1.x Example
PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4
# Local File System Example
PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
# HBase Example
PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
# AWS S3 Example
# PIO_STORAGE_SOURCES_S3_TYPE=s3
# PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
# PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
Thanks,
Jim
--
Jim Miller
Re: Unable to connect to all storage backends successfully
Posted by Donald Szeto <do...@apache.org>.
Hey Jim,
Did you build PIO 0.12 with ES 1.4 support? ES 1.x is being deprecated in
0.12 so the default build will use ES 5.x support.
See the upcoming release notes:
https://github.com/apache/incubator-predictionio/blob/release/0.12.0/RELEASE.md#behavior-changes
Regards,
Donald
On Wed, Sep 20, 2017 at 8:14 AM, Jim Miller <je...@gmail.com> wrote:
> Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0:
> https://medium.freecodecamp.org/building-an-recommendation-engine-with-
> apache-prediction-io-ml-server-aed0319e0d8
>
> --
> Jim Miller
>
> On September 20, 2017 at 10:51:39 AM, Pat Ferrel (pat@occamsmachete.com)
> wrote:
>
> meaning is “firstcluster” the cluster name in your Elasticsearch
> configuration?
>
>
> On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <va...@gmail.com> wrote:
>
> I think the problem is with Elasticsearch, are you sure the cluster exists
> in elasticsearch configuration?
>
> On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <je...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
>>
>> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
>> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
>> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>>
>>
>> All starts with no errors but with pio status I get:
>>
>> [INFO] [Management$] Inspecting PredictionIO...
>> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
>> /home/vagrant/pio/PredictionIO-0.12.0-incubating
>> [INFO] [Management$] Inspecting Apache Spark...
>> [INFO] [Management$] Apache Spark is installed at
>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/
>> spark-1.5.1-bin-hadoop2.6
>> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum
>> requirement of 1.3.0)
>> [INFO] [Management$] Inspecting storage backend connections...
>> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
>> [ERROR] [Management$] Unable to connect to all storage backends
>> successfully.
>> The following shows the error message from the storage backend.
>>
>> Connection closed (org.apache.predictionio.shade
>> d.org.apache.http.ConnectionClosedException)
>>
>> Dumping configuration of initialized storage backend sources.
>> Please make sure they are correct.
>>
>> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
>> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4,
>> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE ->
>> elasticsearch
>>
>>
>> Can anyone give me an idea of what I need to fix this issue? Here is
>>
>>
>> # PredictionIO Main Configuration
>> #
>> # This section controls core behavior of PredictionIO. It is very likely
>> that
>> # you need to change these to fit your site.
>>
>> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
>> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
>> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
>>
>> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
>> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>>
>> # ES_CONF_DIR: You must configure this if you have advanced configuration
>> for
>> # your Elasticsearch setup.
>> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
>>
>> # HADOOP_CONF_DIR: You must configure this if you intend to run
>> PredictionIO
>> # with Hadoop 2.
>> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
>>
>> # HBASE_CONF_DIR: You must configure this if you intend to run
>> PredictionIO
>> # with HBase on a remote cluster.
>> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
>>
>> # Filesystem paths where PredictionIO uses as block storage.
>> PIO_FS_BASEDIR=$HOME/.pio_store
>> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
>> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>>
>> # PredictionIO Storage Configuration
>> #
>> # This section controls programs that make use of PredictionIO's built-in
>> # storage facilities. Default values are shown below.
>> #
>> # For more information on storage configuration please refer to
>> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>>
>> # Storage Repositories
>>
>> # Default is to use PostgreSQL
>> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
>> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>>
>> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
>> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>>
>> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
>> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>>
>> # Storage Data Sources
>>
>> # PostgreSQL Default Settings
>> # Please change "pio" to your database name in
>> PIO_STORAGE_SOURCES_PGSQL_URL
>> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
>> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
>> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
>> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
>> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>>
>> # MySQL Example
>> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
>> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
>> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
>> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>>
>> # Elasticsearch Example
>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/
>> elasticsearch-5.5.2
>> # Optional basic HTTP auth
>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
>> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
>> # Elasticsearch 1.x Example
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
>> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/
>> elasticsearch-1.4.4
>>
>> # Local File System Example
>> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
>> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>>
>> # HBase Example
>> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
>> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
>>
>> # AWS S3 Example
>> # PIO_STORAGE_SOURCES_S3_TYPE=s3
>> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
>> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>>
>>
>> Thanks,
>>
>> Jim
>>
>> --
>> Jim Miller
>>
>
>
>
Re: Unable to connect to all storage backends successfully
Posted by Jim Miller <je...@gmail.com>.
Yes. I’m following this tutorial but using 0.12.0 instead of 0.10.0:
https://medium.freecodecamp.org/building-an-recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8
--
Jim Miller
On September 20, 2017 at 10:51:39 AM, Pat Ferrel (pat@occamsmachete.com) wrote:
meaning is “firstcluster” the cluster name in your Elasticsearch configuration?
On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <va...@gmail.com> wrote:
I think the problem is with Elasticsearch, are you sure the cluster exists in elasticsearch configuration?
On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <je...@gmail.com> wrote:
Hi,
I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
All starts with no errors but with pio status I get:
[INFO] [Management$] Inspecting PredictionIO...
[INFO] [Management$] PredictionIO 0.12.0-incubating is installed at /home/vagrant/pio/PredictionIO-0.12.0-incubating
[INFO] [Management$] Inspecting Apache Spark...
[INFO] [Management$] Apache Spark is installed at /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
[INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum requirement of 1.3.0)
[INFO] [Management$] Inspecting storage backend connections...
[INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
[ERROR] [Management$] Unable to connect to all storage backends successfully.
The following shows the error message from the storage backend.
Connection closed (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedException)
Dumping configuration of initialized storage backend sources.
Please make sure they are correct.
Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME -> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4, HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE -> elasticsearch
Can anyone give me an idea of what I need to fix this issue? Here is
# PredictionIO Main Configuration
#
# This section controls core behavior of PredictionIO. It is very likely that
# you need to change these to fit your site.
# SPARK_HOME: Apache Spark is a hard dependency and must be configured.
# SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
# ES_CONF_DIR: You must configure this if you have advanced configuration for
# your Elasticsearch setup.
ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
# HADOOP_CONF_DIR: You must configure this if you intend to run PredictionIO
# with Hadoop 2.
HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
# HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO
# with HBase on a remote cluster.
HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
# Filesystem paths where PredictionIO uses as block storage.
PIO_FS_BASEDIR=$HOME/.pio_store
PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
# PredictionIO Storage Configuration
#
# This section controls programs that make use of PredictionIO's built-in
# storage facilities. Default values are shown below.
#
# For more information on storage configuration please refer to
# http://predictionio.incubator.apache.org/system/anotherdatastore/
# Storage Repositories
# Default is to use PostgreSQL
PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
# Storage Data Sources
# PostgreSQL Default Settings
# Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL
# Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
# PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
# PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
# PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
# PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
# PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
# MySQL Example
# PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
# PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
# PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
# PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
# Elasticsearch Example
# PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
# PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
# PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
# Optional basic HTTP auth
# PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
# PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
# Elasticsearch 1.x Example
PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4
# Local File System Example
PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
# HBase Example
PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
# AWS S3 Example
# PIO_STORAGE_SOURCES_S3_TYPE=s3
# PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
# PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
Thanks,
Jim
--
Jim Miller
Re: Unable to connect to all storage backends successfully
Posted by Pat Ferrel <pa...@occamsmachete.com>.
meaning is “firstcluster” the cluster name in your Elasticsearch configuration?
On Sep 19, 2017, at 8:54 PM, Vaghawan Ojha <va...@gmail.com> wrote:
I think the problem is with Elasticsearch, are you sure the cluster exists in elasticsearch configuration?
On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <jemiller1963@gmail.com <ma...@gmail.com>> wrote:
Hi,
I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
All starts with no errors but with pio status I get:
[INFO] [Management$] Inspecting PredictionIO...
[INFO] [Management$] PredictionIO 0.12.0-incubating is installed at /home/vagrant/pio/PredictionIO-0.12.0-incubating
[INFO] [Management$] Inspecting Apache Spark...
[INFO] [Management$] Apache Spark is installed at /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
[INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum requirement of 1.3.0)
[INFO] [Management$] Inspecting storage backend connections...
[INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
[ERROR] [Management$] Unable to connect to all storage backends successfully.
The following shows the error message from the storage backend.
Connection closed (org.apache.predictionio.shaded.org.apache.http.ConnectionClosedException)
Dumping configuration of initialized storage backend sources.
Please make sure they are correct.
Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME -> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4, HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE -> elasticsearch
Can anyone give me an idea of what I need to fix this issue? Here is
# PredictionIO Main Configuration
#
# This section controls core behavior of PredictionIO. It is very likely that
# you need to change these to fit your site.
# SPARK_HOME: Apache Spark is a hard dependency and must be configured.
# SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
# ES_CONF_DIR: You must configure this if you have advanced configuration for
# your Elasticsearch setup.
ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
# HADOOP_CONF_DIR: You must configure this if you intend to run PredictionIO
# with Hadoop 2.
HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
# HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO
# with HBase on a remote cluster.
HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
# Filesystem paths where PredictionIO uses as block storage.
PIO_FS_BASEDIR=$HOME/.pio_store
PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
# PredictionIO Storage Configuration
#
# This section controls programs that make use of PredictionIO's built-in
# storage facilities. Default values are shown below.
#
# For more information on storage configuration please refer to
# http://predictionio.incubator.apache.org/system/anotherdatastore/ <http://predictionio.incubator.apache.org/system/anotherdatastore/>
# Storage Repositories
# Default is to use PostgreSQL
PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
# Storage Data Sources
# PostgreSQL Default Settings
# Please change "pio" to your database name in PIO_STORAGE_SOURCES_PGSQL_URL
# Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
# PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
# PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
# PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
# PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
# PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
# MySQL Example
# PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
# PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
# PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
# PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
# Elasticsearch Example
# PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
# PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
# PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
# PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-5.5.2
# Optional basic HTTP auth
# PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
# PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
# Elasticsearch 1.x Example
PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/vendors/elasticsearch-1.4.4
# Local File System Example
PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
# HBase Example
PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
# AWS S3 Example
# PIO_STORAGE_SOURCES_S3_TYPE=s3
# PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
# PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
Thanks,
Jim
--
Jim Miller
Re: Unable to connect to all storage backends successfully
Posted by Vaghawan Ojha <va...@gmail.com>.
I think the problem is with Elasticsearch, are you sure the cluster exists
in elasticsearch configuration?
On Wed, Sep 20, 2017 at 8:17 AM, Jim Miller <je...@gmail.com> wrote:
> Hi,
>
> I’m using PredictionIO 0.12.0-incubating with ElasticSearch and Hbase:
>
> PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4
> PredictionIO-0.12.0-incubating/vendors/hbase-1.0.0
> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
>
>
> All starts with no errors but with pio status I get:
>
> [INFO] [Management$] Inspecting PredictionIO...
> [INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
> /home/vagrant/pio/PredictionIO-0.12.0-incubating
> [INFO] [Management$] Inspecting Apache Spark...
> [INFO] [Management$] Apache Spark is installed at /home/vagrant/pio/
> PredictionIO-0.12.0-incubating/vendors/spark-1.5.1-bin-hadoop2.6
> [INFO] [Management$] Apache Spark 1.5.1 detected (meets minimum
> requirement of 1.3.0)
> [INFO] [Management$] Inspecting storage backend connections...
> [INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
> [ERROR] [Management$] Unable to connect to all storage backends
> successfully.
> The following shows the error message from the storage backend.
>
> Connection closed (org.apache.predictionio.shaded.org.apache.http.
> ConnectionClosedException)
>
> Dumping configuration of initialized storage backend sources.
> Please make sure they are correct.
>
> Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
> /home/vagrant/pio/PredictionIO-0.12.0-incubating/vendors/elasticsearch-1.4.4,
> HOSTS -> localhost, PORTS -> 9300, CLUSTERNAME -> firstcluster, TYPE ->
> elasticsearch
>
>
> Can anyone give me an idea of what I need to fix this issue? Here is
>
>
> # PredictionIO Main Configuration
> #
> # This section controls core behavior of PredictionIO. It is very likely
> that
> # you need to change these to fit your site.
>
> # SPARK_HOME: Apache Spark is a hard dependency and must be configured.
> # SPARK_HOME=$PIO_HOME/vendors/spark-2.0.2-bin-hadoop2.7
> SPARK_HOME=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6
>
> POSTGRES_JDBC_DRIVER=$PIO_HOME/lib/postgresql-42.0.0.jar
> MYSQL_JDBC_DRIVER=$PIO_HOME/lib/mysql-connector-java-5.1.41.jar
>
> # ES_CONF_DIR: You must configure this if you have advanced configuration
> for
> # your Elasticsearch setup.
> ES_CONF_DIR=$PIO_HOME/vendors/elasticsearch-1.4.4/conf
>
> # HADOOP_CONF_DIR: You must configure this if you intend to run
> PredictionIO
> # with Hadoop 2.
> HADOOP_CONF_DIR=$PIO_HOME/vendors/spark-1.5.1-bin-hadoop2.6/conf
>
> # HBASE_CONF_DIR: You must configure this if you intend to run PredictionIO
> # with HBase on a remote cluster.
> HBASE_CONF_DIR=$PIO_HOME/vendors/hbase-1.0.0/conf
>
> # Filesystem paths where PredictionIO uses as block storage.
> PIO_FS_BASEDIR=$HOME/.pio_store
> PIO_FS_ENGINESDIR=$PIO_FS_BASEDIR/engines
> PIO_FS_TMPDIR=$PIO_FS_BASEDIR/tmp
>
> # PredictionIO Storage Configuration
> #
> # This section controls programs that make use of PredictionIO's built-in
> # storage facilities. Default values are shown below.
> #
> # For more information on storage configuration please refer to
> # http://predictionio.incubator.apache.org/system/anotherdatastore/
>
> # Storage Repositories
>
> # Default is to use PostgreSQL
> PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta
> PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH
>
> PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event
> PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE
>
> PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model
> PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS
>
> # Storage Data Sources
>
> # PostgreSQL Default Settings
> # Please change "pio" to your database name in
> PIO_STORAGE_SOURCES_PGSQL_URL
> # Please change PIO_STORAGE_SOURCES_PGSQL_USERNAME and
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD accordingly
> # PIO_STORAGE_SOURCES_PGSQL_TYPE=jdbc
> # PIO_STORAGE_SOURCES_PGSQL_URL=jdbc:postgresql://localhost/pio
> # PIO_STORAGE_SOURCES_PGSQL_USERNAME=pio
> # PIO_STORAGE_SOURCES_PGSQL_PASSWORD=pio
>
> # MySQL Example
> # PIO_STORAGE_SOURCES_MYSQL_TYPE=jdbc
> # PIO_STORAGE_SOURCES_MYSQL_URL=jdbc:mysql://localhost/pio
> # PIO_STORAGE_SOURCES_MYSQL_USERNAME=pio
> # PIO_STORAGE_SOURCES_MYSQL_PASSWORD=pio
>
> # Elasticsearch Example
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_SCHEMES=http
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/
> vendors/elasticsearch-5.5.2
> # Optional basic HTTP auth
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_USERNAME=my-name
> # PIO_STORAGE_SOURCES_ELASTICSEARCH_PASSWORD=my-secret
> # Elasticsearch 1.x Example
> PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch
> PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=firstcluster
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost
> PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9300
> PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=$PIO_HOME/
> vendors/elasticsearch-1.4.4
>
> # Local File System Example
> PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs
> PIO_STORAGE_SOURCES_LOCALFS_PATH=$PIO_FS_BASEDIR/models
>
> # HBase Example
> PIO_STORAGE_SOURCES_HBASE_TYPE=hbase
> PIO_STORAGE_SOURCES_HBASE_HOME=$PIO_HOME/vendors/hbase-1.0.0
>
> # AWS S3 Example
> # PIO_STORAGE_SOURCES_S3_TYPE=s3
> # PIO_STORAGE_SOURCES_S3_BUCKET_NAME=pio_bucket
> # PIO_STORAGE_SOURCES_S3_BASE_PATH=pio_model
>
>
> Thanks,
>
> Jim
>
> --
> Jim Miller
>