You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@predictionio.apache.org by Harsh Mathur <ha...@gmail.com> on 2016/11/23 14:56:08 UTC
Universal Recommender Creating Index in ElasticSearch but not able to
persist the model
Hi,
The problem is in training phase of the UR template, the model
(correlators), is built, but unable to persist in elastic search. The
connectivity is there because indices are getting created, though no data
is being pushed. Help appreciated.
My SparkConf is as follows:
"es.nodes": "hostname",
"es.port": port,
"es.net.http.auth.user": "username",
"es.net.http.auth.pass": "password",
"es.net.ssl": true
However, I can see a new index getting created but no data being pushed
(timeout exception)
https://basic_auth@host:port/_cat/indices?v
green open urindex_1479911005241 4 0 0 0
460b 460b
green open pio_meta 4 0 19 1
210.8kb 210.8kb
green open urindex_1479909625467 4 0 0 0
460b 460b
green open urindex_1479908293213 4 0 0 0
460b 460b
green open urindex_1479911811014 4 0 0 0
460b 460b
green open urindex_1479910315092 4 0 0 0
460b 460b
As you can see after every retry, I can see a new index.
The Exception I get is:
[INFO] [URAlgorithm] Correlators created now putting into URModel
[INFO] [URModel] Converting cooccurrence matrices into correlators
[INFO] [URModel] Group all properties RDD
[INFO] [URModel] ES fields[10]: List(available, id, defaultRank,
categories, expires, popRank, purchase, countries, view, date)
[Stage 103:> (0 + 2)
/ 5][INFO] [HttpMethodDirector] I/O exception (java.net.ConnectException)
caught when processing request: Connection timed out (Connection timed out)
[INFO] [HttpMethodDirector] Retrying request
[INFO] [HttpMethodDirector] I/O exception (java.net.ConnectException)
caught when processing request: Connection timed out (Connection timed out)
[INFO] [HttpMethodDirector] Retrying request
Regards
Harsh Mathur
harshmathur.1990@gmail.com
*“Perseverance is the hard work you do after you get tired of doing the
hard work you already did."*
Re: Universal Recommender Creating Index in ElasticSearch but not
able to persist the model
Posted by Paul Blizzard <pa...@gmail.com>.
unsubscribe
On Wed, Nov 23, 2016 at 9:56 AM, Harsh Mathur <ha...@gmail.com>
wrote:
> Hi,
> The problem is in training phase of the UR template, the model
> (correlators), is built, but unable to persist in elastic search. The
> connectivity is there because indices are getting created, though no data
> is being pushed. Help appreciated.
>
> My SparkConf is as follows:
> "es.nodes": "hostname",
> "es.port": port,
> "es.net.http.auth.user": "username",
> "es.net.http.auth.pass": "password",
> "es.net.ssl": true
>
>
> However, I can see a new index getting created but no data being pushed
> (timeout exception)
>
> https://basic_auth@host:port/_cat/indices?v
> green open urindex_1479911005241 4 0 0 0
> 460b 460b
> green open pio_meta 4 0 19 1
> 210.8kb 210.8kb
> green open urindex_1479909625467 4 0 0 0
> 460b 460b
> green open urindex_1479908293213 4 0 0 0
> 460b 460b
> green open urindex_1479911811014 4 0 0 0
> 460b 460b
> green open urindex_1479910315092 4 0 0 0
> 460b 460b
>
>
> As you can see after every retry, I can see a new index.
>
> The Exception I get is:
>
> [INFO] [URAlgorithm] Correlators created now putting into URModel
> [INFO] [URModel] Converting cooccurrence matrices into correlators
> [INFO] [URModel] Group all properties RDD
> [INFO] [URModel] ES fields[10]: List(available, id, defaultRank,
> categories, expires, popRank, purchase, countries, view, date)
> [Stage 103:> (0 +
> 2) / 5][INFO] [HttpMethodDirector] I/O exception
> (java.net.ConnectException) caught when processing request: Connection
> timed out (Connection timed out)
> [INFO] [HttpMethodDirector] Retrying request
> [INFO] [HttpMethodDirector] I/O exception (java.net.ConnectException)
> caught when processing request: Connection timed out (Connection timed out)
> [INFO] [HttpMethodDirector] Retrying request
>
>
>
>
> Regards
> Harsh Mathur
> harshmathur.1990@gmail.com
>
> *“Perseverance is the hard work you do after you get tired of doing the
> hard work you already did."*
>
Re: Universal Recommender Creating Index in ElasticSearch but not
able to persist the model
Posted by Harsh Mathur <ha...@gmail.com>.
Hi,
Thanks for the help and you are right, REST API was not working due to the
problem I found below.
The problem and solution, just in case someone else encounters the same:
The problem was we are using a hosted ES as a service and while writing the
data to index, it first discovers all the ES nodes and then try to write.
The process of discovery gives it different hosts and ports then the one
provided by me in config.
Hence in in UR template (build.sbt), I needed to update:
"org.elasticsearch" % "elasticsearch-spark_2.10" % "2.1.2"
to
"org.elasticsearch" % "elasticsearch-spark_2.10" % "2.2.0"
Also in sparkConf in engine.json, I needed to provide:
"es.nodes.wan.only": true
Ofcourse, The alternate solution would be to get the ES Cluster and UR
running in same network but it is not possible for us as ES is outsourced
and UR is in aws.
Relevant Page: https://www.elastic.co/guide/en/elasticsearch/hadoop/
2.2/cloud.html
Regards
Harsh Mathur
harshmathur.1990@gmail.com
*“Perseverance is the hard work you do after you get tired of doing the
hard work you already did."*
On Fri, Nov 25, 2016 at 5:05 AM, Pat Ferrel <pa...@occamsmachete.com> wrote:
> I think the index is created using the transport client so requires no
> auth and happens before the writing starts, which uses the REST API. So the
> REST API seems to not be working at all.
>
> Check that the Spark Executor can connect to REST with your config. Also
> include "es.index.auto.create": "true"
>
>
>
> On Nov 23, 2016, at 6:56 AM, Harsh Mathur <ha...@gmail.com>
> wrote:
>
> Hi,
> The problem is in training phase of the UR template, the model
> (correlators), is built, but unable to persist in elastic search. The
> connectivity is there because indices are getting created, though no data
> is being pushed. Help appreciated.
>
> My SparkConf is as follows:
> "es.nodes": "hostname",
> "es.port": port,
> "es.net.http.auth.user": "username",
> "es.net.http.auth.pass": "password",
> "es.net.ssl": true
>
>
> However, I can see a new index getting created but no data being pushed
> (timeout exception)
>
> https://basic_auth@host:port/_cat/indices?v
> green open urindex_1479911005241 4 0 0 0
> 460b 460b
> green open pio_meta 4 0 19 1
> 210.8kb 210.8kb
> green open urindex_1479909625467 4 0 0 0
> 460b 460b
> green open urindex_1479908293213 4 0 0 0
> 460b 460b
> green open urindex_1479911811014 4 0 0 0
> 460b 460b
> green open urindex_1479910315092 4 0 0 0
> 460b 460b
>
>
> As you can see after every retry, I can see a new index.
>
> The Exception I get is:
>
> [INFO] [URAlgorithm] Correlators created now putting into URModel
> [INFO] [URModel] Converting cooccurrence matrices into correlators
> [INFO] [URModel] Group all properties RDD
> [INFO] [URModel] ES fields[10]: List(available, id, defaultRank,
> categories, expires, popRank, purchase, countries, view, date)
> [Stage 103:> (0 +
> 2) / 5][INFO] [HttpMethodDirector] I/O exception
> (java.net.ConnectException) caught when processing request: Connection
> timed out (Connection timed out)
> [INFO] [HttpMethodDirector] Retrying request
> [INFO] [HttpMethodDirector] I/O exception (java.net.ConnectException)
> caught when processing request: Connection timed out (Connection timed out)
> [INFO] [HttpMethodDirector] Retrying request
>
>
>
>
> Regards
> Harsh Mathur
> harshmathur.1990@gmail.com
>
> *“Perseverance is the hard work you do after you get tired of doing the
> hard work you already did."*
>
>
Re: Universal Recommender Creating Index in ElasticSearch but not
able to persist the model
Posted by Pat Ferrel <pa...@occamsmachete.com>.
I think the index is created using the transport client so requires no auth and happens before the writing starts, which uses the REST API. So the REST API seems to not be working at all.
Check that the Spark Executor can connect to REST with your config. Also include "es.index.auto.create": "true"
On Nov 23, 2016, at 6:56 AM, Harsh Mathur <ha...@gmail.com> wrote:
Hi,
The problem is in training phase of the UR template, the model (correlators), is built, but unable to persist in elastic search. The connectivity is there because indices are getting created, though no data is being pushed. Help appreciated.
My SparkConf is as follows:
"es.nodes": "hostname",
"es.port": port,
"es.net.http.auth.user": "username",
"es.net.http.auth.pass": "password",
"es.net.ssl": true
However, I can see a new index getting created but no data being pushed (timeout exception)
https://basic_auth@host:port/_cat/indices?v
green open urindex_1479911005241 4 0 0 0 460b 460b
green open pio_meta 4 0 19 1 210.8kb 210.8kb
green open urindex_1479909625467 4 0 0 0 460b 460b
green open urindex_1479908293213 4 0 0 0 460b 460b
green open urindex_1479911811014 4 0 0 0 460b 460b
green open urindex_1479910315092 4 0 0 0 460b 460b
As you can see after every retry, I can see a new index.
The Exception I get is:
[INFO] [URAlgorithm] Correlators created now putting into URModel
[INFO] [URModel] Converting cooccurrence matrices into correlators
[INFO] [URModel] Group all properties RDD
[INFO] [URModel] ES fields[10]: List(available, id, defaultRank, categories, expires, popRank, purchase, countries, view, date)
[Stage 103:> (0 + 2) / 5][INFO] [HttpMethodDirector] I/O exception (java.net.ConnectException) caught when processing request: Connection timed out (Connection timed out)
[INFO] [HttpMethodDirector] Retrying request
[INFO] [HttpMethodDirector] I/O exception (java.net.ConnectException) caught when processing request: Connection timed out (Connection timed out)
[INFO] [HttpMethodDirector] Retrying request
Regards
Harsh Mathur
harshmathur.1990@gmail.com <ma...@gmail.com>
“Perseverance is the hard work you do after you get tired of doing the hard work you already did."