[jira] [Created] (PIO-106) Elasticsearch 5.x StorageClient should reuse RestClient

["Mars Hall (JIRA)" <ji...@apache.org>]

Mars Hall created PIO-106:
-----------------------------

             Summary: Elasticsearch 5.x StorageClient should reuse RestClient
                 Key: PIO-106
                 URL: https://issues.apache.org/jira/browse/PIO-106
             Project: PredictionIO
          Issue Type: Improvement
          Components: Core
    Affects Versions: 0.11.0-incubating
            Reporter: Mars Hall


When using the proposed [PIO-105 Batch Predictions|https://issues.apache.org/jira/browse/PIO-105] feature with an engine that queries Elasticsearch in {{Algorithm#predict}}, Elasticsearch's REST interface appears to become overloaded, ending with the Spark job being killed from errors like:

{noformat}
[ERROR] [ESChannels] Failed to access to /pio_meta/channels/_search
[ERROR] [Utils] Aborting task
[ERROR] [ESApps] Failed to access to /pio_meta/apps/_search
[ERROR] [Executor] Exception in task 747.0 in stage 1.0 (TID 749)
[ERROR] [Executor] Exception in task 735.0 in stage 1.0 (TID 737)
[ERROR] [Common$] Invalid app name ur
[ERROR] [Utils] Aborting task
[ERROR] [URAlgorithm] Error when read recent events: java.lang.IllegalArgumentException: Invalid app name ur
[ERROR] [Executor] Exception in task 749.0 in stage 1.0 (TID 751)
[ERROR] [Utils] Aborting task
[ERROR] [Executor] Exception in task 748.0 in stage 1.0 (TID 750)
[WARN] [TaskSetManager] Lost task 749.0 in stage 1.0 (TID 751, localhost, executor driver): java.net.BindException: Can't assign requested address
  at sun.nio.ch.Net.connect0(Native Method)
  at sun.nio.ch.Net.connect(Net.java:454)
  at sun.nio.ch.Net.connect(Net.java:446)
  at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
  at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processSessionRequests(DefaultConnectingIOReactor.java:273)
  at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:139)
  at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348)
  at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
  at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
  at java.lang.Thread.run(Thread.java:745)
{noformat}

After these errors happen & the job is killed, Elasticsearch immediately recovers. It responds to queries normally. I researched what could cause this and found an [old issue in the main Elasticsearch repo|https://github.com/elastic/elasticsearch/issues/3647]. With the hints given therein about *using keep-alive in the ES client* to avoid these performance issues, I investigated how PredictionIO's [Elasticsearch StorageClient|https://github.com/apache/incubator-predictionio/tree/develop/storage/elasticsearch/src/main/scala/org/apache/predictionio/data/storage/elasticsearch] manages its connections.

I found that unlike the other StorageClients (Elasticsearch1, HBase, JDBC), Elasticsearch creates a new underlying Elasticsearch RestClient for [every|https://github.com/apache/incubator-predictionio/blob/develop/storage/elasticsearch/src/main/scala/org/apache/predictionio/data/storage/elasticsearch/ESApps.scala#L80] [single|https://github.com/apache/incubator-predictionio/blob/develop/storage/elasticsearch/src/main/scala/org/apache/predictionio/data/storage/elasticsearch/ESApps.scala#L157] [query|https://github.com/apache/incubator-predictionio/blob/develop/storage/elasticsearch/src/main/scala/org/apache/predictionio/data/storage/elasticsearch/ESChannels.scala#L78] & [interaction|https://github.com/apache/incubator-predictionio/blob/develop/storage/elasticsearch/src/main/scala/org/apache/predictionio/data/storage/elasticsearch/ESEngineInstances.scala#L205] with its API. As a result, *there is no way Elasticsearch TCP connections can be reused via HTTP keep-alive*.

High-performance workloads with Elasticsearch 5.x will suffer from these issues unless we refactor Elasticsearch StorageClient to share the underlying RestClient instead of [building a new one everytime the client is used|https://github.com/apache/incubator-predictionio/blob/develop/storage/elasticsearch/src/main/scala/org/apache/predictionio/data/storage/elasticsearch/StorageClient.scala#L31].

There are certainly different approaches we could take to sharing a RestClient so that its keep-alive behavior may work as designed:

* maintain a singleton RestClient that is reused throughout the ES storage classes
* create a RestClient on-demand and pass it as an argument to ES storage methods
* other ideas?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)