You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@predictionio.apache.org by Noelia Osés Fernández <no...@vicomtech.org> on 2019/01/24 11:56:00 UTC
HBASE problems: ZooKeeper exists failed after 1 attempts ???
Hi all,
A few months back I set up a recommendation app using PredictionIO
0.12.0_incubating and UR 0.7.0-SNAPSHOT with elasticsearch 5.6.3, hbase
1.2.6, spark 2.1.1_bin_hadoop2.6 and mahout. It was working correctly and
producing results.
Then I stopped the AWS server where it was running for a couple of months
and now that I have restarted it I can't make PIO work. I have added the
output of pio status below. I do not try to manage zookeeper myself. I have
added hbase as per the instructions.
Any help will be much appreciated.
*$ pio status[INFO] [Management$] Inspecting PredictionIO...[INFO]
[Management$] PredictionIO 0.12.0-incubating is installed at
/home/ubuntu/PredictionIO/apache-predictionio-0.12.0-incubating/PredictionIO-0.12.0-incubating[INFO]
[Management$] Inspecting Apache Spark...[INFO] [Management$] Apache Spark
is installed at
/home/ubuntu/PredictionIO/apache-predictionio-0.12.0-incubating/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.6[INFO]
[Management$] Apache Spark 2.1.1 detected (meets minimum requirement of
1.3.0)[INFO] [Management$] Inspecting storage backend connections...[INFO]
[Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...[INFO]
[Storage$] Verifying Model Data Backend (Source: LOCALFS)...[INFO]
[Storage$] Verifying Event Data Backend (Source: HBASE)...[ERROR]
[RecoverableZooKeeper] ZooKeeper exists failed after 1 attempts[ERROR]
[ZooKeeperWatcher] hconnection-0x3afae281, quorum=localhost:2181,
baseZNode=/hbase Received unexpected KeeperException, re-throwing
exception[WARN] [ZooKeeperRegistry] Can't retrieve clusterId from
Zookeeper[ERROR] [StorageClient] Cannot connect to ZooKeeper (ZooKeeper
ensemble: localhost). Please make sure that the configuration is pointing
at the correct ZooKeeper ensemble. By default, HBase manages its own
ZooKeeper, so if you have not configured HBase to use an external
ZooKeeper, that means your HBase is not started or configured
properly.[ERROR] [Storage$] Error initializing storage client for source
HBASE.org.apache.hadoop.hbase.ZooKeeperConnectionException: Can't connect
to ZooKeeper at
org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:2358)
at
org.apache.predictionio.data.storage.hbase.StorageClient.<init>(StorageClient.scala:53)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
org.apache.predictionio.data.storage.Storage$.getClient(Storage.scala:252)
at
org.apache.predictionio.data.storage.Storage$.org$apache$predictionio$data$storage$Storage$$updateS2CM(Storage.scala:283)
at
org.apache.predictionio.data.storage.Storage$$anonfun$sourcesToClientMeta$1.apply(Storage.scala:244)
at
org.apache.predictionio.data.storage.Storage$$anonfun$sourcesToClientMeta$1.apply(Storage.scala:244)
at
scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80) at
org.apache.predictionio.data.storage.Storage$.sourcesToClientMeta(Storage.scala:244)
at
org.apache.predictionio.data.storage.Storage$.getDataObject(Storage.scala:315)
at
org.apache.predictionio.data.storage.Storage$.getDataObjectFromRepo(Storage.scala:300)
at
org.apache.predictionio.data.storage.Storage$.getLEvents(Storage.scala:448)
at
org.apache.predictionio.data.storage.Storage$.verifyAllDataObjects(Storage.scala:384)
at
org.apache.predictionio.tools.commands.Management$.status(Management.scala:156)
at org.apache.predictionio.tools.console.Pio$.status(Pio.scala:155) at
org.apache.predictionio.tools.console.Console$$anonfun$main$1.apply(Console.scala:721)
at
org.apache.predictionio.tools.console.Console$$anonfun$main$1.apply(Console.scala:656)
at scala.Option.map(Option.scala:146) at
org.apache.predictionio.tools.console.Console$.main(Console.scala:656)
at org.apache.predictionio.tools.console.Console.main(Console.scala)Caused
by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase at
org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at
org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at
org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) at
org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073) at
org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:2349)
... 23 more[ERROR] [Management$] Unable to connect to all storage backends
successfully.The following shows the error message from the storage
backend.Data source HBASE was not properly initialized.
(org.apache.predictionio.data.storage.StorageClientException)Dumping
configuration of initialized storage backend sources.Please make sure they
are correct.Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration:
HOME ->
/home/ubuntu/PredictionIO/apache-predictionio-0.12.0-incubating/PredictionIO-0.12.0-incubating/vendors/elasticsearch-5.6.3,
HOSTS -> localhost, PORTS -> 9200, CLUSTERNAME -> antolaturES, TYPE ->
elasticsearchSource Name: LOCALFS; Type: localfs; Configuration: PATH ->
/home/ubuntu/.pio_store/models, TYPE -> localfsSource Name: HBASE; Type:
(error); Configuration: (error)*
Best regards,
Noelia
Re: HBASE problems: ZooKeeper exists failed after 1 attempts ???
Posted by Noelia Osés Fernández <no...@vicomtech.org>.
Since I wrote the email I have stopped and started pio again (without
making any changes anywhere) and I get a slightly different error. I'm very
confused and frustrated. This is the output of pio status now:
$ pio status
[INFO] [Management$] Inspecting PredictionIO...
[INFO] [Management$] PredictionIO 0.12.0-incubating is installed at
/home/ubuntu/PredictionIO/apache-predictionio-0.12.0-incubating/PredictionIO-0.12.0-incubating
[INFO] [Management$] Inspecting Apache Spark...
[INFO] [Management$] Apache Spark is installed at
/home/ubuntu/PredictionIO/apache-predictionio-0.12.0-incubating/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.6
[INFO] [Management$] Apache Spark 2.1.1 detected (meets minimum requirement
of 1.3.0)
[INFO] [Management$] Inspecting storage backend connections...
[INFO] [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...
[INFO] [Storage$] Verifying Model Data Backend (Source: LOCALFS)...
[INFO] [Storage$] Verifying Event Data Backend (Source: HBASE)...
[INFO] [Storage$] Test writing to Event Store (App Id 0)...
[INFO] [HBLEvents] The table pio_event:events_0 doesn't exist yet. Creating
now...
[ERROR] [RecoverableZooKeeper] ZooKeeper getData failed after 2 attempts
[ERROR] [ZooKeeperWatcher]
hconnection-0x76db540e-0x1687fbd39fb000a-0x1687fbd39fb000a,
quorum=localhost:2181, baseZNode=/hbase Received unexpected
KeeperException, re-throwing exception
[ERROR] [RecoverableZooKeeper] ZooKeeper getData failed after 2 attempts
[ERROR] [ZooKeeperWatcher]
hconnection-0x76db540e-0x1687fbd39fb000a-0x1687fbd39fb000a,
quorum=localhost:2181, baseZNode=/hbase Received unexpected
KeeperException, re-throwing exception
[ERROR] [RecoverableZooKeeper] ZooKeeper getData failed after 2 attempts
[ERROR] [ZooKeeperWatcher]
hconnection-0x76db540e-0x1687fbd39fb000a-0x1687fbd39fb000a,
quorum=localhost:2181, baseZNode=/hbase Received unexpected
KeeperException, re-throwing exception
[ERROR] [RecoverableZooKeeper] ZooKeeper getData failed after 2 attempts
[ERROR] [ZooKeeperWatcher]
hconnection-0x76db540e-0x1687fbd39fb000a-0x1687fbd39fb000a,
quorum=localhost:2181, baseZNode=/hbase Received unexpected
KeeperException, re-throwing exception
[ERROR] [RecoverableZooKeeper] ZooKeeper getData failed after 2 attempts
[ERROR] [ZooKeeperWatcher]
hconnection-0x76db540e-0x1687fbd39fb000a-0x1687fbd39fb000a,
quorum=localhost:2181, baseZNode=/hbase Received unexpected
KeeperException, re-throwing exception
[ERROR] [RecoverableZooKeeper] ZooKeeper getData failed after 2 attempts
[ERROR] [ZooKeeperWatcher]
hconnection-0x76db540e-0x1687fbd39fb000a-0x1687fbd39fb000a,
quorum=localhost:2181, baseZNode=/hbase Received unexpected
KeeperException, re-throwing exception
[ERROR] [RecoverableZooKeeper] ZooKeeper getData failed after 2 attempts
[ERROR] [ZooKeeperWatcher]
hconnection-0x76db540e-0x1687fbd39fb000a-0x1687fbd39fb000a,
quorum=localhost:2181, baseZNode=/hbase Received unexpected
KeeperException, re-throwing exception
[ERROR] [RecoverableZooKeeper] ZooKeeper getData failed after 2 attempts
[ERROR] [ZooKeeperWatcher]
hconnection-0x76db540e-0x1687fbd39fb000a-0x1687fbd39fb000a,
quorum=localhost:2181, baseZNode=/hbase Received unexpected
KeeperException, re-throwing exception
[ERROR] [RecoverableZooKeeper] ZooKeeper getData failed after 2 attempts
[ERROR] [ZooKeeperWatcher]
hconnection-0x76db540e-0x1687fbd39fb000a-0x1687fbd39fb000a,
quorum=localhost:2181, baseZNode=/hbase Received unexpected
KeeperException, re-throwing exception
[ERROR] [Management$] Unable to connect to all storage backends
successfully.
The following shows the error message from the storage backend.
Futures timed out after [60 seconds] (java.util.concurrent.TimeoutException)
Dumping configuration of initialized storage backend sources.
Please make sure they are correct.
Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration: HOME ->
/home/ubuntu/PredictionIO/apache-predictionio-0.12.0-incubating/PredictionIO-0.12.0-incubating/vendors/elasticsearch-5.6.3,
HOSTS -> localhost, PORTS -> 9200, CLUSTERNAME -> antolaturES, TYPE ->
elasticsearch
Source Name: LOCALFS; Type: localfs; Configuration: PATH ->
/home/ubuntu/.pio_store/models, TYPE -> localfs
Source Name: HBASE; Type: hbase; Configuration: TYPE -> hbase, HOME ->
/home/ubuntu/PredictionIO/apache-predictionio-0.12.0-incubating/PredictionIO-0.12.0-incubating/vendors/hbase-1.2.6
On Thu, 24 Jan 2019 at 12:56, Noelia Osés Fernández <no...@vicomtech.org>
wrote:
> Hi all,
>
> A few months back I set up a recommendation app using PredictionIO
> 0.12.0_incubating and UR 0.7.0-SNAPSHOT with elasticsearch 5.6.3, hbase
> 1.2.6, spark 2.1.1_bin_hadoop2.6 and mahout. It was working correctly and
> producing results.
>
> Then I stopped the AWS server where it was running for a couple of months
> and now that I have restarted it I can't make PIO work. I have added the
> output of pio status below. I do not try to manage zookeeper myself. I have
> added hbase as per the instructions.
>
> Any help will be much appreciated.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *$ pio status[INFO] [Management$] Inspecting PredictionIO...[INFO]
> [Management$] PredictionIO 0.12.0-incubating is installed at
> /home/ubuntu/PredictionIO/apache-predictionio-0.12.0-incubating/PredictionIO-0.12.0-incubating[INFO]
> [Management$] Inspecting Apache Spark...[INFO] [Management$] Apache Spark
> is installed at
> /home/ubuntu/PredictionIO/apache-predictionio-0.12.0-incubating/PredictionIO-0.12.0-incubating/vendors/spark-2.1.1-bin-hadoop2.6[INFO]
> [Management$] Apache Spark 2.1.1 detected (meets minimum requirement of
> 1.3.0)[INFO] [Management$] Inspecting storage backend connections...[INFO]
> [Storage$] Verifying Meta Data Backend (Source: ELASTICSEARCH)...[INFO]
> [Storage$] Verifying Model Data Backend (Source: LOCALFS)...[INFO]
> [Storage$] Verifying Event Data Backend (Source: HBASE)...[ERROR]
> [RecoverableZooKeeper] ZooKeeper exists failed after 1 attempts[ERROR]
> [ZooKeeperWatcher] hconnection-0x3afae281, quorum=localhost:2181,
> baseZNode=/hbase Received unexpected KeeperException, re-throwing
> exception[WARN] [ZooKeeperRegistry] Can't retrieve clusterId from
> Zookeeper[ERROR] [StorageClient] Cannot connect to ZooKeeper (ZooKeeper
> ensemble: localhost). Please make sure that the configuration is pointing
> at the correct ZooKeeper ensemble. By default, HBase manages its own
> ZooKeeper, so if you have not configured HBase to use an external
> ZooKeeper, that means your HBase is not started or configured
> properly.[ERROR] [Storage$] Error initializing storage client for source
> HBASE.org.apache.hadoop.hbase.ZooKeeperConnectionException: Can't connect
> to ZooKeeper at
> org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:2358)
> at
> org.apache.predictionio.data.storage.hbase.StorageClient.<init>(StorageClient.scala:53)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.predictionio.data.storage.Storage$.getClient(Storage.scala:252)
> at
> org.apache.predictionio.data.storage.Storage$.org$apache$predictionio$data$storage$Storage$$updateS2CM(Storage.scala:283)
> at
> org.apache.predictionio.data.storage.Storage$$anonfun$sourcesToClientMeta$1.apply(Storage.scala:244)
> at
> org.apache.predictionio.data.storage.Storage$$anonfun$sourcesToClientMeta$1.apply(Storage.scala:244)
> at
> scala.collection.mutable.MapLike$class.getOrElseUpdate(MapLike.scala:194)
> at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80) at
> org.apache.predictionio.data.storage.Storage$.sourcesToClientMeta(Storage.scala:244)
> at
> org.apache.predictionio.data.storage.Storage$.getDataObject(Storage.scala:315)
> at
> org.apache.predictionio.data.storage.Storage$.getDataObjectFromRepo(Storage.scala:300)
> at
> org.apache.predictionio.data.storage.Storage$.getLEvents(Storage.scala:448)
> at
> org.apache.predictionio.data.storage.Storage$.verifyAllDataObjects(Storage.scala:384)
> at
> org.apache.predictionio.tools.commands.Management$.status(Management.scala:156)
> at org.apache.predictionio.tools.console.Pio$.status(Pio.scala:155) at
> org.apache.predictionio.tools.console.Console$$anonfun$main$1.apply(Console.scala:721)
> at
> org.apache.predictionio.tools.console.Console$$anonfun$main$1.apply(Console.scala:656)
> at scala.Option.map(Option.scala:146) at
> org.apache.predictionio.tools.console.Console$.main(Console.scala:656)
> at org.apache.predictionio.tools.console.Console.main(Console.scala)Caused
> by: org.apache.zookeeper.KeeperException$ConnectionLossException:
> KeeperErrorCode = ConnectionLoss for /hbase at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at
> org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045) at
> org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1073) at
> org.apache.hadoop.hbase.client.HBaseAdmin.checkHBaseAvailable(HBaseAdmin.java:2349)
> ... 23 more[ERROR] [Management$] Unable to connect to all storage backends
> successfully.The following shows the error message from the storage
> backend.Data source HBASE was not properly initialized.
> (org.apache.predictionio.data.storage.StorageClientException)Dumping
> configuration of initialized storage backend sources.Please make sure they
> are correct.Source Name: ELASTICSEARCH; Type: elasticsearch; Configuration:
> HOME ->
> /home/ubuntu/PredictionIO/apache-predictionio-0.12.0-incubating/PredictionIO-0.12.0-incubating/vendors/elasticsearch-5.6.3,
> HOSTS -> localhost, PORTS -> 9200, CLUSTERNAME -> antolaturES, TYPE ->
> elasticsearchSource Name: LOCALFS; Type: localfs; Configuration: PATH ->
> /home/ubuntu/.pio_store/models, TYPE -> localfsSource Name: HBASE; Type:
> (error); Configuration: (error)*
>
>
> Best regards,
> Noelia
>
>
>
>
>
>
>
>
--
<http://www.vicomtech.org/>
Noelia Osés Fernández, PhD
Senior Researcher | Investigadora Senior
noses@vicomtech.org
+[34] 943 30 92 30
Data Intelligence for Energy and Industrial Processes | Inteligencia de
Datos para Energía y Procesos Industriales
<https://www.linkedin.com/company/vicomtech>
<https://www.youtube.com/user/VICOMTech> <ht...@Vicomtech>
member of: <http://www.graphicsmedia.net/>
Legal Notice - Privacy policy <http://www.vicomtech.org/en/proteccion-datos>