RE:Unable to connect hbase-1.1.1 from apache phoenix-4.10.0 /4.11.0, after upgrading phoenix from 4.8.1

[Pratap Mutadak <pr...@whiteklay.com>]

Hi Team,

                   I am using 4 nodes MapR hadoop cluster with version 5.2.0
having hbase-1.1.1( 1hbasemaster and 3 hbaseregionservers). I am able to
connect from phoenix -4.8.1 to hbase-1.1.1 but since last few days we are
facing issue with hbase region server crashing issue and phoenix queryserver
getting died . So we  planned to upgrade phoenix -4.8.1 to latest version
4.10.0 . We followed below steps 

1)      Downloaded phoenix4.10.0 and extracted it on mapr cluster. 

2)      Moved existing phoenix4.8.1 jars from
/opt/mapr/hbase/hbase-1.1.1/lib and added below jar on the same location and
restarted hbase services on cluster.

phoenix-4.10.0-HBase-1.1-server.jar

phoenix-core-4.10.0-HBase-1.1.jar

3)      We are able to connect hbase from hbase shell , but from phoenix
when we are trying  from sqlline.py <zookeeperNode>:5181

We are getting below error .

Error: org.apache.hadoop.hbase.snapshot.SnapshotCreationException:
org.apache.hadoop.hbase.snapshot.SnapshotCreationException: Table
'SYSTEM.CATALOG' doesn't exist, can't take snapshot.

        at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(Snapsho
tManager.java:562)

        at
org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.
java:1247)

        at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.call
BlockingMethod(MasterProtos.java:48522)

        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)

        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)

        at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)

        at
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)

        at java.lang.Thread.run(Thread.java:745) (state=,code=0)

java.sql.SQLException:
org.apache.hadoop.hbase.snapshot.SnapshotCreationException:
org.apache.hadoop.hbase.snapshot.SnapshotCreationException: Table
'SYSTEM.CATALOG' doesn't exist, can't take snapshot.

        at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(Snapsho
tManager.java:562)

        at
org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.
java:1247)

        at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.call
BlockingMethod(MasterProtos.java:48522)

        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)

        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)

        at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)

        at
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)

        at java.lang.Thread.run(Thread.java:745)

 

        at
org.apache.phoenix.query.ConnectionQueryServicesImpl.createSnapshot(Connecti
onQueryServicesImpl.java:3032)

        at
org.apache.phoenix.query.ConnectionQueryServicesImpl.upgradeSystemTables(Con
nectionQueryServicesImpl.java:2598)

        at
org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQuery
ServicesImpl.java:2468)

        at
org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQuery
ServicesImpl.java:2389)

        at
org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.j
ava:76)

        at
org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQuerySer
vicesImpl.java:2389)

        at
org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriv
er.java:255)

        at
org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbedd
edDriver.java:150)

        at
org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)

        at sqlline.DatabaseConnection.connect(DatabaseConnection.java:157)

        at
sqlline.DatabaseConnection.getConnection(DatabaseConnection.java:203)

        at sqlline.Commands.connect(Commands.java:1064)

        at sqlline.Commands.connect(Commands.java:996)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:38)

        at sqlline.SqlLine.dispatch(SqlLine.java:809)

        at sqlline.SqlLine.initArgs(SqlLine.java:588)

        at sqlline.SqlLine.begin(SqlLine.java:661)

        at sqlline.SqlLine.start(SqlLine.java:398)

        at sqlline.SqlLine.main(SqlLine.java:291)

Caused by: org.apache.hadoop.hbase.snapshot.SnapshotCreationException:
org.apache.hadoop.hbase.snapshot.SnapshotCreationException: Table
'SYSTEM.CATALOG' doesn't exist, can't take snapshot.

        at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(Snapsho
tManager.java:562)

        at
org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.
java:1247)

        at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.call
BlockingMethod(MasterProtos.java:48522)

        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)

        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)

        at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)

        at
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)

        at java.lang.Thread.run(Thread.java:745)

 

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

        at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces
sorImpl.java:57)

        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

        at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.j
ava:106)

        at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.
java:95)

        at
org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetry
ingCaller.java:236)

        at
org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetry
ingCaller.java:250)

        at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:150)

        at
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:42
12)

        at
org.apache.hadoop.hbase.client.HBaseAdmin.takeSnapshotAsync(HBaseAdmin.java:
3572)

        at
org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:3516)

        at
org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:3474)

        at
org.apache.hadoop.hbase.client.HBaseAdmin.snapshot(HBaseAdmin.java:3393)

        at
org.apache.phoenix.query.ConnectionQueryServicesImpl.createSnapshot(Connecti
onQueryServicesImpl.java:3028)

        ... 22 more

Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbas
e.snapshot.SnapshotCreationException):
org.apache.hadoop.hbase.snapshot.SnapshotCreationException: Table
'SYSTEM.CATALOG' doesn't exist, can't take snapshot.

        at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(Snapsho
tManager.java:562)

        at
org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.
java:1247)

        at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.call
BlockingMethod(MasterProtos.java:48522)

        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)

        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)

        at
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)

        at
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)

        at java.lang.Thread.run(Thread.java:745)

 

        at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1234)

        at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpc
Client.java:223)

        at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementati
on.callBlockingMethod(AbstractRpcClient.java:328)

        at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$Blocki
ngStub.snapshot(MasterProtos.java:54993)

        at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$4
.snapshot(ConnectionManager.java:1874)

        at
org.apache.hadoop.hbase.client.HBaseAdmin$40.call(HBaseAdmin.java:3577)

        at
org.apache.hadoop.hbase.client.HBaseAdmin$40.call(HBaseAdmin.java:3572)

        at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetrying
Caller.java:136)

        ... 28 more

sqlline version 1.2.0

4)      We are still able to connect hbase-1.1.1 from apache phoenix-4.8.1.
Need your help to connect hbase-1.1.1 from apache phoenix-4.10.0

 

Regards

Pratap