You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Dinesh Nithyanandam (Jira)" <ji...@apache.org> on 2020/04/23 04:42:00 UTC
[jira] [Created] (HBASE-24243) Unable to start HRegionserver and
Master node considers as a dead region
Dinesh Nithyanandam created HBASE-24243:
-------------------------------------------
Summary: Unable to start HRegionserver and Master node considers as a dead region
Key: HBASE-24243
URL: https://issues.apache.org/jira/browse/HBASE-24243
Project: HBase
Issue Type: Brainstorming
Components: regionserver
Reporter: Dinesh Nithyanandam
Hi Team,
I am currently using Apache Hbase version - 1.3.6 and I am trying to run Master and region server separately and then join the cluster dynamically but it was region server was not starting and always reports that "*The RegionServer is initializing*!"
Commands used as below: (Master and region are on separate nodes )
Node A - Hbase Master - /opt/hbase/bin/hbase-daemon.sh --config /usr/local/bin/hbase/conf start master
Node B - Hbase Region - /opt/hbase/bin/hbase-daemon.sh --config /usr/local/bin/hbase/conf start regionserver
Environment - Google Compute Engine (GCE) Instance groups/VM's
OS Type - CentOS -7
Also not sure on how to enable reverse DNS across both the machines and whether that is the problem and please do advice on how do i achieve it
*Master logs:*
From the below master logs it clearly says that master is trying to connect to region and then eventually getting disconnected from the client region server
* "*DEBUG [RpcServer.reader=1,bindAddress=pinpoint-master-v000-rh5k.c.gcp-ushi-telemetry-npe.internal,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: DISCONNECTING client 10.148.6.13:45732 because read count=-1. Number of active connections: 1"*
*complete logs*
2020-04-22 19:38:24,812 DEBUG [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: connection from 10.148.6.13:45732; # active connections: 1
2020-04-22 19:38:24,961 DEBUG [RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16000] ipc.RpcServer: RpcServer.FifoWFPBQ.default.handler=29,queue=2,port=16000: callId: 0 service: RegionServerStatusService methodName: RegionServerStartup size: 47 connection: 10.148.6.13:45732
2020-04-22 19:38:30,591 DEBUG [*pinpoint-master-v000-rh5k:16000*.activeMasterManager] ipc.RpcClientImpl: Connecting to *pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal/10.148.6.13:16020*
2020-04-22 19:38:31,268 *DEBUG [hconnection-0x5f02b9cb-shared--pool3-t1] ipc.RpcClientImpl: Connecting to pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal/10.148.6.13:16020*
2020-04-22 19:38:31,478 DEBUG [ProcedureExecutor-3] ipc.RpcClientImpl: Connecting to pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal/10.148.6.13:16020
2020-04-22 19:39:32,714 *DEBUG [RpcServer.reader=1,bindAddress=pinpoint-master-v000-rh5k.c.gcp-ushi-telemetry-npe.internal,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: DISCONNECTING client 10.148.6.13:45732 because read count=-1. Number of active connections: 1*
*Region server logs:*
From the below logs region server discovers the master on it's own but unable to join the cluster with below logs
===============================================================
2020-04-22 19:38:24,675 INFO *[regionserver/pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal/10.148.6.13:16020] regionserver.HRegionServer: reportForDuty to master=pinpoint-master-v000-rh5k.c.gcp-ushi-telemetry-npe.internal,16000*,1587584303253 with port=16020, startcode=1587583634667
2020-04-22 19:38:24,801 DEBUG [regionserver/pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal/10.148.6.13:16020] ipc.RpcClientImpl: Connecting to pinpoint-master-v000-rh5k.c.gcp-ushi-telemetry-npe.internal/10.148.6.154:16000
2020-04-22 19:38:28,005 INFO [regionserver/pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal/10.148.6.13:16020] regionserver.HRegionServer: reportForDuty to master=pinpoint-master-v000-rh5k.c.gcp-ushi-telemetry-npe.internal,16000,1587584303253 with port=16020, startcode=1587583634667
2020-04-22 19:38:28,033 INFO [regionserver/pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal/10.148.6.13:16020] regionserver.HRegionServer: Config from master: hbase.rootdir=hdfs://10.148.6.68:9000/hbase
2020-04-22 19:38:28,033 INFO [regionserver/pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal/10.148.6.13:16020] regionserver.HRegionServer: Config from master: fs.defaultFS=hdfs://10.148.6.68:9000
2020-04-22 19:38:28,033 INFO [regionserver/pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal/10.148.6.13:16020] regionserver.HRegionServer: Config from master: hbase.master.info.port=16010
===============================================================
2020-04-22 19:38:24,801 DEBUG [regionserver/pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal/10.148.6.13:16020] ipc.RpcClientImpl: Connecting to pinpoint-master-v000-rh5k.c.gcp-ushi-telemetry-npe.internal/10.148.6.154:16000
2020-04-22 19:38:30,592 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.148.6.154:53050; # active connections: 1
2020-04-22 19:38:31,269 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.148.6.154:53052; # active connections: 2
2020-04-22 19:38:31,479 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.148.6.154:53056; # active connections: 3
2020-04-22 19:39:32,413 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.RpcServer: RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020: callId: 3 service: AdminService methodName: OpenRegion size: 81 connection: 10.148.6.154:53050
2020-04-22 19:39:32,440 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.RpcServer: RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020: callId: 4 service: AdminService methodName: OpenRegion size: 81 connection: 10.148.6.154:53050
2020-04-22 19:39:32,443 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.RpcServer: RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020: callId: 5 service: AdminService methodName: OpenRegion size: 81 connection: 10.148.6.154:53050
2020-04-22 19:39:32,445 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.RpcServer: RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020: callId: 6 service: AdminService methodName: OpenRegion size: 81 connection: 10.148.6.154:53050
2020-04-22 19:39:32,447 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.RpcServer: RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020: callId: 7 service: AdminService methodName: OpenRegion size: 81 connection: 10.148.6.154:53050
2020-04-22 19:39:32,450 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.RpcServer: RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020: callId: 8 service: AdminService methodName: OpenRegion size: 81 connection: 10.148.6.154:53050
2020-04-22 19:39:32,452 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.RpcServer: RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020: callId: 9 service: AdminService methodName: OpenRegion size: 81 connection: 10.148.6.154:53050
2020-04-22 19:39:32,454 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.RpcServer: RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020: callId: 10 service: AdminService methodName: OpenRegion size: 81 connection: 10.148.6.154:53050
2020-04-22 19:39:32,456 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.RpcServer: RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020: callId: 11 service: AdminService methodName: OpenRegion size: 81 connection: 10.148.6.154:53050
2020-04-22 19:39:32,458 DEBUG [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020] ipc.RpcServer: RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16020: callId: 12 service: AdminService methodName: OpenRegion size: 81 connection: 10.148.6.154:53050
===============================================================
2020-04-23 04:40:07,751 DEBUG [RpcServer.reader=3,bindAddress=pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 10.148.6.13:44272 because read count=-1. Number of active connections: 1
2020-04-23 04:40:17,751 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.148.6.13:44280; # active connections: 1
2020-04-23 04:40:17,752 DEBUG [RpcServer.reader=4,bindAddress=pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 10.148.6.13:44280 because read count=-1. Number of active connections: 1
2020-04-23 04:40:27,752 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.148.6.13:44282; # active connections: 1
2020-04-23 04:40:27,752 DEBUG [RpcServer.reader=5,bindAddress=pinpoint-r-v000-976s.c.gcp-ushi-telemetry-npe.internal,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client 10.148.6.13:44282 because read count=-1. Number of active connections: 1
2020-04-23 04:40:37,752 DEBUG [RpcServer.listener,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: connection from 10.148.6.13:44284; # active connections: 1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)