You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Austin Heyne <ah...@ccri.com> on 2018/07/10 23:47:20 UTC
EMR Read Replica Metadata Table Name
I currently have an EMR cluster that's running a continuous ingest. I'd
like to spin up read-only clusters with Spark and Zeppelin to query
with. I've gotten the replica up and running with Spark but when an
executor tries to query HBase it's throwing NotServingRegionExceptions.
"""
18/07/10 23:23:36 INFO RpcRetryingCaller: Call exception, tries=10,
retries=35, started=38532 ms ago, cancelled=false,
msg=org.apache.hadoop.hbase.NotServingRegionException: Region
hbase:meta,,1 is not online on
ip-10-0-24-63.ec2.internal,16020,1531253339025
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3008)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1144)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2476)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2757)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
"""
Which makes sense because on an EMR read replica the metadata table
should be something like hbase:meta_j-2ZMF9CFOOBAR,,1 not hbase:meta,,1.
Further hbase:meta_j-2ZMF9CFOOBAR,,1 is available on
ip-10-0-24-63.ec2.internal. Does anyone know why this is happening or
how to fix it?
Thanks,
--
Austin L. Heyne
Re: EMR Read Replica Metadata Table Name
Posted by Austin Heyne <ah...@ccri.com>.
To expand on this, I'm also having the inverse issue. I had to take down
our main HBase today and now when I try to run hbck it is trying to look
for the hbase:meta,,1 table on a region server that is serving a read
replica metadata table and failing.
It seems like something is messed up on HBase knowing which metadata
table to use when and where that metadata table is located. I image this
is all state that should be maintained in zookeeper but I don't know
where things are going wrong.
-Austin
On 07/10/2018 07:47 PM, Austin Heyne wrote:
> I currently have an EMR cluster that's running a continuous ingest.
> I'd like to spin up read-only clusters with Spark and Zeppelin to
> query with. I've gotten the replica up and running with Spark but when
> an executor tries to query HBase it's throwing
> NotServingRegionExceptions.
>
> """
> 18/07/10 23:23:36 INFO RpcRetryingCaller: Call exception, tries=10,
> retries=35, started=38532 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.NotServingRegionException: Region
> hbase:meta,,1 is not online on
> ip-10-0-24-63.ec2.internal,16020,1531253339025
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3008)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1144)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.newRegionScanner(RSRpcServices.java:2476)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2757)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2339)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
> """
>
> Which makes sense because on an EMR read replica the metadata table
> should be something like hbase:meta_j-2ZMF9CFOOBAR,,1 not
> hbase:meta,,1. Further hbase:meta_j-2ZMF9CFOOBAR,,1 is available on
> ip-10-0-24-63.ec2.internal. Does anyone know why this is happening or
> how to fix it?
>
> Thanks,
>
--
Austin L. Heyne