Create table with region replication hung

[Melvin Kanasseril <Me...@Sophos.com>]

Hi folks,

We have run into an odd issue where when we try to create a table with region replication > 1,  the operation gets hung. This happens when we do the same over the shell or the java client.

On the shell -
create 'myTable', 'myCF', {REGION_REPLICATION => ‘3'} gets hung, but
create 'myTable', 'myCF', {REGION_REPLICATION => ‘1'} does not and creates the table just fine.

On the client - we never come out of Admin.createTable. Looking at the logs on the master, we see the table is created message and then it gets hung after 2-3 occurrences of these kinds of messages:

INFO org.apache.hadoop.hbase.master.RegionStates        Transition {1ffb0227eca53d899f1b52bd255b9c31 state=OPEN, ts=1454709689241, server=serverlocation,60020,1454709073523} to {1ffb0227eca53d899f1b52bd255b9c31 state=OFFLINE, ts=1454709689241, server=serverlocation,60020,1454709073523}.

I do not see any errors in the logs. We tried the same both with :
hbase.region.replica.replication.enabled=false, hbase.regionserver.storefile.refresh.period=value and
hbase.region.replica.replication.enabled=true and saw no difference in behavior.

We are running CDH 5.5.1. This is a small test cluster with 2 Masters(active/passive) and 3 Region Servers.

Have you seen this before? Any thoughts on what might be causing this?

Thanks,
Melvin