You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Stas Maksimov <ma...@gmail.com> on 2012/09/04 17:28:36 UTC
Help with troubleshooting the HBase replication setup
Hi there,
I'm trying to set up replication in master-slave mode between two clusters,
and when this works set up master-master replication. Following the
replication FAQ step-by-step, but I can't make it work and have no idea how
to troubleshoot. There seem to be given only one way to find out when it
works, this is to look for these in the region server logs:
Considering 1 rs, with ratio 0.1
Getting 1 rs from peer cluster # 0
Choosing peer 170.22.64.15:62020
Well, whatever I do, I can not see them. When I run add_peer nothing
happens. It's the week as I'm stuck with it -
stopping/starting/reinstalling my clusters, to no avail.
Both clusters are CDH4.0.1. I have hbase.replication=true on both clusters.
My table exists on both clusters. The family is marked with
REPLICATION_SCOPE=1 on both clusters. Machines in both of the clusters can
access each other machine.
Can anyone help please? Where do I look to understand what is wrong?
Setting logging to DEBUG in HBase doesn't give me anything apart from a lot
more noise.
Thanks,
Stas
cannot create a table having a snappy compression algorithm in mac os x.
Posted by Henry JunYoung KIM <he...@gmail.com>.
Hi, hbase users.
I have a problem to create a table with a snappy algorithm in Mac OS X (lion).
I am sure that there is a snappy jar file in $HBASE_HOME/lib, but, it couldn't create it without errors.
environment :
hbase vesion : 0.92.1
distribution mode : pseudo-distributed (1 master, 1 region-server)
region server log :
2012-09-05 20:28:11,882 ERROR org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of region=STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783.
java.io.IOException: Compression algorithm 'snappy' previously failed test.
at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:78)
at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:3237)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3226)
at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3176)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:331)
at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:107)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
master log :
2012-09-05 20:31:18,569 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783. destination server is 192.168.162.80,60020,1346843417187
2012-09-05 20:31:18,569 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan for region STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783.; plan=hri=STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783., src=, dest=192.168.162.80,60020,1346843417187
2012-09-05 20:31:18,569 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783. to 192.168.162.80,60020,1346843417187
2012-09-05 20:31:18,569 DEBUG org.apache.hadoop.hbase.master.ServerManager: New connection to 192.168.162.80,60020,1346843417187
2012-09-05 20:31:18,570 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_OPENING, server=192.168.162.80,60020,1346843417187, region=1fa032ecf18ef9d194cc619f0205f783
2012-09-05 20:31:18,571 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=RS_ZK_REGION_FAILED_OPEN, server=192.168.162.80,60020,1346843417187, region=1fa032ecf18ef9d194cc619f0205f783
2012-09-05 20:31:18,571 DEBUG org.apache.hadoop.hbase.master.handler.ClosedRegionHandler: Handling CLOSED event for 1fa032ecf18ef9d194cc619f0205f783
2012-09-05 20:31:18,571 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Forcing OFFLINE; was=STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783. state=CLOSED, ts=1346844678571, server=192.168.162.80,60020,1346843417187
2012-09-05 20:31:18,571 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: master:60000-0x139961fb9f00000 Creating (or updating) unassigned node for 1fa032ecf18ef9d194cc619f0205f783 with OFFLINE state
2012-09-05 20:31:18,572 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Handling transition=M_ZK_REGION_OFFLINE, server=192.168.162.80,60000,1346843415199, region=1fa032ecf18ef9d194cc619f0205f783
2012-09-05 20:31:18,572 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Found an existing plan for STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783. destination server is 192.168.162.80,60020,1346843417187
2012-09-05 20:31:18,572 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Using pre-existing plan for region STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783.; plan=hri=STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783., src=, dest=192.168.162.80,60020,1346843417187
2012-09-05 20:31:18,572 DEBUG org.apache.hadoop.hbase.master.AssignmentManager: Assigning region STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783. to 192.168.162.80,60020,1346843417187
2012-09-05 20:31:18,572 DEBUG org.apache.hadoop.hbase.master.ServerManager: New connection to 192.168.162.80,60020,1346843417187
2012-09-05 20:31:18,575 ERROR org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: 192.168.162.80,60020,1346843417187 due to org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783. ,which we are already trying to OPEN.
org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783. ,which we are already trying to OPEN.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1624)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1329)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1304)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1299)
at org.apache.hadoop.hbase.master.handler.ClosedRegionHandler.process(ClosedRegionHandler.java:102)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Caused by: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: Received:OPEN for the region:STORAGE_TEST,,1346844467727.1fa032ecf18ef9d194cc619f0205f783. ,which we are already trying to OPEN.
at org.apache.hadoop.hbase.regionserver.HRegionServer.checkIfRegionInTransition(HRegionServer.java:2479)
at org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:2437)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:918)
at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:150)
at $Proxy10.openRegion(Unknown Source)
at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:435)
at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:1592)
... 8 more
Would you share your experiences with this situation ?
Re: Help with troubleshooting the HBase replication setup
Posted by Stas Maksimov <ma...@gmail.com>.
This issue is now solved. Having installed two new clusters, everything
works as expected.
Thanks,
Stas
On Tue, Sep 4, 2012 at 4:28 PM, Stas Maksimov <ma...@gmail.com> wrote:
> Hi there,
>
> I'm trying to set up replication in master-slave mode between two
> clusters, and when this works set up master-master replication. Following
> the replication FAQ step-by-step, but I can't make it work and have no idea
> how to troubleshoot. There seem to be given only one way to find out when
> it works, this is to look for these in the region server logs:
>
> Considering 1 rs, with ratio 0.1
> Getting 1 rs from peer cluster # 0
> Choosing peer 170.22.64.15:62020
>
> Well, whatever I do, I can not see them. When I run add_peer nothing
> happens. It's the week as I'm stuck with it -
> stopping/starting/reinstalling my clusters, to no avail.
>
> Both clusters are CDH4.0.1. I have hbase.replication=true on both
> clusters. My table exists on both clusters. The family is marked with
> REPLICATION_SCOPE=1 on both clusters. Machines in both of the clusters can
> access each other machine.
>
> Can anyone help please? Where do I look to understand what is wrong?
> Setting logging to DEBUG in HBase doesn't give me anything apart from a lot
> more noise.
>
> Thanks,
> Stas
>
>
>
>