[jira] [Commented] (HBASE-15573) Indefinite pause while trying to cleanup data

["Jorge Figueira (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-15573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15220856#comment-15220856 ] 

Jorge Figueira commented on HBASE-15573:
----------------------------------------

After 2 months with Hbase working fine this error occurred with the RPC java client. Hbase server tells its all OK with Region and Master but it cant scan any data with the same configuration i have from 2 months ago that was working perfectly until yesterday.

After many tries(repairs,updates) i decided to clean all data and my client start working again (without any configuration changed). Something occurred with Hbase and it wasn't any change in any configuration file inside Hbase (server and client) or network. Clearly was a issue inside Hbase.

I have a snapshot of that state to try to understand what occurred and prevent happening again.


With the bug described above when i try to export data from Hbase using (hbase org.apache.hadoop.hbase.mapreduce.Export JAMES_MAILBOXES xpto.tmp) some nasty exceptions occur
2016-03-31 15:37:39,745 DEBUG [B.defaultRpcServer.handler=23,queue=2,port=16020] ipc.RpcServer: B.defaultRpcServer.handler=23,queue=2,port=16020: callId: 36796 service: ClientService methodName: Scan size: 586 connection: HBASE:57813
org.apache.hadoop.hbase.exceptions.RegionOpeningException: Region JAMES_MAILBOXES,,1456359497674.b038389cb6f8d608abe86265dd90079f. is opening on HBASE,16020,1459435051912
	at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2895)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:947)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2234)
	at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32205)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
	at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
	at java.lang.Thread.run(Thread.java:745)



> Indefinite pause while trying to cleanup data
> ---------------------------------------------
>
>                 Key: HBASE-15573
>                 URL: https://issues.apache.org/jira/browse/HBASE-15573
>             Project: HBase
>          Issue Type: Bug
>    Affects Versions: 1.1.2, 1.1.4
>            Reporter: Jorge Figueira
>            Priority: Blocker
>         Attachments: hbase-hadoop-master-HBASE.log, hbase-hadoop-regionserver-HBASE.log, hbase-hadoop-zookeeper-HBASE.log
>
>
> Can't retrieve any information with hbase  rpc java client.
> With hbase shell its possible to scan data and retrieve all the information normally.
> But with any rpc client region server don't retrieve data, all data come with null values.
> Region Server log:
> DEBUG [RpcServer.reader=2,bindAddress=HBASE,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client SERVER:37088 because read count=-1
> DEBUG [RpcServer.reader=2,bindAddress=HBASE,port=16020] ipc.RpcServer: RpcServer.listener,port=16020: DISCONNECTING client SERVER2:36997 because read count=-1
> Master log:
> 2016-03-31 18:16:27,998 DEBUG [ProcedureExecutorTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner: No completed procedures to cleanup.
> 2016-03-31 18:16:57,998 DEBUG [ProcedureExecutorTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner: No completed procedures to cleanup.
> 2016-03-31 18:17:27,998 DEBUG [ProcedureExecutorTimeout] procedure2.ProcedureExecutor$CompletedProcedureCleaner: No completed procedures to cleanup



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)