You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "huaxiang sun (JIRA)" <ji...@apache.org> on 2016/07/27 21:57:20 UTC
[jira] [Commented] (HBASE-16293)
TestSnapshotFromMaster#testSnapshotHFileArchiving flakey
[ https://issues.apache.org/jira/browse/HBASE-16293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15396482#comment-15396482 ]
huaxiang sun commented on HBASE-16293:
--------------------------------------
Spent some time internally with Matteo, we found the issue.
The issue is that compaction is disabled for the table. hbase.hstore.blockingStoreFiles is 12, and the test tries to create 20 store files. When 12 is reached, it tries to compact but compaction is disabled. Then the memstore size is exceeded and region server starts to return RegionTooBusyException to the client. The client will retry and stuck forever. Looking into the logic of the unitest, it does not assume how many store files it needs to create.
It is related with HBASE-16235.
2016-07-27 13:06:18,272 INFO [hconnection-0x5b178b4b-shared-pool15-t22] client.AsyncProcess$AsyncRequestFutureImpl(1266): #26, table=test, attempt=13/36 failed=4468ops, last exception: org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Above memstore limit, regionName=test,,1469649897929.2e3e92fa2512073f502609defa3845f8., server=172.16.1.53,53409,1469649884198, memstoreSize=1363248, blockingMemStoreSize=100000
at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:3719)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2945)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:2896)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:878)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:840)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2359)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:38447)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2212)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:118)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
on 172.16.1.53,53409,1469649884198, tracking started null, retrying after=20180ms, replay=4468ops
> TestSnapshotFromMaster#testSnapshotHFileArchiving flakey
> --------------------------------------------------------
>
> Key: HBASE-16293
> URL: https://issues.apache.org/jira/browse/HBASE-16293
> Project: HBase
> Issue Type: Bug
> Components: test
> Affects Versions: 2.0.0
> Reporter: huaxiang sun
> Assignee: huaxiang sun
>
> Got the following stack trace for this failure, not sure if it is related with HBASE-9072
> -------------------------------------------------------
> T E S T S
> -------------------------------------------------------
> Running org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 336.042 sec <<< FAILURE! - in org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster
> testSnapshotHFileArchiving(org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster) Time elapsed: 303.771 sec <<< ERROR!
> org.junit.runners.model.TestTimedOutException: test timed out after 300000 milliseconds
> at java.lang.Object.wait(Native Method)
> at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1810)
> at org.apache.hadoop.hbase.client.AsyncProcess.waitForMaximumCurrentTasks(AsyncProcess.java:1784)
> at org.apache.hadoop.hbase.client.AsyncProcess.waitForAllPreviousOpsAndReset(AsyncProcess.java:1860)
> at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:241)
> at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:191)
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:979)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:576)
> at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:2002)
> at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1979)
> at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1967)
> at org.apache.hadoop.hbase.HBaseTestingUtility.loadTable(HBaseTestingUtility.java:1945)
> at org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster.testSnapshotHFileArchiving(TestSnapshotFromMaster.java:297)
> Results :
> Tests in error:
> TestSnapshotFromMaster.testSnapshotHFileArchiving:297->Object.wait:-2 ยป TestTimedOut
> Tests run: 4, Failures: 0, Errors: 1, Skipped: 0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)