You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Phil Yang (JIRA)" <ji...@apache.org> on 2016/08/09 07:06:20 UTC
[jira] [Commented] (HBASE-16368) test*WhenRegionMove in
TestPartialResultsFromClientSide is flaky
[ https://issues.apache.org/jira/browse/HBASE-16368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15413074#comment-15413074 ]
Phil Yang commented on HBASE-16368:
-----------------------------------
I think the reason is the scanner timeout in this test is only 2000ms. If the machine is slow and we can not complete the moving within 2000ms, we will fail. Let me upload a patch for a larger timeout
> test*WhenRegionMove in TestPartialResultsFromClientSide is flaky
> ----------------------------------------------------------------
>
> Key: HBASE-16368
> URL: https://issues.apache.org/jira/browse/HBASE-16368
> Project: HBase
> Issue Type: Bug
> Components: Scanners
> Affects Versions: 1.4.0
> Reporter: Guanghao Zhang
>
> This test fail when Hadoop QA run preCommit:
> https://builds.apache.org/job/PreCommit-HBASE-Build/2971/testReport/org.apache.hadoop.hbase/TestPartialResultsFromClientSide/testReversedCompleteResultWhenRegionMove/.
> And I found it is in Flaky Tests Dashboard: http://hbase.x10host.com/flaky-tests/. I run it in my local machine and it may fail, too.
> Test results show that the region location didn't update when scanner callable get a NotServingRegionException or RegionMovedException.
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
> Sat Aug 06 05:55:52 UTC 2016, null, java.net.SocketTimeoutException: callTimeout=2000, callDuration=2157: org.apache.hadoop.hbase.NotServingRegionException: testReversedCompleteResultWhenRegionMove,,1470462949504.5069bd63bf6eda5108acec4fcc087b0e. is closing
> at org.apache.hadoop.hbase.regionserver.HRegion.startRegionOperation(HRegion.java:8233)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2634)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2629)
> at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2623)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.scan(RSRpcServices.java:2490)
> at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34950)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2264)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:118)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> row '' on table 'testReversedCompleteResultWhenRegionMove' at region=testReversedCompleteResultWhenRegionMove,,1470462949504.5069bd63bf6eda5108acec4fcc087b0e., hostname=asf907.gq1.ygridcore.net,38914,1470462943053, seqNum=2
> at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:281)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:213)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
> at org.apache.hadoop.hbase.client.ReversedClientScanner.nextScanner(ReversedClientScanner.java:118)
> at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:166)
> at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:161)
> at org.apache.hadoop.hbase.client.ReversedClientScanner.<init>(ReversedClientScanner.java:56)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:785)
> at org.apache.hadoop.hbase.TestPartialResultsFromClientSide.testReversedCompleteResultWhenRegionMove(TestPartialResultsFromClientSide.java:986)
> {code}
> {code}
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
> Sat Aug 06 16:27:22 CST 2016, null, java.net.SocketTimeoutException: callTimeout=2000, callDuration=3035: Region moved to: hostname=localhost port=58351 startCode=1470472007714. As of locationSeqNum=6. row 'testRow0' on table 'testPartialResultWhenRegionMove' at region=testPartialResultWhenRegionMove,,1470472035048.977faf05c1d6d9990b5559b17aa18913., hostname=localhost,40425,1470472007646, seqNum=2
> at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:281)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:213)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:61)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:326)
> at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:301)
> at org.apache.hadoop.hbase.client.ClientScanner.possiblyNextScanner(ClientScanner.java:247)
> at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:541)
> at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:370)
> at org.apache.hadoop.hbase.TestPartialResultsFromClientSide.testPartialResultWhenRegionMove(TestPartialResultsFromClientSide.java:884)
> Caused by: java.net.SocketTimeoutException: callTimeout=2000, callDuration=3035: Region moved to: hostname=localhost port=58351 startCode=1470472007714. As of locationSeqNum=6. row 'testRow0' on table 'testPartialResultWhenRegionMove' at region=testPartialResultWhenRegionMove,,1470472035048.977faf05c1d6d9990b5559b17aa18913., hostname=localhost,40425,1470472007646, seqNum=2
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:171)
> at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hbase.exceptions.RegionMovedException: Region moved to: hostname=localhost port=58351 startCode=1470472007714. As of locationSeqNum=6.
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332)
> at org.apache.hadoop.hbase.client.ScannerCallable.openScanner(ScannerCallable.java:406)
> at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:210)
> at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:64)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:367)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:341)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:137)
> at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)