You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Nick Dimiduk (JIRA)" <ji...@apache.org> on 2016/02/12 07:39:18 UTC
[jira] [Updated] (HBASE-15140) Fix ResultBoundedCompletionService
deadlock
[ https://issues.apache.org/jira/browse/HBASE-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Dimiduk updated HBASE-15140:
---------------------------------
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Pushed cherry-pick of HBASE-14812 to branch-1.1. Closing this one as duplicate. Thanks for pointing it out [~alok_at_cloudability].
> Fix ResultBoundedCompletionService deadlock
> --------------------------------------------
>
> Key: HBASE-15140
> URL: https://issues.apache.org/jira/browse/HBASE-15140
> Project: HBase
> Issue Type: Bug
> Components: Client, hbase
> Affects Versions: 1.1.2
> Reporter: Alok Singh
> Priority: Critical
> Fix For: 1.1.4
>
> Attachments: 14812-cherrypick-branch-1.1.patch
>
>
> See: https://issues.apache.org/jira/browse/HBASE-14812
> We ran into this deadlock issue on the hbase 1.1.2 build
> {code}
> phoenix-1-thread-1340 id=3183 state=WAITING
> - waiting on <0x1ad981d3> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x1ad981d3> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
> at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
> at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Locked synchronizers: count = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@55145434
> phoenix-1-thread-1341 id=3184 state=WAITING
> - waiting on <0x22e46b2c> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x22e46b2c> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
> at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
> at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Locked synchronizers: count = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@25efbbe8
> phoenix-1-thread-1342 id=3185 state=WAITING
> - waiting on <0x6723add5> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x6723add5> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
> at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
> at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Locked synchronizers: count = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@8561ef4
> phoenix-1-thread-1343 id=3186 state=WAITING
> - waiting on <0x31fd0245> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x31fd0245> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
> at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
> at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Locked synchronizers: count = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@13961870
> phoenix-1-thread-1344 id=3187 state=WAITING
> - waiting on <0x79f31d27> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x79f31d27> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
> at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
> at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:105)
> at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:100)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Locked synchronizers: count = 1
> - java.util.concurrent.ThreadPoolExecutor$Worker@64d45bc2
> phoenix-1-thread-1345 id=3188 state=WAITING
> - waiting on <0x331112aa> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> - locked <0x331112aa> (a [Lorg.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture;)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.take(ResultBoundedCompletionService.java:148)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:188)
> at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:59)
> at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
> at org.apache.phoenix.iterate.TableResultIterator.getDelegate(TableResultIterator.java:67)
> at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:88)
> at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:79)
> {code}
> The patch attached to HBASE-14812 applies cleanly to version 1.1.2, so, it should easy to port and apply.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)