[jira] [Updated] (PHOENIX-6571) Hang of connections after OutOfMemoryError

["Aleksey Stavrov (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PHOENIX-6571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aleksey Stavrov updated PHOENIX-6571:
-------------------------------------
    Description: 
We often have connection hangs after OOM. Hangs often occur on locks.

I made a test case locally that reproduces this problem, but I don't know Java, so I have a complex bundle (perl -> dbd_jdbc -> hbase/phoenix) to use phoenix, so because of this i am not showing a minimal example to reproduce this behavior.

Here is the error log shown by dbd_jdbc: 
{noformat}
21/10/08 09:59:32 [main] INFO jdbc.Server: [Server] accepting connections
 21/10/08 09:59:58 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 09:59:58 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 09:59:58 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 09:59:58 [Thread-0] INFO jdbc.Connection: Client started
 21/10/08 09:59:58 [Thread-2] INFO jdbc.Connection: Client started
 21/10/08 09:59:58 [Thread-1] INFO jdbc.Connection: Client started
 21/10/08 09:59:58 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 09:59:58 [Thread-3] INFO jdbc.Connection: Client started
 WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/hbase/phoenix-hbase-1.4-4.16.1-bin/phoenix-client-hbase-1.4-4.16.1.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
21/10/08 09:59:59 [Thread-2] WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
 21/10/08 09:59:59 [Thread-2] WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
 21/10/08 10:00:04 [Thread-1] WARN jdbc.Connection: Error
 org.apache.phoenix.exception.PhoenixIOException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
        at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
        at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
        at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
        at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
        at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
        at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
        at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
        at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
        at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
        at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
        ... 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
21/10/08 10:00:04 [Thread-2] WARN jdbc.Connection: Error
 org.apache.phoenix.exception.PhoenixIOException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
        at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
        at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
        at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
        at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
        at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
        at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
        at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
        at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
        at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
        at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
        ... 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
21/10/08 10:00:04 [Thread-3] WARN jdbc.Connection: Error
 org.apache.phoenix.exception.PhoenixIOException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
        at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
        at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
        at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
        at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
        at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
        at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
        at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
        at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
        at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
        at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
        ... 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
        at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:293)
        at org.apache.phoenix.iterate.OrderedResultIterator.peek(OrderedResultIterator.java:319)
        at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:138)
        at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:121)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:185)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        ... 1 more
21/10/08 10:00:04 [Thread-0] WARN jdbc.Connection: Error
 org.apache.phoenix.exception.PhoenixIOException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
        at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
        at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
        at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
        at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
        at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
        at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
        at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
        at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
        at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
        at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
        ... 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
21/10/08 10:00:08 [Thread-2] WARN jdbc.Connection: Error
 org.apache.phoenix.exception.PhoenixIOException: java.lang.OutOfMemoryError: Java heap space
        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
        at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
        at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
        at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
        at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
        at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
        at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
        at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
        at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
        at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
        at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
        ... 12 more
Caused by: java.lang.OutOfMemoryError: Java heap space
21/10/08 10:00:08 [Thread-2] INFO jdbc.Connection: Client done
 21/10/08 10:00:30 [Thread-3] INFO jdbc.Connection: Client done
 21/10/08 10:00:30 [Thread-0] INFO jdbc.Connection: Client done
 21/10/08 10:00:30 [Thread-1] INFO jdbc.Connection: Client done


21/10/08 10:00:53 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 10:00:53 [Thread-9] INFO jdbc.Connection: Client started
 21/10/08 10:00:53 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 10:00:53 [Thread-10] INFO jdbc.Connection: Client started
 21/10/08 10:00:53 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 10:00:53 [Thread-11] INFO jdbc.Connection: Client started
 21/10/08 10:00:53 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 10:00:53 [Thread-12] INFO jdbc.Connection: Client started
 21/10/08 10:00:56 [Thread-11] WARN jdbc.Connection: Rollback due to fatal error
 21/10/08 10:00:56 [Thread-11] FATAL jdbc.Connection: Error; ending connection
 java.lang.OutOfMemoryError: Java heap space
21/10/08 10:00:56 [Thread-11] INFO jdbc.Connection: Client done
 21/10/08 10:00:56 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 10:00:56 [Thread-13] INFO jdbc.Connection: Client started

21/10/08 10:01:18 [Thread-12] INFO jdbc.Connection: Client done
 21/10/08 10:01:18 [Thread-10] INFO jdbc.Connection: Client done
 21/10/08 10:01:21 [Thread-9] INFO jdbc.Connection: Client done
 21/10/08 10:01:21 [Thread-13] INFO jdbc.Connection: Client done

21/10/08 10:01:27 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 10:01:27 [Thread-14] INFO jdbc.Connection: Client started
 21/10/08 10:01:27 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 10:01:27 [Thread-15] INFO jdbc.Connection: Client started
 21/10/08 10:01:27 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 10:01:27 [Thread-16] INFO jdbc.Connection: Client started
 21/10/08 10:01:27 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 10:01:27 [Thread-17] INFO jdbc.Connection: Client started
 Exception in thread "IPC Client (1380923847) connection to bca736ff9b7d/172.17.0.2:35469 from root" java.lang.OutOfMemoryError: Java heap space
21/10/08 10:01:49 [Thread-14] INFO jdbc.Connection: Client done
 21/10/08 10:01:49 [Thread-16] INFO jdbc.Connection: Client done
 21/10/08 10:03:29 [Thread-15] WARN jdbc.Connection: Error
 org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60279: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000 r
ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359

        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
        at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
        at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
        at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
        at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
        at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
        at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
        at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
        at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
        at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
        at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60279: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000 r
ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359

        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
        ... 12 more
Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60279: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000 r
ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359

        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
        at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:192)
        at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
        at org.apache.phoenix.iterate.OrderedResultIterator.peek(OrderedResultIterator.java:319)
        at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:138)
        at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:121)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:185)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        ... 1 more
Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60279: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000 r
ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359


        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:329)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:242)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
        at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
        at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
        at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:162)
        ... 10 more
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60279: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000 row '' on table 'Y' at reg
ion=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)
        at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
        ... 3 more
Caused by: java.io.IOException: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000
        at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
        at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96)
        at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:188)
        at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:625)
        at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:700)
        at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:428)
        ... 1 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000
        ... 5 more
21/10/08 10:03:29 [Thread-17] WARN jdbc.Connection: Error
 org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60104: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999 r
ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359

        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
        at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
        at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
        at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
        at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
        at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
        at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
        at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
        at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
        at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
        at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60104: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999 r
ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359

        at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
        ... 12 more
Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60104: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999 r
ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359

        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
        at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:192)
        at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
        at org.apache.phoenix.iterate.OrderedResultIterator.peek(OrderedResultIterator.java:319)
        at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:138)
        at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:121)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:185)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        ... 1 more
Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60104: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999 r
ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359

        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
        at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:174)
        at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:185)
        ... 9 more
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60104: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999 r
ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359

        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:329)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:242)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
        at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
        at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
        at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:162)
        ... 10 more
Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60104: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999 row '' on table 'Y' at reg
ion=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)
        at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
        ... 3 more
Caused by: java.io.IOException: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999
        at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
        at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
        at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96)
        at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:188)
        at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:625)
        at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:700)
        at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:428)
        ... 1 more
Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999
        ... 5 more

21/10/08 10:07:07 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
 21/10/08 10:07:07 [Thread-18] INFO jdbc.Connection: Client started
{noformat}

After the OOM happened, all connections to dbd_jdbc hang and very often have such a stack trace:
{noformat}
"Thread-18" #197 prio=5 os_prio=0 cpu=21.70ms elapsed=1822.79s tid=0x00007ffaa4124800 nid=0x1c84 waiting on condition  [0x00007ff9c0ecc000]                                                                                                                                    
   java.lang.Thread.State: WAITING (parking)                                                                                                                                                                                                                                   
        at jdk.internal.misc.Unsafe.park(java.base@11.0.12/Native Method)                                                                                                                                                                                                      
        - parking to wait for  <0x00000000f82e9998> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)                                                                                                                                                                   
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.12/LockSupport.java:194)                                                                                                                                                                                 
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.12/AbstractQueuedSynchronizer.java:885)                                                                                                                                  
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.12/AbstractQueuedSynchronizer.java:917)                                                                                                                                          
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.12/AbstractQueuedSynchronizer.java:1240)                                                                                                                                               
        at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.12/ReentrantLock.java:267)                                                                                                                                                                             
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1326)                                                                                                                                          
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1230)                                                                                                                                                
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1214)                                                                                                                                                
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1171)                                                                                                                                                
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:1003)                                                                                                                                           
        at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:80)                                                                                                                                                                             
        at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:519)                                                                                                                                                                                            
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:735)                                                                                                                                                                                     
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:705)                                                                                                                                                                                     
        at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1808)                                                                                                                                                                                         
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1763)                                                                                                                                                                                          
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1743)                                                                                                                                                                                          
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1769)                                                                                                                                                 
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1749)                                                                                                                                                 
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1996)                                                                                                                                                                
        at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:657)                                                                                                                                                                                       
        at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:545)                                                                                                                                                                                       
        at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:772)                                                                                                                                                                    
        at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:442)                                                                                                                                                                     
        at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:227)                                                                                                                                                                                  
        at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:205)                                                                                                                                                                                  
        at org.apache.phoenix.util.ParseNodeUtil.rewrite(ParseNodeUtil.java:177)                                                                                                                                                                                               
        at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:489)                                                                                                                                                           
        at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:462)                                                                                                                                                           
        at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:304)                                                                                                                                                                                          
        at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:293)                                                                                                                                                                                          
        at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)                
        at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:292)
        at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:285)
        at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
        at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
        at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:744)                          
        at com.vizdom.dbd.jdbc.Connection.run(Connection.java:233)          
        at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)

{noformat}
 But sometimes after such exception
{noformat}
Exception in thread "IPC Client (55674532) connection to bca736ff9b7d/172.17.0.2:35469 from root" java.lang.OutOfMemoryError: Java heap space
21/10/08 09:19:53 [Thread-0-SendThread(127.0.0.1:2181)] WARN server.ZooKeeperThread: Exception occurred from thread Thread-0-SendThread(127.0.0.1:2181)
 java.lang.OutOfMemoryError: Java heap space  
{noformat}
we have such stacktrace:
{noformat}
Thread-15" #146 prio=5 os_prio=0 cpu=2.06ms elapsed=190.12s tid=0x00007f130c110000 nid=0x1a1a in Object.wait()  [0x00007f11eefe9000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(java.base@11.0.12/Native Method)
        - waiting on <no object reference available>                                                                                                                                                                                                                          
        at java.lang.Object.wait(java.base@11.0.12/Object.java:328)
        at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
        - waiting to re-lock in wait() <0x00000000ff6e4d98> (a org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$Packet)
        at org.apache.phoenix.shaded.org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:629)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:487)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:168)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:607)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1266)
        - locked <0x00000000f73ea510> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1227)
        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:356)                                                                                                                                    
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
        at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
        at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1341)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1230)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1214)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1171)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:1003)
        at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:80)
        at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:519)
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:735)
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:705)
        at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1808)
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1763)
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1743)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1769)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1749)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1996)
        at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:657)
        at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:545)
        at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:772)
        at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:442)
        at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:227)
        at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:205)
        at org.apache.phoenix.util.ParseNodeUtil.rewrite(ParseNodeUtil.java:177)
        at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:489)
        at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:462)
        at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:304)
        at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:293)
        at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
        at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:292)
        at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:285)
        at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
        at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
        at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:744)
        at com.vizdom.dbd.jdbc.Connection.run(Connection.java:233)
        at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)
{noformat}
If it is difficult to reproduce this problem, then write to me here, I will try to reproduce with a minimal example in Java.

  was:
We often have connection hangs after OOM. Hangs often occur on locks.

I made a test case locally that reproduces this problem, but I don't know Java, so I have a complex bundle (perl -> dbd_jdbc -> hbase/phoenix) to use phoenix, so because of this i am not showing a minimal example to reproduce this behavior.

Here is the error log shown by dbd_jdbc:  [https://pastebin.com/sHcH2iVq]

After the OOM happened, all connections to dbd_jdbc hang and very often have such a stack trace:
{noformat}
"Thread-18" #197 prio=5 os_prio=0 cpu=21.70ms elapsed=1822.79s tid=0x00007ffaa4124800 nid=0x1c84 waiting on condition  [0x00007ff9c0ecc000]                                                                                                                                    
   java.lang.Thread.State: WAITING (parking)                                                                                                                                                                                                                                   
        at jdk.internal.misc.Unsafe.park(java.base@11.0.12/Native Method)                                                                                                                                                                                                      
        - parking to wait for  <0x00000000f82e9998> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)                                                                                                                                                                   
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.12/LockSupport.java:194)                                                                                                                                                                                 
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.12/AbstractQueuedSynchronizer.java:885)                                                                                                                                  
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.12/AbstractQueuedSynchronizer.java:917)                                                                                                                                          
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.12/AbstractQueuedSynchronizer.java:1240)                                                                                                                                               
        at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.12/ReentrantLock.java:267)                                                                                                                                                                             
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1326)                                                                                                                                          
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1230)                                                                                                                                                
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1214)                                                                                                                                                
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1171)                                                                                                                                                
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:1003)                                                                                                                                           
        at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:80)                                                                                                                                                                             
        at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:519)                                                                                                                                                                                            
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:735)                                                                                                                                                                                     
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:705)                                                                                                                                                                                     
        at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1808)                                                                                                                                                                                         
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1763)                                                                                                                                                                                          
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1743)                                                                                                                                                                                          
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1769)                                                                                                                                                 
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1749)                                                                                                                                                 
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1996)                                                                                                                                                                
        at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:657)                                                                                                                                                                                       
        at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:545)                                                                                                                                                                                       
        at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:772)                                                                                                                                                                    
        at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:442)                                                                                                                                                                     
        at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:227)                                                                                                                                                                                  
        at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:205)                                                                                                                                                                                  
        at org.apache.phoenix.util.ParseNodeUtil.rewrite(ParseNodeUtil.java:177)                                                                                                                                                                                               
        at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:489)                                                                                                                                                           
        at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:462)                                                                                                                                                           
        at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:304)                                                                                                                                                                                          
        at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:293)                                                                                                                                                                                          
        at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)                
        at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:292)
        at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:285)
        at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
        at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
        at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:744)                          
        at com.vizdom.dbd.jdbc.Connection.run(Connection.java:233)          
        at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)

{noformat}
 But sometimes after such exception
{noformat}
Exception in thread "IPC Client (55674532) connection to bca736ff9b7d/172.17.0.2:35469 from root" java.lang.OutOfMemoryError: Java heap space
21/10/08 09:19:53 [Thread-0-SendThread(127.0.0.1:2181)] WARN server.ZooKeeperThread: Exception occurred from thread Thread-0-SendThread(127.0.0.1:2181)
 java.lang.OutOfMemoryError: Java heap space  
{noformat}
we have such stacktrace:
{noformat}
Thread-15" #146 prio=5 os_prio=0 cpu=2.06ms elapsed=190.12s tid=0x00007f130c110000 nid=0x1a1a in Object.wait()  [0x00007f11eefe9000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(java.base@11.0.12/Native Method)
        - waiting on <no object reference available>                                                                                                                                                                                                                          
        at java.lang.Object.wait(java.base@11.0.12/Object.java:328)
        at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
        - waiting to re-lock in wait() <0x00000000ff6e4d98> (a org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$Packet)
        at org.apache.phoenix.shaded.org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:629)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:487)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:168)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:607)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
        at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
        at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1266)
        - locked <0x00000000f73ea510> (a java.lang.Object)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1227)
        at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:356)                                                                                                                                    
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
        at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
        at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1341)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1230)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1214)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1171)
        at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:1003)
        at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:80)
        at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:519)
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:735)
        at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:705)
        at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1808)
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1763)
        at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1743)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1769)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1749)
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1996)
        at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:657)
        at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:545)
        at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:772)
        at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:442)
        at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:227)
        at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:205)
        at org.apache.phoenix.util.ParseNodeUtil.rewrite(ParseNodeUtil.java:177)
        at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:489)
        at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:462)
        at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:304)
        at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:293)
        at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
        at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:292)
        at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:285)
        at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
        at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
        at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:744)
        at com.vizdom.dbd.jdbc.Connection.run(Connection.java:233)
        at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)
{noformat}
If it is difficult to reproduce this problem, then write to me here, I will try to reproduce with a minimal example in Java.


> Hang of connections after OutOfMemoryError 
> -------------------------------------------
>
>                 Key: PHOENIX-6571
>                 URL: https://issues.apache.org/jira/browse/PHOENIX-6571
>             Project: Phoenix
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 4.16.1
>            Reporter: Aleksey Stavrov
>            Priority: Major
>
> We often have connection hangs after OOM. Hangs often occur on locks.
> I made a test case locally that reproduces this problem, but I don't know Java, so I have a complex bundle (perl -> dbd_jdbc -> hbase/phoenix) to use phoenix, so because of this i am not showing a minimal example to reproduce this behavior.
> Here is the error log shown by dbd_jdbc: 
> {noformat}
> 21/10/08 09:59:32 [main] INFO jdbc.Server: [Server] accepting connections
>  21/10/08 09:59:58 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 09:59:58 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 09:59:58 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 09:59:58 [Thread-0] INFO jdbc.Connection: Client started
>  21/10/08 09:59:58 [Thread-2] INFO jdbc.Connection: Client started
>  21/10/08 09:59:58 [Thread-1] INFO jdbc.Connection: Client started
>  21/10/08 09:59:58 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 09:59:58 [Thread-3] INFO jdbc.Connection: Client started
>  WARNING: An illegal reflective access operation has occurred
> WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/hbase/phoenix-hbase-1.4-4.16.1-bin/phoenix-client-hbase-1.4-4.16.1.jar) to method sun.security.krb5.Config.getInstance()
> WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
> WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
> WARNING: All illegal access operations will be denied in a future release
> 21/10/08 09:59:59 [Thread-2] WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
>  21/10/08 09:59:59 [Thread-2] WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-phoenix.properties,hadoop-metrics2.properties
>  21/10/08 10:00:04 [Thread-1] WARN jdbc.Connection: Error
>  org.apache.phoenix.exception.PhoenixIOException: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
>         at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
>         at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
>         at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
>         at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
>         at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
>         at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
>         at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
>         at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
>         at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
>         at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
>         ... 12 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 21/10/08 10:00:04 [Thread-2] WARN jdbc.Connection: Error
>  org.apache.phoenix.exception.PhoenixIOException: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
>         at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
>         at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
>         at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
>         at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
>         at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
>         at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
>         at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
>         at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
>         at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
>         at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
>         ... 12 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 21/10/08 10:00:04 [Thread-3] WARN jdbc.Connection: Error
>  org.apache.phoenix.exception.PhoenixIOException: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
>         at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
>         at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
>         at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
>         at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
>         at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
>         at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
>         at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
>         at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
>         at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
>         at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
>         ... 12 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:293)
>         at org.apache.phoenix.iterate.OrderedResultIterator.peek(OrderedResultIterator.java:319)
>         at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:138)
>         at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:121)
>         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>         at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:185)
>         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>         ... 1 more
> 21/10/08 10:00:04 [Thread-0] WARN jdbc.Connection: Error
>  org.apache.phoenix.exception.PhoenixIOException: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
>         at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
>         at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
>         at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
>         at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
>         at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
>         at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
>         at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
>         at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
>         at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
>         at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
>         ... 12 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 21/10/08 10:00:08 [Thread-2] WARN jdbc.Connection: Error
>  org.apache.phoenix.exception.PhoenixIOException: java.lang.OutOfMemoryError: Java heap space
>         at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
>         at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
>         at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
>         at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
>         at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
>         at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
>         at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
>         at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
>         at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
>         at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
>         at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
>         ... 12 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 21/10/08 10:00:08 [Thread-2] INFO jdbc.Connection: Client done
>  21/10/08 10:00:30 [Thread-3] INFO jdbc.Connection: Client done
>  21/10/08 10:00:30 [Thread-0] INFO jdbc.Connection: Client done
>  21/10/08 10:00:30 [Thread-1] INFO jdbc.Connection: Client done
> 21/10/08 10:00:53 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 10:00:53 [Thread-9] INFO jdbc.Connection: Client started
>  21/10/08 10:00:53 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 10:00:53 [Thread-10] INFO jdbc.Connection: Client started
>  21/10/08 10:00:53 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 10:00:53 [Thread-11] INFO jdbc.Connection: Client started
>  21/10/08 10:00:53 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 10:00:53 [Thread-12] INFO jdbc.Connection: Client started
>  21/10/08 10:00:56 [Thread-11] WARN jdbc.Connection: Rollback due to fatal error
>  21/10/08 10:00:56 [Thread-11] FATAL jdbc.Connection: Error; ending connection
>  java.lang.OutOfMemoryError: Java heap space
> 21/10/08 10:00:56 [Thread-11] INFO jdbc.Connection: Client done
>  21/10/08 10:00:56 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 10:00:56 [Thread-13] INFO jdbc.Connection: Client started
> 21/10/08 10:01:18 [Thread-12] INFO jdbc.Connection: Client done
>  21/10/08 10:01:18 [Thread-10] INFO jdbc.Connection: Client done
>  21/10/08 10:01:21 [Thread-9] INFO jdbc.Connection: Client done
>  21/10/08 10:01:21 [Thread-13] INFO jdbc.Connection: Client done
> 21/10/08 10:01:27 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 10:01:27 [Thread-14] INFO jdbc.Connection: Client started
>  21/10/08 10:01:27 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 10:01:27 [Thread-15] INFO jdbc.Connection: Client started
>  21/10/08 10:01:27 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 10:01:27 [Thread-16] INFO jdbc.Connection: Client started
>  21/10/08 10:01:27 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 10:01:27 [Thread-17] INFO jdbc.Connection: Client started
>  Exception in thread "IPC Client (1380923847) connection to bca736ff9b7d/172.17.0.2:35469 from root" java.lang.OutOfMemoryError: Java heap space
> 21/10/08 10:01:49 [Thread-14] INFO jdbc.Connection: Client done
>  21/10/08 10:01:49 [Thread-16] INFO jdbc.Connection: Client done
>  21/10/08 10:03:29 [Thread-15] WARN jdbc.Connection: Error
>  org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
> Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60279: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000 r
> ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
>         at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
>         at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
>         at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
>         at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
>         at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
>         at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
>         at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
>         at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
>         at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
> Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60279: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000 r
> ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
>         at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
>         at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
>         ... 12 more
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
> Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60279: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000 r
> ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
>         at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
>         at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:192)
>         at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
>         at org.apache.phoenix.iterate.OrderedResultIterator.peek(OrderedResultIterator.java:319)
>         at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:138)
>         at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:121)
>         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>         at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:185)
>         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>         ... 1 more
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
> Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60279: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000 r
> ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
>         at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:329)
>         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:242)
>         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
>         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
>         at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
>         at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
>         at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
>         at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:162)
>         ... 10 more
> Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60279: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000 row '' on table 'Y' at reg
> ion=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
>         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)
>         at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
>         ... 3 more
> Caused by: java.io.IOException: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000
>         at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
>         at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96)
>         at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:188)
>         at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:625)
>         at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:700)
>         at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:428)
>         ... 1 more
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=144, waitTime=60001, rpcTimetout=60000
>         ... 5 more
> 21/10/08 10:03:29 [Thread-17] WARN jdbc.Connection: Error
>  org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
> Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60104: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999 r
> ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
>         at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1349)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1288)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.getMinHeap(MergeSortResultIterator.java:72)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.minIterator(MergeSortResultIterator.java:93)
>         at org.apache.phoenix.iterate.MergeSortResultIterator.next(MergeSortResultIterator.java:58)
>         at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingResultIterator.java:66)
>         at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
>         at org.apache.phoenix.iterate.OrderedResultIterator.next(OrderedResultIterator.java:258)
>         at org.apache.phoenix.iterate.OrderedAggregatingResultIterator.next(OrderedAggregatingResultIterator.java:52)
>         at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:841)
>         at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:826)
>         at com.vizdom.dbd.jdbc.Connection.run(Connection.java:236)
>         at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
> Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60104: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999 r
> ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
>         at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
>         at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:205)
>         at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:1344)
>         ... 12 more
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
> Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60104: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999 r
> ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
>         at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
>         at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:192)
>         at org.apache.phoenix.iterate.OrderedResultIterator.getResultIterator(OrderedResultIterator.java:282)
>         at org.apache.phoenix.iterate.OrderedResultIterator.peek(OrderedResultIterator.java:319)
>         at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:138)
>         at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:121)
>         at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>         at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:185)
>         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>         ... 1 more
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
> Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60104: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999 r
> ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
>         at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:146)
>         at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:174)
>         at org.apache.phoenix.iterate.TableResultIterator.next(TableResultIterator.java:185)
>         ... 9 more
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=36, exceptions:
> Fri Oct 08 10:02:29 UTC 2021, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=60104: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999 r
> ow '' on table 'Y' at region=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
>         at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.throwEnrichedException(RpcRetryingCallerWithReadReplicas.java:329)
>         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:242)
>         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
>         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
>         at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
>         at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
>         at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
>         at org.apache.phoenix.iterate.ScanningResultIterator.next(ScanningResultIterator.java:162)
>         ... 10 more
> Caused by: java.net.SocketTimeoutException: callTimeout=60000, callDuration=60104: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999 row '' on table 'Y' at reg
> ion=Y,,1633434993536.f40badd438889901753324a943b4ab41., hostname=bca736ff9b7d,35469,1633684529739, seqNum=103359
>         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:178)
>         at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80)
>         ... 3 more
> Caused by: java.io.IOException: Call to bca736ff9b7d/172.17.0.2:35469 failed on local exception: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999
>         at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:389)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
>         at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
>         at org.apache.hadoop.hbase.ipc.Call.setTimeout(Call.java:96)
>         at org.apache.hadoop.hbase.ipc.RpcConnection$1.run(RpcConnection.java:188)
>         at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:625)
>         at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$HashedWheelBucket.expireTimeouts(HashedWheelTimer.java:700)
>         at org.apache.phoenix.shaded.io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:428)
>         ... 1 more
> Caused by: org.apache.hadoop.hbase.ipc.CallTimeoutException: Call id=145, waitTime=60003, rpcTimetout=59999
>         ... 5 more
> 21/10/08 10:07:07 [main] INFO jdbc.Server: [Server] received a connection from /172.17.0.1
>  21/10/08 10:07:07 [Thread-18] INFO jdbc.Connection: Client started
> {noformat}
> After the OOM happened, all connections to dbd_jdbc hang and very often have such a stack trace:
> {noformat}
> "Thread-18" #197 prio=5 os_prio=0 cpu=21.70ms elapsed=1822.79s tid=0x00007ffaa4124800 nid=0x1c84 waiting on condition  [0x00007ff9c0ecc000]                                                                                                                                    
>    java.lang.Thread.State: WAITING (parking)                                                                                                                                                                                                                                   
>         at jdk.internal.misc.Unsafe.park(java.base@11.0.12/Native Method)                                                                                                                                                                                                      
>         - parking to wait for  <0x00000000f82e9998> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)                                                                                                                                                                   
>         at java.util.concurrent.locks.LockSupport.park(java.base@11.0.12/LockSupport.java:194)                                                                                                                                                                                 
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.12/AbstractQueuedSynchronizer.java:885)                                                                                                                                  
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.12/AbstractQueuedSynchronizer.java:917)                                                                                                                                          
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.12/AbstractQueuedSynchronizer.java:1240)                                                                                                                                               
>         at java.util.concurrent.locks.ReentrantLock.lock(java.base@11.0.12/ReentrantLock.java:267)                                                                                                                                                                             
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1326)                                                                                                                                          
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1230)                                                                                                                                                
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1214)                                                                                                                                                
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1171)                                                                                                                                                
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:1003)                                                                                                                                           
>         at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:80)                                                                                                                                                                             
>         at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:519)                                                                                                                                                                                            
>         at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:735)                                                                                                                                                                                     
>         at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:705)                                                                                                                                                                                     
>         at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1808)                                                                                                                                                                                         
>         at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1763)                                                                                                                                                                                          
>         at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1743)                                                                                                                                                                                          
>         at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1769)                                                                                                                                                 
>         at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1749)                                                                                                                                                 
>         at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1996)                                                                                                                                                                
>         at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:657)                                                                                                                                                                                       
>         at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:545)                                                                                                                                                                                       
>         at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:772)                                                                                                                                                                    
>         at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:442)                                                                                                                                                                     
>         at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:227)                                                                                                                                                                                  
>         at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:205)                                                                                                                                                                                  
>         at org.apache.phoenix.util.ParseNodeUtil.rewrite(ParseNodeUtil.java:177)                                                                                                                                                                                               
>         at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:489)                                                                                                                                                           
>         at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:462)                                                                                                                                                           
>         at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:304)                                                                                                                                                                                          
>         at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:293)                                                                                                                                                                                          
>         at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)                
>         at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:292)
>         at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:285)
>         at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
>         at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
>         at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:744)                          
>         at com.vizdom.dbd.jdbc.Connection.run(Connection.java:233)          
>         at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)
> {noformat}
>  But sometimes after such exception
> {noformat}
> Exception in thread "IPC Client (55674532) connection to bca736ff9b7d/172.17.0.2:35469 from root" java.lang.OutOfMemoryError: Java heap space
> 21/10/08 09:19:53 [Thread-0-SendThread(127.0.0.1:2181)] WARN server.ZooKeeperThread: Exception occurred from thread Thread-0-SendThread(127.0.0.1:2181)
>  java.lang.OutOfMemoryError: Java heap space  
> {noformat}
> we have such stacktrace:
> {noformat}
> Thread-15" #146 prio=5 os_prio=0 cpu=2.06ms elapsed=190.12s tid=0x00007f130c110000 nid=0x1a1a in Object.wait()  [0x00007f11eefe9000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(java.base@11.0.12/Native Method)
>         - waiting on <no object reference available>                                                                                                                                                                                                                          
>         at java.lang.Object.wait(java.base@11.0.12/Object.java:328)
>         at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1406)
>         - waiting to re-lock in wait() <0x00000000ff6e4d98> (a org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$Packet)
>         at org.apache.phoenix.shaded.org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1210)
>         at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getData(RecoverableZooKeeper.java:354)
>         at org.apache.hadoop.hbase.zookeeper.ZKUtil.getData(ZKUtil.java:629)
>         at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionState(MetaTableLocator.java:487)
>         at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.getMetaRegionLocation(MetaTableLocator.java:168)
>         at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:607)
>         at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:588)
>         at org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:561)
>         at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1266)
>         - locked <0x00000000f73ea510> (a java.lang.Object)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1227)
>         at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:356)                                                                                                                                    
>         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
>         at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
>         at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
>         at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
>         at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
>         at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1341)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1230)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1214)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1171)
>         at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getRegionLocation(ConnectionManager.java:1003)
>         at org.apache.hadoop.hbase.client.HRegionLocator.getRegionLocation(HRegionLocator.java:80)
>         at org.apache.hadoop.hbase.client.HTable.getRegionLocation(HTable.java:519)
>         at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:735)
>         at org.apache.hadoop.hbase.client.HTable.getKeysAndRegionsInRange(HTable.java:705)
>         at org.apache.hadoop.hbase.client.HTable.getStartKeysInRange(HTable.java:1808)
>         at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1763)
>         at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1743)
>         at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1769)
>         at org.apache.phoenix.query.ConnectionQueryServicesImpl.metaDataCoprocessorExec(ConnectionQueryServicesImpl.java:1749)
>         at org.apache.phoenix.query.ConnectionQueryServicesImpl.getTable(ConnectionQueryServicesImpl.java:1996)
>         at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:657)
>         at org.apache.phoenix.schema.MetaDataClient.updateCache(MetaDataClient.java:545)
>         at org.apache.phoenix.compile.FromCompiler$BaseColumnResolver.createTableRef(FromCompiler.java:772)
>         at org.apache.phoenix.compile.FromCompiler$SingleTableColumnResolver.<init>(FromCompiler.java:442)
>         at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:227)
>         at org.apache.phoenix.compile.FromCompiler.getResolverForQuery(FromCompiler.java:205)
>         at org.apache.phoenix.util.ParseNodeUtil.rewrite(ParseNodeUtil.java:177)
>         at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:489)
>         at org.apache.phoenix.jdbc.PhoenixStatement$ExecutableSelectStatement.compilePlan(PhoenixStatement.java:462)
>         at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:304)
>         at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:293)
>         at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
>         at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:292)
>         at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:285)
>         at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:176)
>         at org.apache.phoenix.jdbc.PhoenixPreparedStatement.execute(PhoenixPreparedStatement.java:183)
>         at com.vizdom.dbd.jdbc.Connection.handleRequest(Connection.java:744)
>         at com.vizdom.dbd.jdbc.Connection.run(Connection.java:233)
>         at java.lang.Thread.run(java.base@11.0.12/Thread.java:829)
> {noformat}
> If it is difficult to reproduce this problem, then write to me here, I will try to reproduce with a minimal example in Java.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)