You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@phoenix.apache.org by "James Taylor (JIRA)" <ji...@apache.org> on 2016/01/23 23:12:39 UTC
[jira] [Resolved] (PHOENIX-2001) Join create OOM with java heap
space on phoenix client
[ https://issues.apache.org/jira/browse/PHOENIX-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
James Taylor resolved PHOENIX-2001.
-----------------------------------
Resolution: Not A Problem
Phoenix has config parameters that you can use to limit the amount of memory used by hash joins.
> Join create OOM with java heap space on phoenix client
> ------------------------------------------------------
>
> Key: PHOENIX-2001
> URL: https://issues.apache.org/jira/browse/PHOENIX-2001
> Project: Phoenix
> Issue Type: Bug
> Affects Versions: 4.3.1
> Reporter: Krunal
>
> Hi
> I have 2 issues with phoenix client:
> 1. Heap memory is not cleanup after each query is finished. So, it keeps increasing every time when we submit new query.
> 2. I am try to do a normal join operation on two tables but getting exception. Below is the details:
> These are some sample queries I tried:
> 1. select p1.host, count(1) from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2 where p1.host = p2.host group by p1.host;
> 2. select p1.host from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2 where p1.host = p2.host group by p1.host;
> 3. select count(1) from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2 where p1.host = p2.host group by p1.host;
> Here is explain plan:
> explain select count(1) from PERFORMANCE_5000000 p1, PERFORMANCE_25000000 p2 where p1.host = p2.host group by p1.host;
> +------------------------------------------+
> | PLAN |
> +------------------------------------------+
> | CLIENT 9-CHUNK PARALLEL 1-WAY FULL SCAN OVER PERFORMANCE_5000000 |
> | SERVER FILTER BY FIRST KEY ONLY |
> | SERVER AGGREGATE INTO ORDERED DISTINCT ROWS BY [HOST] |
> | CLIENT MERGE SORT |
> | PARALLEL INNER-JOIN TABLE 0 (SKIP MERGE) |
> | CLIENT 18-CHUNK PARALLEL 1-WAY FULL SCAN OVER PERFORMANCE_25000000 |
> | SERVER FILTER BY FIRST KEY ONLY |
> | DYNAMIC SERVER FILTER BY HOST IN (P2.HOST) |
> +------------------------------------------+
> 8 rows selected (0.127 seconds)
> Phoenix client heap size is 16GB. ( noticed that above queries are dumping data in local heap, I see millions of instances for org.apache.phoenix.expression.literalexpression)
> phoenix version: 4.3.1
> hbase version: 0.98.1
> and my exceptions are:
> java.sql.SQLException: Encountered exception in sub plan [0] execution.
> at org.apache.phoenix.execute.HashJoinPlan.iterator(HashJoinPlan.java:156)
> at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:235)
> at org.apache.phoenix.jdbc.PhoenixStatement$1.call(PhoenixStatement.java:226)
> at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
> at org.apache.phoenix.jdbc.PhoenixStatement.executeQuery(PhoenixStatement.java:225)
> at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1066)
> at sqlline.Commands.execute(Commands.java:822)
> at sqlline.Commands.sql(Commands.java:732)
> at sqlline.SqlLine.dispatch(SqlLine.java:808)
> at sqlline.SqlLine.begin(SqlLine.java:681)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> Caused by: java.sql.SQLException: java.util.concurrent.ExecutionException: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
> at org.apache.phoenix.cache.ServerCacheClient.addServerCache(ServerCacheClient.java:247)
> at org.apache.phoenix.join.HashCacheClient.addHashCache(HashCacheClient.java:83)
> at org.apache.phoenix.execute.HashJoinPlan$HashSubPlan.execute(HashJoinPlan.java:338)
> at org.apache.phoenix.execute.HashJoinPlan$1.call(HashJoinPlan.java:135)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:206)
> at org.apache.phoenix.cache.ServerCacheClient.addServerCache(ServerCacheClient.java:239)
> ... 7 more
> Caused by: java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
> at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:212)
> at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:182)
> ... 4 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> May 20, 2015 4:58:01 PM ServerCommunicatorAdmin reqIncoming
> WARNING: The server has decided to close this client connection.
> 15/05/20 16:56:43 WARN client.HTable: Error calling coprocessor service org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService for row CSGoogle\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620)
> at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577)
> at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:188)
> at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:182)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at com.google.protobuf.ByteString$CodedBuilder.<init>(ByteString.java:907)
> at com.google.protobuf.ByteString$CodedBuilder.<init>(ByteString.java:902)
> at com.google.protobuf.ByteString.newCodedBuilder(ByteString.java:898)
> at com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49)
> at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:81)
> at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:57)
> at org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService$Stub.addServerCache(ServerCachingProtos.java:3270)
> at org.apache.phoenix.cache.ServerCacheClient$1$1.call(ServerCacheClient.java:204)
> at org.apache.phoenix.cache.ServerCacheClient$1$1.call(ServerCacheClient.java:189)
> at org.apache.hadoop.hbase.client.HTable$17.call(HTable.java:1608)
> ... 4 more
> 0: jdbc:phoenix:prbhadoop004iad.io.askjeeves.> 15/05/20 16:56:43 WARN client.HTable: Error calling coprocessor service org.apache.phoenix.coprocessor.generated.ServerCachingProtos$ServerCachingService for row EUGoogle\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1620)
> at org.apache.hadoop.hbase.client.HTable.coprocessorService(HTable.java:1577)
> at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:188)
> at org.apache.phoenix.cache.ServerCacheClient$1.call(ServerCacheClient.java:182)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> 15/05/20 16:59:37 WARN client.HConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it
> org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:403)
> at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:321)
> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
> Error: (state=,code=0)
> java.sql.SQLFeatureNotSupportedException
> at org.apache.phoenix.jdbc.PhoenixStatement.cancel(PhoenixStatement.java:958)
> at sqlline.DispatchCallback.forceKillSqlQuery(DispatchCallback.java:83)
> at sqlline.SqlLine.begin(SqlLine.java:695)
> at sqlline.SqlLine.start(SqlLine.java:398)
> at sqlline.SqlLine.main(SqlLine.java:292)
> Can someone please help?
> Thanks!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)