You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Siddharth Seth (JIRA)" <ji...@apache.org> on 2017/01/26 08:03:24 UTC

[jira] [Commented] (HIVE-15693) LLAP: cached threadpool in AMReporter creates too many threads leading to OOM

    [ https://issues.apache.org/jira/browse/HIVE-15693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839399#comment-15839399 ] 

Siddharth Seth commented on HIVE-15693:
---------------------------------------

Instead of 2x executors - think this needs to be based on the concurrency. New config parameter to set an upper bound? Lower bound to number of executors?
Killed attempts is more likely to be based on number of AMs communicating, rather than the number of executors in the daemon.

Eventually, I think we need to have a certain number of threads per AM - and also ensure that all threads don't end up blocking because of one bad AM. I'll create a follow up jira for this.

> LLAP: cached threadpool in AMReporter creates too many threads leading to OOM
> -----------------------------------------------------------------------------
>
>                 Key: HIVE-15693
>                 URL: https://issues.apache.org/jira/browse/HIVE-15693
>             Project: Hive
>          Issue Type: Bug
>          Components: llap
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>            Priority: Critical
>         Attachments: HIVE-15693.1.patch
>
>
> branch: master
> {noformat}
> 2017-01-22T19:52:42,470 WARN  [IPC Server handler 3 on 34642 ()] org.apache.hadoop.ipc.Server: IPC Server handler 3 on 34642, call org.apache.hadoop.hive.llap.protocol.LlapProtocolBlockingPB.submitWork ...Call#17257 Retry#0
> java.lang.OutOfMemoryError: unable to create new native thread
>         at java.lang.Thread.start0(Native Method) ~[?:1.8.0_77]
>         at java.lang.Thread.start(Thread.java:714) [?:1.8.0_77]
>         at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:950) ~[?:1.8.0_77]
>         at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1368) ~[?:1.8.0_77]
>         at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:480) ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>         at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:61) ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>         at org.apache.hadoop.hive.llap.daemon.impl.AMReporter.taskKilled(AMReporter.java:231) ~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>         at org.apache.hadoop.hive.llap.daemon.impl.ContainerRunnerImpl$KilledTaskHandlerImpl.taskKilled(ContainerRunnerImpl.java:501) ~[hive-llap-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)