[jira] [Commented] (HIVE-10081) LLAP: Make the low-level IO threadpool configurable

["Lefty Leverenz (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-10081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14389929#comment-14389929 ] 

Lefty Leverenz commented on HIVE-10081:
---------------------------------------

This adds *hive.llap.io.threadpool.size* to HiveConf.java, so I'm linking it to HIVE-9850 (documentation for llap).

> LLAP: Make the low-level IO threadpool configurable
> ---------------------------------------------------
>
>                 Key: HIVE-10081
>                 URL: https://issues.apache.org/jira/browse/HIVE-10081
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Gopal V
>            Assignee: Prasanth Jayachandran
>         Attachments: HIVE-10081.1.patch
>
>
> The LLAP low level reader thread-pool is hard-limited to 10-threads, which is not sufficient to max out the network bandwidth on a 10GigE network.
> These threads are often seen in IOWAIT, since they are reading remote data.
> A dumb fix for my 12-core instance was to use a higher thread-pool count for the IO read-ahead.
> {code}
> diff --git a/llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java b/llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java
> index 3f9ddfb..b7cd177 100644
> --- a/llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java
> +++ b/llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java
> @@ -105,7 +105,7 @@ private LlapIoImpl(Configuration conf) throws IOException {
>        cachePolicy.setEvictionListener(metadataCache);
>      }
>      // Arbitrary thread pool. Listening is used for unhandled errors for now (TODO: remove?)
> -    executor = MoreExecutors.listeningDecorator(Executors.newFixedThreadPool(10));
> +    executor = MoreExecutors.listeningDecorator(Executors.newFixedThreadPool(24));
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)