You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Qinghui Xu (Jira)" <ji...@apache.org> on 2019/11/12 21:30:00 UTC
[jira] [Commented] (HIVE-22306) Use nonblocking thrift server for
metastore
[ https://issues.apache.org/jira/browse/HIVE-22306?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972799#comment-16972799 ]
Qinghui Xu commented on HIVE-22306:
-----------------------------------
Some update on the progress:
I pushed a PR for THRIFT-4889, and we patched our hive metastore to use this PR.
We are now testing this nonblocking hive metastore in our preprod. It's quite stable since several days, and our monitoring shows that it has smaller memory footprint and same cpu usage comparing to its blocking variant under a moderate load.
We are going to run some load tests and to compare the behaviors.
I would appreciate the help from the community to review my [thrift PR|[https://github.com/apache/thrift/pull/1892]]
> Use nonblocking thrift server for metastore
> -------------------------------------------
>
> Key: HIVE-22306
> URL: https://issues.apache.org/jira/browse/HIVE-22306
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
> Reporter: Qinghui Xu
> Priority: Major
>
> Currently hive metastore's threads are blocking for network io (it's using `TThreadPoolServer` behind the scene), which means with increasing use cases (in our tech stack there are different services relying on it, hiveserver2, spark, presto, and more, all with a significant number of users), to handle all connections it needs either a big thread pool or many instances with smaller thread pools. And often, those metastores will see their thread pool saturated, while the cpu usage is still quite low, just because most connections stay idle and only run a query from time to time. This is thus a great misuse of the computation resources.
> Thus I propose to use a non blocking threading model, and run computation asynchronously.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)