[jira] [Commented] (HIVE-16979) Cache UGI for metastore

["Gopal V (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-16979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109684#comment-16109684 ] 

Gopal V commented on HIVE-16979:
--------------------------------

[~taoli-hwx]: does this fail queries which take > 24hours?

Is there something we can do to mark "liveness" from the query progress loop to make sure the FileSystem.closeAllForUgi() -> deleteOnExit doesn't cleanup any directory currently being written to inside the cluster?

> Cache UGI for metastore
> -----------------------
>
>                 Key: HIVE-16979
>                 URL: https://issues.apache.org/jira/browse/HIVE-16979
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Tao Li
>            Assignee: Tao Li
>         Attachments: HIVE-16979.1.patch, HIVE-16979.2.patch, HIVE-16979.3.patch
>
>
> FileSystem.closeAllForUGI is called per request against metastore to dispose UGI, which involves talking to HDFS name node and is time consuming. So the perf improvement would be caching and reusing the UGI.
> Per FileSystem.closeAllForUG call could take up to 20 ms as E2E latency against HDFS. Usually a Hive query could result in several calls against metastore, so we can save up to 50-100 ms per hive query.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)