[jira] [Commented] (SENTRY-1687) FullUpdateInitializer can be more efficient

["Alexander Kolbasov (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SENTRY-1687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15960317#comment-15960317 ] 

Alexander Kolbasov commented on SENTRY-1687:
--------------------------------------------

It seems that we can do quite a few things to improve this code.

* Instead of using Thrift data structures, emit pairs {objName, objPath} into a non-blocking queue.
* Use one thread (probably the one waiting for result) to poll from this queue and populate the final map.
* Do not parse paths into lists that are then joined back together.

This will avoid most of the synchronization on the {{update}} object that is happening now.

> FullUpdateInitializer can be more efficient
> -------------------------------------------
>
>                 Key: SENTRY-1687
>                 URL: https://issues.apache.org/jira/browse/SENTRY-1687
>             Project: Sentry
>          Issue Type: Sub-task
>          Components: Sentry
>    Affects Versions: sentry-ha-redesign
>            Reporter: Alexander Kolbasov
>            Assignee: Alexander Kolbasov
>            Priority: Minor
>
> The FullUpdateInitializer follows the {{MetastoreCacheInitializer}}. It reads a bunch of information from HMS and uses Thrift structures to pass around, but in the end it just constructs {{Map<String, Set<String>>}}. It uses concurrent fetches from HMS, but synchronizes a lot on common data structures to update them.
> I think that we can refactor all this code to make it faster and consume less memory. The idea is the following:
> Use background threads to collect Thrift results from HMS calls (database, table and partition data). Then we can use a single thread to construct the resulting update and return it without using intermediate Thrift methods.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)