[jira] [Created] (RANGER-4741) Hive plugin optimization to avoid excessive metastore API calls

["Madhan Neethiraj (Jira)" <ji...@apache.org>]

Madhan Neethiraj created RANGER-4741:
----------------------------------------

             Summary: Hive plugin optimization to avoid excessive metastore API calls
                 Key: RANGER-4741
                 URL: https://issues.apache.org/jira/browse/RANGER-4741
             Project: Ranger
          Issue Type: Improvement
          Components: plugins
            Reporter: Madhan Neethiraj
            Assignee: Madhan Neethiraj


Authorizing access to tables with large number of columns can take a long time, as shown below. Time taken to for a table with 400 columns takes about 100 seconds.
{noformat}
0: jdbc:hive2://localhost:10000> SELECT * FROM large_tbl_4000;
...
No rows selected (98.674 seconds)


0: jdbc:hive2://localhost:10000> SELECT * FROM large_tbl_1000;
...
No rows selected (10.4 seconds)
{noformat}
 

For each column referenced in the query, Ranger Hive authorizer calls metastore API to obtain owner of the table. Optimizing to call the metastore API once per table can significantly reduce the time taken to authorize queries.

Here is the time taken to query the same tables with the Ranger Hive authorizer optimized to call metastore API only once per table referenced in the query:
{noformat}
0: jdbc:hive2://localhost:10000> SELECT * FROM large_tbl_4000;
...
No rows selected (1.328 seconds)


0: jdbc:hive2://localhost:10000> SELECT * FROM large_tbl_1000;
...
No rows selected (0.194 seconds)
{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)