[jira] [Created] (DRILL-5182) Storage plugins have no concept of end-of-query to release resources

["Paul Rogers (JIRA)" <ji...@apache.org>]

Paul Rogers created DRILL-5182:
----------------------------------

             Summary: Storage plugins have no concept of end-of-query to release resources
                 Key: DRILL-5182
                 URL: https://issues.apache.org/jira/browse/DRILL-5182
             Project: Apache Drill
          Issue Type: Improvement
    Affects Versions: 1.8.0
            Reporter: Paul Rogers


At planing time, storage plugins have to concepts of scope: global, or local to a specific operator instance. Other JIRA mentioned the problems for security. Here, let us focus on caching.

Suppose we create a storage plugin to access a remote system. To plan the query, we have to ask if a given "table" exists, if the selected columns are valid, if the user can access the table, the number of minor fragments that can be created and so on.

Suppose the target service is a REST service such as Oracle Sales Cloud or any other business service. We certainly do not want to keep sending REST metadata requests to answer each of the above questions. And, for something like Oracle, the answer to the questions depends on the user. (Oracle provides both table and column-level security.)

Instead, we want to hit the service with one metadata request. ("Get me the information for the 'customer' table.") Then, cache that info. This, then, is the issue: how does a storage plugin know who the user is and when to release the cache?

The storage plugin API should provide a query context along with an events that says, "new use of a query context for this plugin" and "query planning done for this context". This allows the plugin to create a object that represents the query planning session, hold cached information for that scope, and release resources at the completion of the scope.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)