You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Paul Rogers (JIRA)" <ji...@apache.org> on 2018/02/11 21:53:00 UTC
[jira] [Commented] (DRILL-6149) Support Query Level Options
[ https://issues.apache.org/jira/browse/DRILL-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16360141#comment-16360141 ]
Paul Rogers commented on DRILL-6149:
------------------------------------
Let's think about this one just a bit. Drill follows SQL semantics. I SQL, there is a global state and a session state. Each query has its own state (represented by a cursor, though Drill does not have this concept.) There is no place in SQL for query-level options.
What may be a bit confusing is that, when executing a fragment, there are query-level options. So doesn't this contradict what was just said above?
No, it doesn't. Query-level options are an implementation detail for the fact that Drill is a distributed system.
The user maintains a session at a single Drillbit: the Foreman. Drill does not have the idea of a user session as a global object. It could. We've discussed the following scenario. Suppose the user logs into Foreman A, issues a bunch of session options, and runs a query. Foreman A dies and the user connection automatically routes to Foreman B. Cool, we have HA. Well, not quite. The user's session on A was lost, the session on B will run queries without the previously-set session options, resulting in wrong results. To make this work, we'd need a global "state store" for user session artifacts. We don't have that, so there user session is local to the Foreman.
Now, consider what happens when we execute a query. Forman A sends fragments to Drillbits B, C, and D. How, though, does a fragment running in B get access to the session options held on A without a global session state?
The answer is that we use a "closure": the query closes around the session options in effect at the time the query was launched. In piratical terms, we copy the session options into the physical plan so that each fragment runs with the set of options in effect when the query started.
Note that this solution also solves the concurrency problem that would result if we did have a global session state and the query referenced that state. I set option x to 10 and start a long-running query. I then set option x to 20. Which value should the query use: the original one or the new one? Our answer is: the original one. That is, we implicitly use an MVCC solution in which queries use values in effect when queries start. Copying the values creates the "frozen" version.
Note further that fragments NEVER reference session options: they only reference the "frozen" options in the query and global options. (This means, by the way, that we do have race conditions for global options -- we don't use MVCC for them. This is probably a bug.)
So, back to the proposal in this ticket:
* We have no SQL syntax for setting a query-only option. Perhaps we could add transactions to identify which ALTER SESSION commands go with a query and which with a session. But, this would be very awkward.
* Session options are not very useful alway because Tableau and other tools don't have a way to set them. We really need something like a persistent user profile that can span sessions.
* So, we do not need to clean up session/query option confusion because there isn't any -- at least not in the code.
Now, if the purpose here is to do code cleanup, then that's fine; perhaps make that clearer in the title and description.
> Support Query Level Options
> ---------------------------
>
> Key: DRILL-6149
> URL: https://issues.apache.org/jira/browse/DRILL-6149
> Project: Apache Drill
> Issue Type: Bug
> Reporter: Timothy Farkas
> Assignee: Venkata Jyothsna Donapati
> Priority: Major
>
> Currently the line between session options and query level options are blurred. Ideally a session level option can be set and persisted for the duration of a user's session. A query level option can be set for a query. Right now there is no explicit query level option. A query level option is a session option whose validator has a non zero ttl. We should change this to have options that belong to the Query Scope as defined in the OptionDefinition. The ttl should also be configurable by the user in their **alter set** command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)