You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2020/02/25 23:05:34 UTC
[GitHub] [drill] dobesv opened a new pull request #1996: DRILL-7603 Allow
default schema to be set for HTTP queries
dobesv opened a new pull request #1996: DRILL-7603 Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996
# [DRILL-7603](https://issues.apache.org/jira/browse/DRILL-7603): Allow default schema to be set for HTTP queries
## Description
This allows REST API requests and Web UI requests to specify a default
schema. Otherwise this is not possible for HTTP clients because the
"USE" command requires a session, which HTTP clients do not have.
## Documentation
### POST /query.json
Submit a query and return results.
**Parameters**
* `queryType`--SQL, PHYSICAL, or LOGICAL are valid types. Use only "SQL". Other types are for internal use only.
* `query`--A SQL query that runs in Drill.
* `autoLimit`--Limits the number of rows returned from the result set. (Drill 1.16+)
* `defaultSchema`--Specify the default schema for the request, as if the `USE` command was run before the query
## Testing
(Please describe how this PR has been tested.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on issue #1996: DRILL-7603 Allow
default schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on issue #1996: DRILL-7603 Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#issuecomment-591340407
@dobesv for the PR that make changes in the Web UI, please also add screenshots for better visibility.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] dobesv commented on a change in pull request #1996:
DRILL-7603: Allow default schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
dobesv commented on a change in pull request #1996: DRILL-7603: Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#discussion_r384639275
##########
File path: exec/java-exec/src/test/java/org/apache/drill/exec/server/rest/TestQueryWrapper.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.server.rest;
+
+import io.netty.channel.DefaultChannelPromise;
+import io.netty.channel.local.LocalAddress;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.server.Drillbit;
+import org.apache.drill.exec.server.options.SystemOptionManager;
+import org.apache.drill.exec.server.rest.auth.DrillUserPrincipal;
+import org.apache.drill.exec.work.WorkManager;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.junit.Test;
+
+import java.io.IOException;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+
+public class TestQueryWrapper extends ClusterTest {
+
+ private static DrillRestServer server;
+ private static WorkManager workManager;
+ private static WebServer webServer;
+
+ private void setupServer() throws IOException {
+ boolean impersonationEnabled = false;
+ setupServer(impersonationEnabled);
+ }
+
+ private void setupServer(boolean impersonationEnabled) {
+ cluster = ClusterFixture.bareBuilder(dirTestWatcher)
+ .clusterSize(1)
+ .configProperty(ExecConstants.ALLOW_LOOPBACK_ADDRESS_BINDING, true)
+ .configProperty(ExecConstants.IMPERSONATION_ENABLED, impersonationEnabled)
+ .build();
+ Drillbit drillbit = cluster.drillbit();
+ workManager = drillbit.getManager();
+ webServer = drillbit.getWebServer();
+ final ServletContextHandler servletContextHandler = new ServletContextHandler(ServletContextHandler.SESSIONS);
+ servletContextHandler.setContextPath("/");
+ server = new DrillRestServer(workManager, servletContextHandler.getServletContext(), drillbit);
+ }
+
+ private QueryWrapper.QueryResult run(String sql) throws Exception {
+ return runWithDefaultSchema(sql, null);
+ }
+
+ private QueryWrapper.QueryResult runWithDefaultSchema(String sql, String defaultSchema) throws Exception {
+ if (cluster == null) {
Review comment:
Some of the tests use different options. Although maybe not in this particular PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#1996: DRILL-7603: Allow default schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #1996: DRILL-7603: Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#discussion_r384981753
##########
File path: exec/java-exec/src/main/resources/rest/query/query.ftl
##########
@@ -83,6 +83,8 @@
Submit
</button>
<input type="checkbox" name="forceLimit" value="limit" <#if model.isAutoLimitEnabled()>checked</#if>> Limit results to <input type="text" id="autoLimit" name="autoLimit" min="0" value="${model.getDefaultRowsAutoLimited()?c}" size="6" pattern="[0-9]*"> rows <span class="glyphicon glyphicon-info-sign" title="Limits the number of records retrieved in the query. Ignored if query has a limit already" style="cursor:pointer"></span>
+ <label> Default Schema <input type="text" size="10" name="defaultSchema" id="defaultSchema"> </label>
+ <span class="glyphicon glyphicon-info-sign" title="Set the default schema used to find table names, and for SHOW FILES and SHOW TABLES" style="cursor:pointer"></span>
Review comment:
Not particularly for that, schema can be set to avoid indicating it in queries.
Plus I am still waiting for screenshots to see how changes will look on Web UI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on issue #1996: DRILL-7603 Allow
default schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on issue #1996: DRILL-7603 Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#issuecomment-591273554
@dobesv the builds are pretty noisy (too much detail about Maven downloads), but it turns out that if I pawed though all the noise, I found your stack trace:
```
[ERROR] org.apache.drill.exec.server.rest.TestQueryWrapper.testSpecifyDefaultSchema Time elapsed: 0.111 s <<< ERROR!
java.lang.NullPointerException
at org.apache.drill.exec.server.rest.TestQueryWrapper.runWithDefaultSchema(TestQueryWrapper.java:85)
at org.apache.drill.exec.server.rest.TestQueryWrapper.testSpecifyDefaultSchema(TestQueryWrapper.java:99)
[ERROR] org.apache.drill.exec.server.rest.TestQueryWrapper.testShowSchemas Time elapsed: 0.049 s <<< ERROR!
java.lang.NullPointerException
at org.apache.drill.exec.server.rest.TestQueryWrapper.runWithDefaultSchema(TestQueryWrapper.java:85)
at org.apache.drill.exec.server.rest.TestQueryWrapper.run(TestQueryWrapper.java:65)
at org.apache.drill.exec.server.rest.TestQueryWrapper.testShowSchemas(TestQueryWrapper.java:90)
202
```
The "Search Logs" function in GitHub seems to not work. I had to download the logs as an archive (cleverly hidden in the "..." button.)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] asfgit closed pull request #1996: DRILL-7603: Allow default
schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #1996: DRILL-7603: Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] dobesv commented on issue #1996: DRILL-7603 Allow default
schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
dobesv commented on issue #1996: DRILL-7603 Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#issuecomment-591255233
@paul-rogers If one test fails, it cancels the others, that's why the tests were cancelled.
As for the test failure I agree it looks related to this change for sure, but I can't reproduce it. Maybe if you pull down the branch and run tests you can see if you can? If I had a stack trace there's a decent chance I could make a really good guess what is stopping it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] dobesv commented on issue #1996: DRILL-7603: Allow default
schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
dobesv commented on issue #1996: DRILL-7603: Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#issuecomment-593055641
Looks like I have fixed the test failures now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] dobesv commented on issue #1996: DRILL-7603: Allow default
schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
dobesv commented on issue #1996: DRILL-7603: Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#issuecomment-591540321
@paul-rogers Thanks for finding that! Unfortunately, I think the stack trace must be incomplete.

I don't see how that line would throw NPE. The NPE must be coming from within the `run` method somewhere.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on issue #1996: DRILL-7603 Allow
default schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on issue #1996: DRILL-7603 Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#issuecomment-591224845
Looks like the Java 13 build failed for real bugs:
```
[ERROR] Errors:
[ERROR] TestQueryWrapper.testShowSchemas:90->run:65->runWithDefaultSchema:85 » NullPointer
[ERROR] TestQueryWrapper.testSpecifyDefaultSchema:99->runWithDefaultSchema:85 » NullPointer
```
Not sure why Java 8 failed:
```
##[error]The operation was canceled.
```
Who would cancel the operation? I've had tests fail due to running too slowly. @vvysotskyi, any suggestions?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] vvysotskyi commented on issue #1996: DRILL-7603 Allow
default schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on issue #1996: DRILL-7603 Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#issuecomment-591305824
+1 regarding autocanceling after the failure of one of the builds.
Also, it may be simpler to look up the logs when selecting "View raw logs".
Regarding this failure, it is also present in JDK 8 build.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on issue #1996: DRILL-7603:
Allow default schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on issue #1996: DRILL-7603: Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#issuecomment-591342777
@dobesv code review summary:
1. please add screenshot;
2. address code review comment;
3. update commit message: must include `:` after Jira number: `DRILL-1234: Jira title`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] paul-rogers commented on a change in pull request #1996:
DRILL-7603 Allow default schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
paul-rogers commented on a change in pull request #1996: DRILL-7603 Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#discussion_r384257662
##########
File path: exec/java-exec/src/main/java/org/apache/drill/exec/server/rest/QueryResources.java
##########
@@ -96,10 +95,12 @@ public QueryResult submitQueryJSON(QueryWrapper query) throws Exception {
@Produces(MediaType.TEXT_HTML)
public Viewable submitQuery(@FormParam("query") String query,
@FormParam("queryType") String queryType,
- @FormParam("autoLimit") String autoLimit) throws Exception {
+ @FormParam("autoLimit") String autoLimit,
+ @FormParam("defaultSchema") String defaultSchema) throws Exception {
Review comment:
Nice improvement! I wonder, would it make sense to accept a list of name/value pairs so we can also pass along session options? We could identify the schema as a REST-specific property in that list.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] arina-ielchiieva commented on a change in pull request
#1996: DRILL-7603 Allow default schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
arina-ielchiieva commented on a change in pull request #1996: DRILL-7603 Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#discussion_r384388046
##########
File path: exec/java-exec/src/test/java/org/apache/drill/exec/server/rest/TestQueryWrapper.java
##########
@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.drill.exec.server.rest;
+
+import io.netty.channel.DefaultChannelPromise;
+import io.netty.channel.local.LocalAddress;
+import org.apache.drill.exec.ExecConstants;
+import org.apache.drill.exec.proto.UserBitShared;
+import org.apache.drill.exec.rpc.user.UserSession;
+import org.apache.drill.exec.server.Drillbit;
+import org.apache.drill.exec.server.options.SystemOptionManager;
+import org.apache.drill.exec.server.rest.auth.DrillUserPrincipal;
+import org.apache.drill.exec.work.WorkManager;
+import org.apache.drill.test.ClusterFixture;
+import org.apache.drill.test.ClusterTest;
+import org.eclipse.jetty.servlet.ServletContextHandler;
+import org.junit.Test;
+
+import java.io.IOException;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+
+public class TestQueryWrapper extends ClusterTest {
+
+ private static DrillRestServer server;
+ private static WorkManager workManager;
+ private static WebServer webServer;
+
+ private void setupServer() throws IOException {
+ boolean impersonationEnabled = false;
+ setupServer(impersonationEnabled);
+ }
+
+ private void setupServer(boolean impersonationEnabled) {
+ cluster = ClusterFixture.bareBuilder(dirTestWatcher)
+ .clusterSize(1)
+ .configProperty(ExecConstants.ALLOW_LOOPBACK_ADDRESS_BINDING, true)
+ .configProperty(ExecConstants.IMPERSONATION_ENABLED, impersonationEnabled)
+ .build();
+ Drillbit drillbit = cluster.drillbit();
+ workManager = drillbit.getManager();
+ webServer = drillbit.getWebServer();
+ final ServletContextHandler servletContextHandler = new ServletContextHandler(ServletContextHandler.SESSIONS);
+ servletContextHandler.setContextPath("/");
+ server = new DrillRestServer(workManager, servletContextHandler.getServletContext(), drillbit);
+ }
+
+ private QueryWrapper.QueryResult run(String sql) throws Exception {
+ return runWithDefaultSchema(sql, null);
+ }
+
+ private QueryWrapper.QueryResult runWithDefaultSchema(String sql, String defaultSchema) throws Exception {
+ if (cluster == null) {
Review comment:
I think you can setup server in @BeforeClass, thus you won't have to do this check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [drill] dobesv commented on issue #1996: DRILL-7603 Allow default
schema to be set for HTTP queries
Posted by GitBox <gi...@apache.org>.
dobesv commented on issue #1996: DRILL-7603 Allow default schema to be set for HTTP queries
URL: https://github.com/apache/drill/pull/1996#issuecomment-591159888
I can't quite see why the tests failed, I guess maven ate the stack trace. Locally the tests pass. Anyone have any hints?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services