You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2020/10/26 13:19:07 UTC

[GitHub] [hive] jshmchenxi opened a new pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

jshmchenxi opened a new pull request #1611:
URL: https://github.com/apache/hive/pull/1611


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
     2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
     3. Ensure you have added or run the appropriate tests for your PR: 
     4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX:  Your PR title ...'.
     5. Be sure to keep the PR description updated to reflect all changes.
     6. Please write your PR title to summarize what this PR proposes.
     7. If possible, provide a concise example to reproduce the issue for a faster review.
   
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   Improvement [HIVE-12371](https://issues.apache.org/jira/browse/HIVE-12371): Adding a timeout connection parameter for JDBC
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   We run into the problem of [HIVE-22196](https://issues.apache.org/jira/browse/HIVE-22196): Socket timeouts happen when other drivers set DriverManager.loginTimeout.
   It is a good idea to add timeout parameter to hiveserver2 jdbc url. So that we do not use the global DriverManager.loginTimeout which may be unexpected set by other components in the program.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   No
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   Tested by local install jar


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] dengzhhu653 commented on pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on pull request #1611:
URL: https://github.com/apache/hive/pull/1611#issuecomment-746171619


   +1, looks good to me!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] jshmchenxi commented on a change in pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
jshmchenxi commented on a change in pull request #1611:
URL: https://github.com/apache/hive/pull/1611#discussion_r550411547



##########
File path: jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java
##########
@@ -1002,11 +1002,19 @@ private String getSessionValue(String varName, String varDefault) {
     return varValue;
   }
 
-  // copy loginTimeout from driver manager. Thrift timeout needs to be in millis
+  // use socketTimeout from jdbc connection url. Thrift timeout needs to be in millis
   private void setupLoginTimeout() {
-    long timeOut = TimeUnit.SECONDS.toMillis(DriverManager.getLoginTimeout());
+    String socketTimeoutStr = sessConfMap.getOrDefault(JdbcConnectionParams.SOCKET_TIMEOUT, "0");
+    long timeOut = 0;

Review comment:
       Sorry for the late reply. If users depends on DriverManager.loginTimeout to control HS2 jdbc timeout, they will have to refactor their code after applying this. However, we haven't yet met a situation that we need this timeout to return quickly when HS2 hangs or something. Our problem is when submitting a big query which needs minutes to compile, the connection is lost because of this DriverManager.loginTimeout. Maybe we need to set a timeout parameter for HS2 on the server side too.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] prasanthj commented on pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
prasanthj commented on pull request #1611:
URL: https://github.com/apache/hive/pull/1611#issuecomment-780011819


   lgtm, +1


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] dengzhhu653 commented on pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on pull request #1611:
URL: https://github.com/apache/hive/pull/1611#issuecomment-752832803


   Hey @jshmchenxi,  you can reopen the pr or push a commit to trigger a new clean build...


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] dengzhhu653 commented on a change in pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on a change in pull request #1611:
URL: https://github.com/apache/hive/pull/1611#discussion_r544235952



##########
File path: jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java
##########
@@ -1002,11 +1002,19 @@ private String getSessionValue(String varName, String varDefault) {
     return varValue;
   }
 
-  // copy loginTimeout from driver manager. Thrift timeout needs to be in millis
+  // use socketTimeout from jdbc connection url. Thrift timeout needs to be in millis
   private void setupLoginTimeout() {
-    long timeOut = TimeUnit.SECONDS.toMillis(DriverManager.getLoginTimeout());
+    String socketTimeoutStr = sessConfMap.getOrDefault(JdbcConnectionParams.SOCKET_TIMEOUT, "0");
+    long timeOut = 0;

Review comment:
       What if the default value set to TimeUnit.SECONDS.toMillis(DriverManager.getLoginTimeout())?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] dengzhhu653 commented on a change in pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on a change in pull request #1611:
URL: https://github.com/apache/hive/pull/1611#discussion_r544236706



##########
File path: jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java
##########
@@ -1002,11 +1002,19 @@ private String getSessionValue(String varName, String varDefault) {
     return varValue;
   }
 
-  // copy loginTimeout from driver manager. Thrift timeout needs to be in millis
+  // use socketTimeout from jdbc connection url. Thrift timeout needs to be in millis
   private void setupLoginTimeout() {
-    long timeOut = TimeUnit.SECONDS.toMillis(DriverManager.getLoginTimeout());
+    String socketTimeoutStr = sessConfMap.getOrDefault(JdbcConnectionParams.SOCKET_TIMEOUT, "0");
+    long timeOut = 0;
+    try {
+      timeOut = Long.parseLong(socketTimeoutStr);
+    } catch (NumberFormatException e) {
+      LOG.info("Failed to parse socketTimeout of value" + socketTimeoutStr);

Review comment:
       A space may be needed here after `of value`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] jshmchenxi commented on a change in pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
jshmchenxi commented on a change in pull request #1611:
URL: https://github.com/apache/hive/pull/1611#discussion_r546534493



##########
File path: jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java
##########
@@ -1002,11 +1002,19 @@ private String getSessionValue(String varName, String varDefault) {
     return varValue;
   }
 
-  // copy loginTimeout from driver manager. Thrift timeout needs to be in millis
+  // use socketTimeout from jdbc connection url. Thrift timeout needs to be in millis
   private void setupLoginTimeout() {
-    long timeOut = TimeUnit.SECONDS.toMillis(DriverManager.getLoginTimeout());
+    String socketTimeoutStr = sessConfMap.getOrDefault(JdbcConnectionParams.SOCKET_TIMEOUT, "0");
+    long timeOut = 0;

Review comment:
       This issue describes the problem: [HIVE-22196](https://issues.apache.org/jira/browse/HIVE-22196) : Socket timeouts happen when other drivers set DriverManager.loginTimeout.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] jshmchenxi commented on pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
jshmchenxi commented on pull request #1611:
URL: https://github.com/apache/hive/pull/1611#issuecomment-748797758


   @dengzhhu653  Thanks for the review.
   If this patch is merged, hive document needs to update for the new JDBC parameter.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] jshmchenxi commented on pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
jshmchenxi commented on pull request #1611:
URL: https://github.com/apache/hive/pull/1611#issuecomment-752868142


   @dengzhhu653  Yes, I rebased the code. The errors seems to be unrelated


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] jshmchenxi commented on pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
jshmchenxi commented on pull request #1611:
URL: https://github.com/apache/hive/pull/1611#issuecomment-720243636


   The failed test seemes to be unralated


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] dengzhhu653 commented on a change in pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on a change in pull request #1611:
URL: https://github.com/apache/hive/pull/1611#discussion_r546701496



##########
File path: jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java
##########
@@ -1002,11 +1002,19 @@ private String getSessionValue(String varName, String varDefault) {
     return varValue;
   }
 
-  // copy loginTimeout from driver manager. Thrift timeout needs to be in millis
+  // use socketTimeout from jdbc connection url. Thrift timeout needs to be in millis
   private void setupLoginTimeout() {
-    long timeOut = TimeUnit.SECONDS.toMillis(DriverManager.getLoginTimeout());
+    String socketTimeoutStr = sessConfMap.getOrDefault(JdbcConnectionParams.SOCKET_TIMEOUT, "0");
+    long timeOut = 0;

Review comment:
       Set the default value to 0 may cause problems for those elder clients using DriverManager.loginTimeout to prevent the client from hang or return quickly if there are some problems in HS2, users have to refactor the uri with the socket timeout parameter after applying this.  After all, there are some preconditions should be met on the HIVE-22196: 1, There are different drivers that created in the same jvm; 2,  One of the driver sets the loginTimeout that is inappropriate for others.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] jshmchenxi commented on a change in pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
jshmchenxi commented on a change in pull request #1611:
URL: https://github.com/apache/hive/pull/1611#discussion_r546536308



##########
File path: jdbc/src/java/org/apache/hive/jdbc/HiveConnection.java
##########
@@ -1002,11 +1002,19 @@ private String getSessionValue(String varName, String varDefault) {
     return varValue;
   }
 
-  // copy loginTimeout from driver manager. Thrift timeout needs to be in millis
+  // use socketTimeout from jdbc connection url. Thrift timeout needs to be in millis
   private void setupLoginTimeout() {
-    long timeOut = TimeUnit.SECONDS.toMillis(DriverManager.getLoginTimeout());
+    String socketTimeoutStr = sessConfMap.getOrDefault(JdbcConnectionParams.SOCKET_TIMEOUT, "0");
+    long timeOut = 0;
+    try {
+      timeOut = Long.parseLong(socketTimeoutStr);
+    } catch (NumberFormatException e) {
+      LOG.info("Failed to parse socketTimeout of value" + socketTimeoutStr);

Review comment:
       fixed




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


[GitHub] [hive] prasanthj merged pull request #1611: HIVE-12371 Adding a timeout connection parameter for JDBC

Posted by GitBox <gi...@apache.org>.
prasanthj merged pull request #1611:
URL: https://github.com/apache/hive/pull/1611


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org