You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@zeppelin.apache.org by zj...@apache.org on 2020/07/12 10:13:03 UTC
[zeppelin] branch master updated: [ZEPPELIN-4941]. added support
for personalized configuration of JDBCInterpreter database connection pool
This is an automated email from the ASF dual-hosted git repository.
zjffdu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/zeppelin.git
The following commit(s) were added to refs/heads/master by this push:
new 706a957 [ZEPPELIN-4941]. added support for personalized configuration of JDBCInterpreter database connection pool
706a957 is described below
commit 706a9578f85e5dd67dad0743bc4d50b1a842568c
Author: xiejiajun <xi...@163.com>
AuthorDate: Wed Jul 8 19:20:22 2020 +0800
[ZEPPELIN-4941]. added support for personalized configuration of JDBCInterpreter database connection pool
### What is this PR for?
- Added support for personalized configuration of the JDBCInterpreter database connection pool.
- This feature can be used to solve the runtime error caused by HiveServer2 Session timeout.
- This connection pool configuration Compatible with [the official configuration of dbcp2 ](http://commons.apache.org/proper/commons-dbcp/configuration.html)
### What type of PR is it?
[Feature]
### Todos
* [ ] - Task
### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-4941
### How should this be tested?
* manually tested
* Supported configuration items
- validationQuery
- testOnBorrow
- testOnCreate
- testOnReturn
- testWhileIdle
- timeBetweenEvictionRunsMillis
- maxWaitMillis
- maxIdle
- minIdle
- maxTotal
### Screenshots (if appropriate)
### Questions:
* Does the licenses files need update? NO
* Is there breaking changes for older versions? NO
* Does this needs documentation? NO
Author: xiejiajun <xi...@163.com>
Author: JakeXie <xi...@163.com>
Author: xie-jia-jun <xi...@163.com>
Closes #3844 from xiejiajun/jdbc-pool-config and squashes the following commits:
3bb622b61 [xiejiajun] use html table display datasource pool configuration in the jdbc.md document
88b1ce0c9 [xiejiajun] jdbc interpreter datasource connection pool document
1f2eb555c [xiejiajun] Compatible with the official configuration of dbcp2 connection pool
5c7f66637 [xiejiajun] JDBCInterprete supports personalized database connection pool configuration to cope with scenarios such as HiveSession timeout
3d5771944 [xiejiajun] Merge branch 'branch-0.9' of https://github.com/apache/zeppelin into apache-0.9
0e5f65a53 [xiejiajun] Merge branch 'branch-0.9' of https://github.com/apache/zeppelin into apache-0.9
076668acb [JakeXie] Merge pull request #7 from apache/branch-0.9
00e14dd8f [JakeXie] Merge pull request #5 from apache/branch-0.9
9bb7341ff [xiejiajun] Merge remote-tracking branch 'origin/branch-0.9' into branch-0.9
9bc56056b [xiejiajun] bug fix: when we removing a paragraph , an interpreter process refused to connect due to it abnormal stop , which eventually caused the remove paragraph operation to fail to complete.
97d271487 [xie-jia-jun] Merge pull request #2 from apache/branch-0.9
9b3c744a0 [xiejiajun] added timeout for getting Thrift client to avoid situations where the interpreter may not be restarted when the interpreter process exits unexpectedly
---
docs/interpreter/jdbc.md | 64 ++++++++++++++++++
.../org/apache/zeppelin/jdbc/JDBCInterpreter.java | 27 +++++++-
.../org/apache/zeppelin/util/PropertiesUtil.java | 76 ++++++++++++++++++++++
3 files changed, 166 insertions(+), 1 deletion(-)
diff --git a/docs/interpreter/jdbc.md b/docs/interpreter/jdbc.md
index 5c6c57b..d556160 100644
--- a/docs/interpreter/jdbc.md
+++ b/docs/interpreter/jdbc.md
@@ -156,6 +156,70 @@ The last step is **Dependency Setting**. Since Zeppelin only includes `PostgreSQ
That's it. You can find more JDBC connection setting examples([Mysql](#mysql), [MariaDB](#mariadb), [Redshift](#redshift), [Apache Hive](#apache-hive), [Apache Phoenix](#apache-phoenix), and [Apache Tajo](#apache-tajo)) in [this section](#examples).
+## JDBC Interpreter Datasource Pool Configuration
+The Jdbc interpreter uses the connection pool technology, and supports users to do some personal configuration of the connection pool. For example, we can configure `default.validationQuery='select 1'` and `default.testOnBorrow=true` in the Interpreter configuration to avoid the "Invalid SessionHandle" runtime error caused by Session timeout when connecting to HiveServer2 through JDBC interpreter.
+
+The Jdbc Interpreter supports the following database connection pool configurations:
+
+<table class="table-configuration">
+ <tr>
+ <th>Property Name</th>
+ <th>Default</th>
+ <th>Description</th>
+ </tr>
+ <tr>
+ <td>testOnBorrow</td>
+ <td>false</td>
+ <td>The indication of whether objects will be validated before being borrowed from the pool. If the object fails to validate, it will be dropped from the pool, and we will attempt to borrow another.</td>
+ </tr>
+ <tr>
+ <td>testOnCreate</td>
+ <td>false</td>
+ <td>The indication of whether objects will be validated after creation. If the object fails to validate, the borrow attempt that triggered the object creation will fail.</td>
+ </tr>
+ <tr>
+ <td>testOnReturn</td>
+ <td>false</td>
+ <td>The indication of whether objects will be validated before being returned to the pool.</td>
+ </tr>
+ <tr>
+ <td>testWhileIdle</td>
+ <td>false</td>
+ <td>The indication of whether objects will be validated by the idle object evictor (if any). If an object fails to validate, it will be dropped from the pool.</td>
+ </tr>
+ <tr>
+ <td>timeBetweenEvictionRunsMillis</td>
+ <td>-1L</td>
+ <td>The number of milliseconds to sleep between runs of the idle object evictor thread. When non-positive, no idle object evictor thread will be run.</td>
+ </tr>
+ <tr>
+ <td>maxWaitMillis</td>
+ <td>-1L</td>
+ <td>The maximum number of milliseconds that the pool will wait (when there are no available connections) for a connection to be returned before throwing an exception, or -1 to wait indefinitely.</td>
+ </tr>
+ <tr>
+ <td>maxIdle</td>
+ <td>8</td>
+ <td>The maximum number of connections that can remain idle in the pool, without extra ones being released, or negative for no limit.</td>
+ </tr>
+ <tr>
+ <td>minIdle</td>
+ <td>0</td>
+ <td>The minimum number of connections that can remain idle in the pool, without extra ones being created, or zero to create none.</td>
+ </tr>
+ <tr>
+ <td>maxTotal</td>
+ <td>-1</td>
+ <td>The maximum number of active connections that can be allocated from this pool at the same time, or negative for no limit.</td>
+ </tr>
+ <tr>
+ <td>validationQuery</td>
+ <td>show database</td>
+ <td>The SQL query that will be used to validate connections from this pool before returning them to the caller. If specified, this query MUST be an SQL SELECT statement that returns at least one row. If not specified, connections will be validation by calling the isValid() method.</td>
+ </tr>
+</table>
+
+
## More properties
There are more JDBC interpreter properties you can specify like below.
diff --git a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
index 5dcf4d0..a0b2789 100644
--- a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
+++ b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
@@ -36,6 +36,7 @@ import org.apache.zeppelin.interpreter.ZeppelinContext;
import org.apache.zeppelin.interpreter.util.SqlSplitter;
import org.apache.zeppelin.jdbc.hive.HiveUtils;
import org.apache.zeppelin.tabledata.TableDataUtils;
+import org.apache.zeppelin.util.PropertiesUtil;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -419,6 +420,28 @@ public class JDBCInterpreter extends KerberosInterpreter {
}
}
+ private void configConnectionPool(GenericObjectPool connectionPool, Properties properties) {
+ boolean testOnBorrow = "true".equalsIgnoreCase(properties.getProperty("testOnBorrow"));
+ boolean testOnCreate = "true".equalsIgnoreCase(properties.getProperty("testOnCreate"));
+ boolean testOnReturn = "true".equalsIgnoreCase(properties.getProperty("testOnReturn"));
+ boolean testWhileIdle = "true".equalsIgnoreCase(properties.getProperty("testWhileIdle"));
+ long timeBetweenEvictionRunsMillis = PropertiesUtil.getLong(properties, "timeBetweenEvictionRunsMillis", -1L);
+ long maxWaitMillis = PropertiesUtil.getLong(properties, "maxWaitMillis", -1L);
+ int maxIdle = PropertiesUtil.getInt(properties, "maxIdle", 8);
+ int minIdle = PropertiesUtil.getInt(properties, "minIdle", 0);
+ int maxTotal = PropertiesUtil.getInt(properties, "maxTotal", -1);
+
+ connectionPool.setTestOnBorrow(testOnBorrow);
+ connectionPool.setTestOnCreate(testOnCreate);
+ connectionPool.setTestOnReturn(testOnReturn);
+ connectionPool.setTestWhileIdle(testWhileIdle);
+ connectionPool.setTimeBetweenEvictionRunsMillis(timeBetweenEvictionRunsMillis);
+ connectionPool.setMaxIdle(maxIdle);
+ connectionPool.setMinIdle(minIdle);
+ connectionPool.setMaxTotal(maxTotal);
+ connectionPool.setMaxWaitMillis(maxWaitMillis);
+ }
+
private void createConnectionPool(String url, String user, String dbPrefix,
Properties properties) throws SQLException, ClassNotFoundException {
@@ -441,8 +464,10 @@ public class JDBCInterpreter extends KerberosInterpreter {
final String maxConnectionLifetime =
StringUtils.defaultIfEmpty(getProperty("zeppelin.jdbc.maxConnLifetime"), "-1");
poolableConnectionFactory.setMaxConnLifetimeMillis(Long.parseLong(maxConnectionLifetime));
- poolableConnectionFactory.setValidationQuery("show databases");
+ poolableConnectionFactory.setValidationQuery(
+ PropertiesUtil.getString(properties, "validationQuery", "show databases"));
ObjectPool connectionPool = new GenericObjectPool(poolableConnectionFactory);
+ this.configConnectionPool((GenericObjectPool)connectionPool, properties);
poolableConnectionFactory.setPool(connectionPool);
Class.forName(driverClass);
diff --git a/zeppelin-interpreter/src/main/java/org/apache/zeppelin/util/PropertiesUtil.java b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/util/PropertiesUtil.java
new file mode 100644
index 0000000..0e0e505
--- /dev/null
+++ b/zeppelin-interpreter/src/main/java/org/apache/zeppelin/util/PropertiesUtil.java
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.zeppelin.util;
+
+import java.util.Properties;
+
+/**
+ * java.util.Properties read utils
+ */
+public class PropertiesUtil {
+
+ /**
+ * read the string property
+ * @param properties the properties object
+ * @param key the property key
+ * @param defaultValue the default value
+ * @return the string result
+ */
+ public static String getString(Properties properties, String key, String defaultValue) {
+ return properties.getProperty(key) == null ? defaultValue : properties.getProperty(key);
+ }
+
+
+ /**
+ * read the long property
+ * @param properties the properties object
+ * @param key the property key
+ * @param defaultValue the default value
+ * @return the long result
+ */
+ public static long getLong(Properties properties, String key, long defaultValue) {
+ String valueString = properties.getProperty(key);
+ if (valueString == null){
+ return defaultValue;
+ }
+ try {
+ return Long.parseLong(valueString);
+ }catch (Exception e){
+ return defaultValue;
+ }
+ }
+
+ /**
+ * read the long property
+ * @param properties the properties object
+ * @param key the property key
+ * @param defaultValue the default value
+ * @return the int result
+ */
+ public static int getInt(Properties properties, String key, int defaultValue) {
+ String valueString = properties.getProperty(key);
+ if (valueString == null){
+ return defaultValue;
+ }
+ try {
+ return Integer.parseInt(valueString);
+ }catch (Exception e){
+ return defaultValue;
+ }
+ }
+}