You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ya...@apache.org on 2021/10/18 02:21:27 UTC
[incubator-kyuubi] branch master updated: [KYUUBI #1209]
kyuubi.engine.share.level.subdomain can be more tolerant
This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 3e199cc [KYUUBI #1209] kyuubi.engine.share.level.subdomain can be more tolerant
3e199cc is described below
commit 3e199ccdce6ae5dae7f23e59efdab9eda4904600
Author: guoqing.yang <gu...@advance.ai>
AuthorDate: Mon Oct 18 10:21:12 2021 +0800
[KYUUBI #1209] kyuubi.engine.share.level.subdomain can be more tolerant
feature: kyuubi.engine.share.level.subdomain can be more tolerant
now end-user can configure kyuubi.engine.share.level.subdomain by zookeeper valid path
I refer to the zookeeper doc http://zookeeper.apache.org/doc/r3.7.0/zookeeperProgrammers.html#:~:text=ZooKeeper%20Basic%20Operations.-,The%20ZooKeeper%20Data%20Model,-ZooKeeper%20has%20a
and source code:
https://github.com/apache/zookeeper/blob/master/zookeeper-server/src/main/java/org/apache/zookeeper/common/PathUtils.java#:~:text=public%20static%20void-,validatePath,-(String%20path)%20throws
Closes #1232 from TyrealBM/feature/valid-subdomain-path.
Closes #1209
8d00c260 [guoqing.yang] [KYUUBI #1209] kyuubi.engine.share.level.subdomain can be more tolerant
Authored-by: guoqing.yang <gu...@advance.ai>
Signed-off-by: Kent Yao <ya...@apache.org>
---
docs/deployment/settings.md | 2 +-
.../org/apache/kyuubi/config/KyuubiConf.scala | 12 ++++++++---
.../org/apache/kyuubi/config/KyuubiConfSuite.scala | 23 ++++++++++++++++++++++
.../kyuubi/engine/spark/SparkSqlEngineSuite.scala | 2 +-
4 files changed, 34 insertions(+), 5 deletions(-)
diff --git a/docs/deployment/settings.md b/docs/deployment/settings.md
index d9c5fa7..aecf08a 100644
--- a/docs/deployment/settings.md
+++ b/docs/deployment/settings.md
@@ -186,7 +186,7 @@ kyuubi\.engine\.pool<br>\.size\.threshold|<div style='width: 65pt;word-wrap: bre
kyuubi\.engine\.session<br>\.initialize\.sql|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>SHOW DATABASES</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>SemiColon-separated list of SQL statements to be initialized in the newly created engine session before queries. This configuration can not be used in JDBC url due to the limitation of Beeline/JDBC driver.</div>|<div style='width: 30pt'>seq</div>|<div style='width: 20pt'>1.3.0</div>
kyuubi\.engine\.share<br>\.level|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>USER</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Engines will be shared in different levels, available configs are: <ul> <li>CONNECTION: engine will not be shared but only used by the current client connection</li> <li>USER: engine will be shared by all sessions created by a unique username, see also kyuubi.engine.share.level.subdomain</li> <li>GROUP: engine w [...]
kyuubi\.engine\.share<br>\.level\.sub\.domain|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>(deprecated) - Using kyuubi.engine.share.level.subdomain instead</div>|<div style='width: 30pt'>string</div>|<div style='width: 20pt'>1.2.0</div>
-kyuubi\.engine\.share<br>\.level\.subdomain|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Allow end-users to create a subdomain for the share level of an engine. A subdomain is a case-insensitive string values in `^[a-zA-Z_-]{1,14}$` form. For example, for `USER` share level, an end-user can share a certain engine within a subdomain, not for all of its clients. End-users are f [...]
+kyuubi\.engine\.share<br>\.level\.subdomain|<div style='width: 65pt;word-wrap: break-word;white-space: normal'><undefined></div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>Allow end-users to create a subdomain for the share level of an engine. A subdomain is a case-insensitive string values that must be a valid zookeeper sub path. For example, for `USER` share level, an end-user can share a certain engine within a subdomain, not for all of its clients. End- [...]
kyuubi\.engine\.single<br>\.spark\.session|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>false</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>When set to true, this engine is running in a single session mode. All the JDBC/ODBC connections share the temporary views, function registries, SQL configuration and the current database.</div>|<div style='width: 30pt'>boolean</div>|<div style='width: 20pt'>1.3.0</div>
kyuubi\.engine\.ui<br>\.retainedSessions|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>200</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>The number of SQL client sessions kept in the Kyuubi Query Engine web UI.</div>|<div style='width: 30pt'>int</div>|<div style='width: 20pt'>1.4.0</div>
kyuubi\.engine\.ui\.stop<br>\.enabled|<div style='width: 65pt;word-wrap: break-word;white-space: normal'>true</div>|<div style='width: 170pt;word-wrap: break-word;white-space: normal'>When true, allows Kyuubi engine to be killed from the Spark Web UI.</div>|<div style='width: 30pt'>boolean</div>|<div style='width: 20pt'>1.3.0</div>
diff --git a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
index 3447a18..ec4d552 100644
--- a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
+++ b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
@@ -714,6 +714,11 @@ object KyuubiConf {
private val validEngineSubDomain: Pattern = "^[a-zA-Z_-]{1,14}$".r.pattern
+ // [ZooKeeper Data Model]
+ // (http://zookeeper.apache.org/doc/r3.7.0/zookeeperProgrammers.html#ch_zkDataModel)
+ private val validEngineSubdomain: Pattern = ("(?!^[\\u002e]{1,2}$)" +
+ "(^[\\u0020-\\u002e\\u0030-\\u007e\\u00a0-\\ud7ff\\uf900-\\uffef]{1,}$)").r.pattern
+
@deprecated(s"using kyuubi.engine.share.level.subdomain instead", "1.4.0")
val ENGINE_SHARE_LEVEL_SUB_DOMAIN: OptionalConfigEntry[String] =
buildConf("engine.share.level.sub.domain")
@@ -721,14 +726,15 @@ object KyuubiConf {
.version("1.2.0")
.stringConf
.transform(_.toLowerCase(Locale.ROOT))
- .checkValue(validEngineSubDomain.matcher(_).matches(),
- "must be [1, 14] length alphabet string, e.g. 'abc', 'apache'")
+ .checkValue(validEngineSubdomain.matcher(_).matches(),
+ "must be valid zookeeper sub path."
+ )
.createOptional
val ENGINE_SHARE_LEVEL_SUBDOMAIN: ConfigEntry[Option[String]] =
buildConf("engine.share.level.subdomain")
.doc("Allow end-users to create a subdomain for the share level of an engine. A" +
- " subdomain is a case-insensitive string values in `^[a-zA-Z_-]{1,14}$` form." +
+ " subdomain is a case-insensitive string values that must be a valid zookeeper sub path." +
" For example, for `USER` share level, an end-user can share a certain engine within" +
" a subdomain, not for all of its clients. End-users are free to create multiple" +
" engines in the `USER` share level")
diff --git a/kyuubi-common/src/test/scala/org/apache/kyuubi/config/KyuubiConfSuite.scala b/kyuubi-common/src/test/scala/org/apache/kyuubi/config/KyuubiConfSuite.scala
index 944316d..fb1bd16 100644
--- a/kyuubi-common/src/test/scala/org/apache/kyuubi/config/KyuubiConfSuite.scala
+++ b/kyuubi-common/src/test/scala/org/apache/kyuubi/config/KyuubiConfSuite.scala
@@ -150,4 +150,27 @@ class KyuubiConfSuite extends KyuubiFunSuite {
val e1 = intercept[IllegalArgumentException](kyuubiConf.get(OPERATION_QUERY_TIMEOUT))
assert(e1.getMessage.contains("must >= 1s if set"))
}
+
+ test("kyuubi conf engine.share.level.subdomain valid path test") {
+ val kyuubiConf = KyuubiConf()
+ kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "")
+ assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
+ kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, ".")
+ assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
+ kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "..")
+ assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
+ kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "/")
+ assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
+ kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "/tmp/")
+ assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
+ kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "tmp/")
+ assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
+ kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "/tmp")
+ assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
+ kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "abc/efg")
+ assertThrows[IllegalArgumentException](kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN))
+ val path = "kyuubi!@#$%^&*()_+-=[]{};:,.<>?"
+ kyuubiConf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, path)
+ assert(kyuubiConf.get(ENGINE_SHARE_LEVEL_SUBDOMAIN).get == path)
+ }
}
diff --git a/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkSqlEngineSuite.scala b/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkSqlEngineSuite.scala
index 7cd1004..a695f63 100644
--- a/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkSqlEngineSuite.scala
+++ b/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/spark/SparkSqlEngineSuite.scala
@@ -72,7 +72,7 @@ class SparkSqlEngineSuite extends WithKyuubiServer with JDBCTestUtils {
test("Fail connections on invalid sub domains") {
- Seq("1", ",", "", "a" * 15, "abc.xyz").foreach { invalid =>
+ Seq("/", "/tmp", "", "abc/efg", ".", "..").foreach { invalid =>
val sparkHiveConfigs = Map(
ENGINE_SHARE_LEVEL.key -> "USER",
ENGINE_SHARE_LEVEL_SUBDOMAIN.key -> invalid)