You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2022/12/05 11:30:43 UTC
[incubator-kyuubi] branch master updated: [KYUUBI #3897] Supplying pluggable GroupProvider
This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 4730d11ad [KYUUBI #3897] Supplying pluggable GroupProvider
4730d11ad is described below
commit 4730d11ad7ac210e47e6f5293b28e260279bafa3
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Mon Dec 5 19:30:33 2022 +0800
[KYUUBI #3897] Supplying pluggable GroupProvider
### _Why are the changes needed?_
Kyuubi supports GROUP engine share level, currently, it just simply delegates the group provider to Hadoop UserGroupInformation, which is not flexible enough for users who want to use other group mapping mechanisms, e.g. LDAP, JDBC.
This PR supplies a pluggable plugin interface `GroupProvider` and provides a built-in `HadoopGroupProvider` which has the same behavior w/ the current implementation.
W/ this change, users can easily implement `LDAPGroupProvider`, `JDBCGroupProvider`, `FileGroupProvider`, `CustomGroupProvider`, etc. then the GROUP engine share level will be more powerful and flexible.
The alternative option is to guide users to learn and extend the Hadoop group mapping system[1].
[1] https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/GroupsMapping.html
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #3897 from pan3793/group.
Closes #3897
b100348e [Cheng Pan] nit
3cce6451 [Cheng Pan] Supplying plugable GroupProvider
Authored-by: Cheng Pan <ch...@apache.org>
Signed-off-by: Cheng Pan <ch...@apache.org>
---
docs/deployment/settings.md | 1 +
.../org/apache/kyuubi/plugin/GroupProvider.java | 31 +++++++++++
.../org/apache/kyuubi/config/KyuubiConf.scala | 15 +++++
.../scala/org/apache/kyuubi/engine/EngineRef.scala | 13 +----
.../org/apache/kyuubi/plugin/PluginLoader.scala | 16 +++++-
.../kyuubi/session/HadoopGroupProvider.scala | 42 ++++++++++++++
.../apache/kyuubi/session/KyuubiSessionImpl.scala | 8 ++-
.../kyuubi/session/KyuubiSessionManager.scala | 8 ++-
.../org/apache/kyuubi/engine/EngineRefTests.scala | 46 ++++++---------
.../engine/EngineRefWithZookeeperSuite.scala | 4 +-
.../apache/kyuubi/plugin/PluginLoaderSuite.scala | 65 ++++++++++++++++------
.../kyuubi/server/api/v1/AdminResourceSuite.scala | 10 ++--
.../kyuubi/server/rest/client/AdminCtlSuite.scala | 2 +-
.../server/rest/client/AdminRestApiSuite.scala | 2 +-
14 files changed, 191 insertions(+), 72 deletions(-)
diff --git a/docs/deployment/settings.md b/docs/deployment/settings.md
index 43be95e0b..e74444ff1 100644
--- a/docs/deployment/settings.md
+++ b/docs/deployment/settings.md
@@ -497,6 +497,7 @@ kyuubi.session.engine.trino.connection.url|<undefined>|The server url that
kyuubi.session.engine.trino.main.resource|<undefined>|The package used to create Trino engine remote job. If it is undefined, Kyuubi will use the default|string|1.5.0
kyuubi.session.engine.trino.showProgress|true|When true, show the progress bar and final info in the trino engine log.|boolean|1.6.0
kyuubi.session.engine.trino.showProgress.debug|false|When true, show the progress debug info in the trino engine log.|boolean|1.6.0
+kyuubi.session.group.provider|hadoop|A group provider plugin for Kyuubi Server. This plugin can provide primary group and groups information for different user or session configs. This config value should be a class which is a child of 'org.apache.kyuubi.plugin.GroupProvider' which has zero-arg constructor. Kyuubi provides the following built-in implementations: <li>hadoop: delegate the user group mapping to hadoop UserGroupInformation.</li>|string|1.7.0
kyuubi.session.idle.timeout|PT6H|session idle timeout, it will be closed when it's not accessed for this duration|duration|1.2.0
kyuubi.session.local.dir.allow.list||The local dir list that are allowed to access by the kyuubi session application. User might set some parameters such as `spark.files` and it will upload some local files when launching the kyuubi engine, if the local dir allow list is defined, kyuubi will check whether the path to upload is in the allow list. Note that, if it is empty, there is no limitation for that and please use absolute path list.|seq|1.6.0
kyuubi.session.name|<undefined>|A human readable name of session and we use empty string by default. This name will be recorded in event. Note that, we only apply this value from session conf.|string|1.4.0
diff --git a/extensions/server/kyuubi-server-plugin/src/main/java/org/apache/kyuubi/plugin/GroupProvider.java b/extensions/server/kyuubi-server-plugin/src/main/java/org/apache/kyuubi/plugin/GroupProvider.java
new file mode 100644
index 000000000..72c372dda
--- /dev/null
+++ b/extensions/server/kyuubi-server-plugin/src/main/java/org/apache/kyuubi/plugin/GroupProvider.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.plugin;
+
+import java.util.Map;
+
+/** Provide groups according session user and session configuration. */
+public interface GroupProvider {
+
+ String primaryGroup(String user, Map<String, String> sessionConf);
+
+ default String[] groups(String user, Map<String, String> sessionConf) {
+ String primaryGroup = primaryGroup(user, sessionConf);
+ return new String[] {primaryGroup};
+ }
+}
diff --git a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
index 53982996a..167da1409 100644
--- a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
+++ b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
@@ -1977,6 +1977,21 @@ object KyuubiConf {
.stringConf
.createOptional
+ val GROUP_PROVIDER: ConfigEntry[String] =
+ buildConf("kyuubi.session.group.provider")
+ .doc("A group provider plugin for Kyuubi Server. This plugin can provide primary group " +
+ "and groups information for different user or session configs. This config value " +
+ "should be a class which is a child of 'org.apache.kyuubi.plugin.GroupProvider' which " +
+ "has zero-arg constructor. Kyuubi provides the following built-in implementations: " +
+ "<li>hadoop: delegate the user group mapping to hadoop UserGroupInformation.</li>")
+ .version("1.7.0")
+ .stringConf
+ .transform {
+ case "hadoop" => "org.apache.kyuubi.session.HadoopGroupProvider"
+ case other => other
+ }
+ .createWithDefault("hadoop")
+
val SERVER_NAME: OptionalConfigEntry[String] =
buildConf("kyuubi.server.name")
.doc("The name of Kyuubi Server.")
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala
index ed6282917..8f96cdb56 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala
@@ -23,7 +23,6 @@ import scala.util.Random
import com.codahale.metrics.MetricRegistry
import com.google.common.annotations.VisibleForTesting
-import org.apache.hadoop.security.UserGroupInformation
import org.apache.kyuubi.{KYUUBI_VERSION, KyuubiSQLException, Logging, Utils}
import org.apache.kyuubi.config.KyuubiConf
@@ -52,6 +51,7 @@ import org.apache.kyuubi.operation.log.OperationLog
private[kyuubi] class EngineRef(
conf: KyuubiConf,
user: String,
+ primaryGroup: String,
engineRefId: String,
engineManager: KyuubiApplicationManager)
extends Logging {
@@ -83,16 +83,7 @@ private[kyuubi] class EngineRef(
// Launcher of the engine
private[kyuubi] val appUser: String = shareLevel match {
case SERVER => Utils.currentUser
- case GROUP =>
- val clientUGI = UserGroupInformation.createRemoteUser(user)
- // Similar to `clientUGI.getPrimaryGroupName` (avoid IOE) to get the Primary GroupName of
- // the client user mapping to
- clientUGI.getGroupNames.headOption match {
- case Some(primaryGroup) => primaryGroup
- case None =>
- warn(s"There is no primary group for $user, use the client user name as group directly")
- user
- }
+ case GROUP => primaryGroup
case _ => user
}
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/plugin/PluginLoader.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/plugin/PluginLoader.scala
index f9183f581..17ad69524 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/plugin/PluginLoader.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/plugin/PluginLoader.scala
@@ -38,7 +38,21 @@ private[kyuubi] object PluginLoader {
throw new KyuubiException(
s"Class ${advisorClass.get} is not a child of '${classOf[SessionConfAdvisor].getName}'.")
case NonFatal(e) =>
- throw new IllegalArgumentException(s"Error while instantiating '${advisorClass.get}':", e)
+ throw new IllegalArgumentException(s"Error while instantiating '${advisorClass.get}': ", e)
+ }
+ }
+
+ def loadGroupProvider(conf: KyuubiConf): GroupProvider = {
+ val groupProviderClass = conf.get(KyuubiConf.GROUP_PROVIDER)
+ try {
+ Class.forName(groupProviderClass).getConstructor().newInstance()
+ .asInstanceOf[GroupProvider]
+ } catch {
+ case _: ClassCastException =>
+ throw new KyuubiException(
+ s"Class $groupProviderClass is not a child of '${classOf[GroupProvider].getName}'.")
+ case NonFatal(e) =>
+ throw new IllegalArgumentException(s"Error while instantiating '$groupProviderClass': ", e)
}
}
}
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/session/HadoopGroupProvider.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/session/HadoopGroupProvider.scala
new file mode 100644
index 000000000..2ae7bb157
--- /dev/null
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/session/HadoopGroupProvider.scala
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.session
+
+import java.util.{Map => JMap}
+
+import org.apache.hadoop.security.UserGroupInformation
+
+import org.apache.kyuubi.Logging
+import org.apache.kyuubi.plugin.GroupProvider
+
+/**
+ * Hadoop based group provider, see more information at
+ * https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/GroupsMapping.html
+ */
+class HadoopGroupProvider extends GroupProvider with Logging {
+ override def primaryGroup(user: String, sessionConf: JMap[String, String]): String =
+ groups(user, sessionConf).head
+
+ override def groups(user: String, sessionConf: JMap[String, String]): Array[String] =
+ UserGroupInformation.createRemoteUser(user).getGroupNames match {
+ case Array() =>
+ warn(s"There is no group for $user, use the client user name as group directly")
+ Array(user)
+ case groups => groups
+ }
+}
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionImpl.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionImpl.scala
index a16a2881c..d6a2a6a98 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionImpl.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionImpl.scala
@@ -74,8 +74,12 @@ class KyuubiSessionImpl(
private lazy val engineCredentials = renewEngineCredentials()
- lazy val engine: EngineRef =
- new EngineRef(sessionConf, user, handle.identifier.toString, sessionManager.applicationManager)
+ lazy val engine: EngineRef = new EngineRef(
+ sessionConf,
+ user,
+ sessionManager.groupProvider.primaryGroup(user, optimizedConf.asJava),
+ handle.identifier.toString,
+ sessionManager.applicationManager)
private[kyuubi] val launchEngineOp = sessionManager.operationManager
.newLaunchEngineOperation(this, sessionConf.get(SESSION_ENGINE_LAUNCH_ASYNC))
diff --git a/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionManager.scala b/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionManager.scala
index 85eaa242c..48cc5b096 100644
--- a/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionManager.scala
+++ b/kyuubi-server/src/main/scala/org/apache/kyuubi/session/KyuubiSessionManager.scala
@@ -32,7 +32,7 @@ import org.apache.kyuubi.engine.KyuubiApplicationManager
import org.apache.kyuubi.metrics.MetricsConstants._
import org.apache.kyuubi.metrics.MetricsSystem
import org.apache.kyuubi.operation.{KyuubiOperationManager, OperationState}
-import org.apache.kyuubi.plugin.{PluginLoader, SessionConfAdvisor}
+import org.apache.kyuubi.plugin.{GroupProvider, PluginLoader, SessionConfAdvisor}
import org.apache.kyuubi.server.metadata.{MetadataManager, MetadataRequestsRetryRef}
import org.apache.kyuubi.server.metadata.api.Metadata
import org.apache.kyuubi.util.SignUtils
@@ -43,8 +43,6 @@ class KyuubiSessionManager private (name: String) extends SessionManager(name) {
val operationManager = new KyuubiOperationManager()
val credentialsManager = new HadoopCredentialsManager()
- // this lazy is must be specified since the conf is null when the class initialization
- lazy val sessionConfAdvisor: SessionConfAdvisor = PluginLoader.loadSessionConfAdvisor(conf)
val applicationManager = new KyuubiApplicationManager()
private lazy val metadataManager: Option[MetadataManager] = {
// Currently, the metadata manager is used by the REST frontend which provides batch job APIs,
@@ -57,6 +55,10 @@ class KyuubiSessionManager private (name: String) extends SessionManager(name) {
}
}
+ // lazy is required for plugins since the conf is null when this class initialization
+ lazy val sessionConfAdvisor: SessionConfAdvisor = PluginLoader.loadSessionConfAdvisor(conf)
+ lazy val groupProvider: GroupProvider = PluginLoader.loadGroupProvider(conf)
+
private var limiter: Option[SessionLimiter] = None
lazy val (signingPrivateKey, signingPublicKey) = SignUtils.generateKeyPair()
diff --git a/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/EngineRefTests.scala b/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/EngineRefTests.scala
index e2cf0d51c..0cd8a3dd5 100644
--- a/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/EngineRefTests.scala
+++ b/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/EngineRefTests.scala
@@ -20,7 +20,6 @@ package org.apache.kyuubi.engine
import java.util.UUID
import java.util.concurrent.Executors
-import org.apache.hadoop.security.UserGroupInformation
import org.scalatest.time.SpanSugar.convertIntToGrainOfTime
import org.apache.kyuubi.{KYUUBI_VERSION, Utils}
@@ -69,7 +68,7 @@ trait EngineRefTests extends KyuubiFunSuite {
Seq(None, Some("suffix")).foreach { domain =>
conf.set(KyuubiConf.ENGINE_SHARE_LEVEL, CONNECTION.toString)
domain.foreach(conf.set(KyuubiConf.ENGINE_SHARE_LEVEL_SUBDOMAIN.key, _))
- val engine = new EngineRef(conf, user, id, null)
+ val engine = new EngineRef(conf, user, "grp", id, null)
assert(engine.engineSpace ===
DiscoveryPaths.makePath(
s"kyuubi_${KYUUBI_VERSION}_${CONNECTION}_${engineType}",
@@ -83,7 +82,7 @@ trait EngineRefTests extends KyuubiFunSuite {
val id = UUID.randomUUID().toString
conf.set(KyuubiConf.ENGINE_SHARE_LEVEL, USER.toString)
conf.set(KyuubiConf.ENGINE_TYPE, FLINK_SQL.toString)
- val appName = new EngineRef(conf, user, id, null)
+ val appName = new EngineRef(conf, user, "grp", id, null)
assert(appName.engineSpace ===
DiscoveryPaths.makePath(
s"kyuubi_${KYUUBI_VERSION}_${USER}_$FLINK_SQL",
@@ -95,7 +94,7 @@ trait EngineRefTests extends KyuubiFunSuite {
k =>
conf.unset(KyuubiConf.ENGINE_SHARE_LEVEL_SUBDOMAIN)
conf.set(k.key, "abc")
- val appName2 = new EngineRef(conf, user, id, null)
+ val appName2 = new EngineRef(conf, user, "grp", id, null)
assert(appName2.engineSpace ===
DiscoveryPaths.makePath(
s"kyuubi_${KYUUBI_VERSION}_${USER}_${FLINK_SQL}",
@@ -109,8 +108,8 @@ trait EngineRefTests extends KyuubiFunSuite {
val id = UUID.randomUUID().toString
conf.set(KyuubiConf.ENGINE_SHARE_LEVEL, GROUP.toString)
conf.set(KyuubiConf.ENGINE_TYPE, SPARK_SQL.toString)
- val engineRef = new EngineRef(conf, user, id, null)
- val primaryGroupName = UserGroupInformation.createRemoteUser(user).getPrimaryGroupName
+ val primaryGroupName = "primary_grp"
+ val engineRef = new EngineRef(conf, user, primaryGroupName, id, null)
assert(engineRef.engineSpace ===
DiscoveryPaths.makePath(
s"kyuubi_${KYUUBI_VERSION}_GROUP_SPARK_SQL",
@@ -123,7 +122,7 @@ trait EngineRefTests extends KyuubiFunSuite {
k =>
conf.unset(k)
conf.set(k.key, "abc")
- val engineRef2 = new EngineRef(conf, user, id, null)
+ val engineRef2 = new EngineRef(conf, user, primaryGroupName, id, null)
assert(engineRef2.engineSpace ===
DiscoveryPaths.makePath(
s"kyuubi_${KYUUBI_VERSION}_${GROUP}_${SPARK_SQL}",
@@ -132,24 +131,13 @@ trait EngineRefTests extends KyuubiFunSuite {
assert(engineRef2.defaultEngineName ===
s"kyuubi_${GROUP}_${SPARK_SQL}_${primaryGroupName}_abc_$id")
}
-
- val userName = "Iamauserwithoutgroup"
- val newUGI = UserGroupInformation.createRemoteUser(userName)
- assert(newUGI.getGroupNames.isEmpty)
- val engineRef3 = new EngineRef(conf, userName, id, null)
- assert(engineRef3.engineSpace ===
- DiscoveryPaths.makePath(
- s"kyuubi_${KYUUBI_VERSION}_GROUP_SPARK_SQL",
- userName,
- Array("abc")))
- assert(engineRef3.defaultEngineName === s"kyuubi_GROUP_SPARK_SQL_${userName}_abc_$id")
}
test("SERVER shared level engine name") {
val id = UUID.randomUUID().toString
conf.set(KyuubiConf.ENGINE_SHARE_LEVEL, SERVER.toString)
conf.set(KyuubiConf.ENGINE_TYPE, FLINK_SQL.toString)
- val appName = new EngineRef(conf, user, id, null)
+ val appName = new EngineRef(conf, user, "grp", id, null)
assert(appName.engineSpace ===
DiscoveryPaths.makePath(
s"kyuubi_${KYUUBI_VERSION}_${SERVER}_${FLINK_SQL}",
@@ -158,7 +146,7 @@ trait EngineRefTests extends KyuubiFunSuite {
assert(appName.defaultEngineName === s"kyuubi_${SERVER}_${FLINK_SQL}_${user}_default_$id")
conf.set(KyuubiConf.ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "abc")
- val appName2 = new EngineRef(conf, user, id, null)
+ val appName2 = new EngineRef(conf, user, "grp", id, null)
assert(appName2.engineSpace ===
DiscoveryPaths.makePath(
s"kyuubi_${KYUUBI_VERSION}_${SERVER}_${FLINK_SQL}",
@@ -173,31 +161,31 @@ trait EngineRefTests extends KyuubiFunSuite {
// set subdomain and disable engine pool
conf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "abc")
conf.set(ENGINE_POOL_SIZE, -1)
- val engine1 = new EngineRef(conf, user, id, null)
+ val engine1 = new EngineRef(conf, user, "grp", id, null)
assert(engine1.subdomain === "abc")
// unset subdomain and disable engine pool
conf.unset(ENGINE_SHARE_LEVEL_SUBDOMAIN)
conf.set(ENGINE_POOL_SIZE, -1)
- val engine2 = new EngineRef(conf, user, id, null)
+ val engine2 = new EngineRef(conf, user, "grp", id, null)
assert(engine2.subdomain === "default")
// set subdomain and 1 <= engine pool size < threshold
conf.set(ENGINE_SHARE_LEVEL_SUBDOMAIN.key, "abc")
conf.set(ENGINE_POOL_SIZE, 1)
- val engine3 = new EngineRef(conf, user, id, null)
+ val engine3 = new EngineRef(conf, user, "grp", id, null)
assert(engine3.subdomain === "abc")
// unset subdomain and 1 <= engine pool size < threshold
conf.unset(ENGINE_SHARE_LEVEL_SUBDOMAIN)
conf.set(ENGINE_POOL_SIZE, 3)
- val engine4 = new EngineRef(conf, user, id, null)
+ val engine4 = new EngineRef(conf, user, "grp", id, null)
assert(engine4.subdomain.startsWith("engine-pool-"))
// unset subdomain and engine pool size > threshold
conf.unset(ENGINE_SHARE_LEVEL_SUBDOMAIN)
conf.set(ENGINE_POOL_SIZE, 100)
- val engine5 = new EngineRef(conf, user, id, null)
+ val engine5 = new EngineRef(conf, user, "grp", id, null)
val engineNumber = Integer.parseInt(engine5.subdomain.substring(12))
val threshold = ENGINE_POOL_SIZE_THRESHOLD.defaultVal.get
assert(engineNumber <= threshold)
@@ -207,7 +195,7 @@ trait EngineRefTests extends KyuubiFunSuite {
val enginePoolName = "test-pool"
conf.set(ENGINE_POOL_NAME, enginePoolName)
conf.set(ENGINE_POOL_SIZE, 3)
- val engine6 = new EngineRef(conf, user, id, null)
+ val engine6 = new EngineRef(conf, user, "grp", id, null)
assert(engine6.subdomain.startsWith(s"$enginePoolName-"))
conf.unset(ENGINE_SHARE_LEVEL_SUBDOMAIN)
@@ -218,7 +206,7 @@ trait EngineRefTests extends KyuubiFunSuite {
conf.set(HighAvailabilityConf.HA_ADDRESSES, getConnectString())
conf.set(ENGINE_POOL_BALANCE_POLICY, "POLLING")
(0 until (10)).foreach { i =>
- val engine7 = new EngineRef(conf, user, id, null)
+ val engine7 = new EngineRef(conf, user, "grp", id, null)
val engineNumber = Integer.parseInt(engine7.subdomain.substring(pool_name.length + 1))
assert(engineNumber == (i % conf.get(ENGINE_POOL_SIZE)))
}
@@ -231,7 +219,7 @@ trait EngineRefTests extends KyuubiFunSuite {
conf.set(KyuubiConf.FRONTEND_THRIFT_BINARY_BIND_PORT, 0)
conf.set(HighAvailabilityConf.HA_NAMESPACE, "engine_test")
conf.set(HighAvailabilityConf.HA_ADDRESSES, getConnectString())
- val engine = new EngineRef(conf, user, id, null)
+ val engine = new EngineRef(conf, user, id, "grp", null)
var port1 = 0
var port2 = 0
@@ -284,7 +272,7 @@ trait EngineRefTests extends KyuubiFunSuite {
executor.execute(() => {
DiscoveryClientProvider.withDiscoveryClient(cloned) { client =>
try {
- new EngineRef(cloned, user, id, null).getOrCreate(client)
+ new EngineRef(cloned, user, "grp", id, null).getOrCreate(client)
} finally {
times(i) = System.currentTimeMillis()
}
diff --git a/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/EngineRefWithZookeeperSuite.scala b/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/EngineRefWithZookeeperSuite.scala
index 75dd5cc09..8695e13c4 100644
--- a/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/EngineRefWithZookeeperSuite.scala
+++ b/kyuubi-server/src/test/scala/org/apache/kyuubi/engine/EngineRefWithZookeeperSuite.scala
@@ -74,7 +74,7 @@ class EngineRefWithZookeeperSuite extends EngineRefTests {
executor.execute(() => {
DiscoveryClientProvider.withDiscoveryClient(conf1) { client =>
try {
- new EngineRef(conf1, user, UUID.randomUUID().toString, null)
+ new EngineRef(conf1, user, "grp", UUID.randomUUID().toString, null)
.getOrCreate(client)
} finally {
times(0) = System.currentTimeMillis()
@@ -84,7 +84,7 @@ class EngineRefWithZookeeperSuite extends EngineRefTests {
executor.execute(() => {
DiscoveryClientProvider.withDiscoveryClient(conf2) { client =>
try {
- new EngineRef(conf2, user, UUID.randomUUID().toString, null)
+ new EngineRef(conf2, user, "grp", UUID.randomUUID().toString, null)
.getOrCreate(client)
} finally {
times(1) = System.currentTimeMillis()
diff --git a/kyuubi-server/src/test/scala/org/apache/kyuubi/plugin/PluginLoaderSuite.scala b/kyuubi-server/src/test/scala/org/apache/kyuubi/plugin/PluginLoaderSuite.scala
index 6edac374a..e24b79c2c 100644
--- a/kyuubi-server/src/test/scala/org/apache/kyuubi/plugin/PluginLoaderSuite.scala
+++ b/kyuubi-server/src/test/scala/org/apache/kyuubi/plugin/PluginLoaderSuite.scala
@@ -21,15 +21,15 @@ import scala.collection.JavaConverters._
import org.apache.kyuubi.{KyuubiException, KyuubiFunSuite}
import org.apache.kyuubi.config.KyuubiConf
-import org.apache.kyuubi.session.FileSessionConfAdvisor
+import org.apache.kyuubi.session.{FileSessionConfAdvisor, HadoopGroupProvider}
class PluginLoaderSuite extends KyuubiFunSuite {
- test("test engine conf advisor wrong class") {
+ test("SessionConfAdvisor - wrong class") {
val conf = new KyuubiConf(false)
assert(PluginLoader.loadSessionConfAdvisor(conf).isInstanceOf[DefaultSessionConfAdvisor])
- conf.set(KyuubiConf.SESSION_CONF_ADVISOR, classOf[FakeAdvisor].getName)
+ conf.set(KyuubiConf.SESSION_CONF_ADVISOR, classOf[InvalidSessionConfAdvisor].getName)
val msg1 = intercept[KyuubiException] {
PluginLoader.loadSessionConfAdvisor(conf)
}.getMessage
@@ -40,10 +40,9 @@ class PluginLoaderSuite extends KyuubiFunSuite {
PluginLoader.loadSessionConfAdvisor(conf)
}.getMessage
assert(msg2.startsWith("Error while instantiating 'non.exists'"))
-
}
- test("test FileSessionConfAdvisor") {
+ test("FileSessionConfAdvisor") {
val conf = new KyuubiConf(false)
conf.set(KyuubiConf.SESSION_CONF_ADVISOR, classOf[FileSessionConfAdvisor].getName)
val advisor = PluginLoader.loadSessionConfAdvisor(conf)
@@ -51,20 +50,52 @@ class PluginLoaderSuite extends KyuubiFunSuite {
assert(emptyConfig.isEmpty)
conf.set(KyuubiConf.SESSION_CONF_PROFILE, "non.exists")
- val nonexistsConfig = advisor.getConfOverlay("chris", conf.getAll.asJava)
- assert(nonexistsConfig.isEmpty)
+ val nonExistsConfig = advisor.getConfOverlay("chris", conf.getAll.asJava)
+ assert(nonExistsConfig.isEmpty)
conf.set(KyuubiConf.SESSION_CONF_PROFILE, "cluster-a")
- val clusteraConf = advisor.getConfOverlay("chris", conf.getAll.asJava)
- assert(clusteraConf.get("kyuubi.ha.namespace") == "kyuubi-ns-a")
- assert(clusteraConf.get("kyuubi.zk.ha.namespace") == null)
- assert(clusteraConf.size() == 5)
-
- val clusteraConfFromCache = advisor.getConfOverlay("chris", conf.getAll.asJava)
- assert(clusteraConfFromCache.get("kyuubi.ha.namespace") == "kyuubi-ns-a")
- assert(clusteraConfFromCache.get("kyuubi.zk.ha.namespace") == null)
- assert(clusteraConfFromCache.size() == 5)
+ val clusterAConf = advisor.getConfOverlay("chris", conf.getAll.asJava)
+ assert(clusterAConf.get("kyuubi.ha.namespace") == "kyuubi-ns-a")
+ assert(clusterAConf.get("kyuubi.zk.ha.namespace") == null)
+ assert(clusterAConf.size() == 5)
+
+ val clusterAConfFromCache = advisor.getConfOverlay("chris", conf.getAll.asJava)
+ assert(clusterAConfFromCache.get("kyuubi.ha.namespace") == "kyuubi-ns-a")
+ assert(clusterAConfFromCache.get("kyuubi.zk.ha.namespace") == null)
+ assert(clusterAConfFromCache.size() == 5)
+ }
+
+ test("GroupProvider - wrong class") {
+ val conf = new KyuubiConf(false)
+ conf.set(KyuubiConf.GROUP_PROVIDER, "hadoop")
+ assert(PluginLoader.loadGroupProvider(conf).isInstanceOf[HadoopGroupProvider])
+
+ conf.set(KyuubiConf.GROUP_PROVIDER, classOf[HadoopGroupProvider].getName)
+ assert(PluginLoader.loadGroupProvider(conf).isInstanceOf[HadoopGroupProvider])
+
+ conf.set(KyuubiConf.GROUP_PROVIDER, classOf[InvalidGroupProvider].getName)
+ val msg1 = intercept[KyuubiException] {
+ PluginLoader.loadGroupProvider(conf)
+ }.getMessage
+ assert(msg1.contains(s"is not a child of '${classOf[GroupProvider].getName}'"))
+
+ conf.set(KyuubiConf.GROUP_PROVIDER, "non.exists")
+ val msg2 = intercept[IllegalArgumentException] {
+ PluginLoader.loadGroupProvider(conf)
+ }.getMessage
+ assert(msg2.startsWith("Error while instantiating 'non.exists'"))
+ }
+
+ test("HadoopGroupProvider") {
+ val conf = new KyuubiConf(false)
+ conf.set(KyuubiConf.GROUP_PROVIDER, "hadoop")
+ val groupProvider = PluginLoader.loadGroupProvider(conf)
+ assert(groupProvider.isInstanceOf[HadoopGroupProvider])
+ val user = "somebody"
+ assert(groupProvider.primaryGroup(user, Map.empty[String, String].asJava) === user)
+ assert(groupProvider.groups(user, Map.empty[String, String].asJava) === Array(user))
}
}
-class FakeAdvisor {}
+class InvalidSessionConfAdvisor
+class InvalidGroupProvider
diff --git a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/AdminResourceSuite.scala b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/AdminResourceSuite.scala
index 4d1088442..d58c1d00a 100644
--- a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/AdminResourceSuite.scala
+++ b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/api/v1/AdminResourceSuite.scala
@@ -73,7 +73,7 @@ class AdminResourceSuite extends KyuubiFunSuite with RestFrontendTestHelper {
conf.set(KyuubiConf.FRONTEND_THRIFT_BINARY_BIND_PORT, 0)
conf.set(HighAvailabilityConf.HA_NAMESPACE, "kyuubi_test")
conf.set(KyuubiConf.ENGINE_IDLE_TIMEOUT, 180000L)
- val engine = new EngineRef(conf.clone, Utils.currentUser, id, null)
+ val engine = new EngineRef(conf.clone, Utils.currentUser, "grp", id, null)
val engineSpace = DiscoveryPaths.makePath(
s"kyuubi_test_${KYUUBI_VERSION}_USER_SPARK_SQL",
@@ -120,7 +120,7 @@ class AdminResourceSuite extends KyuubiFunSuite with RestFrontendTestHelper {
conf.set(KyuubiConf.ENGINE_IDLE_TIMEOUT, 180000L)
val id = UUID.randomUUID().toString
- val engine = new EngineRef(conf.clone, Utils.currentUser, id, null)
+ val engine = new EngineRef(conf.clone, Utils.currentUser, "grp", id, null)
val engineSpace = DiscoveryPaths.makePath(
s"kyuubi_test_${KYUUBI_VERSION}_CONNECTION_SPARK_SQL",
Utils.currentUser,
@@ -156,7 +156,7 @@ class AdminResourceSuite extends KyuubiFunSuite with RestFrontendTestHelper {
conf.set(KyuubiConf.FRONTEND_THRIFT_BINARY_BIND_PORT, 0)
conf.set(HighAvailabilityConf.HA_NAMESPACE, "kyuubi_test")
conf.set(KyuubiConf.ENGINE_IDLE_TIMEOUT, 180000L)
- val engine = new EngineRef(conf.clone, Utils.currentUser, id, null)
+ val engine = new EngineRef(conf.clone, Utils.currentUser, id, "grp", null)
val engineSpace = DiscoveryPaths.makePath(
s"kyuubi_test_${KYUUBI_VERSION}_USER_SPARK_SQL",
@@ -208,14 +208,14 @@ class AdminResourceSuite extends KyuubiFunSuite with RestFrontendTestHelper {
Array(""))
val id1 = UUID.randomUUID().toString
- val engine1 = new EngineRef(conf.clone, Utils.currentUser, id1, null)
+ val engine1 = new EngineRef(conf.clone, Utils.currentUser, "grp", id1, null)
val engineSpace1 = DiscoveryPaths.makePath(
s"kyuubi_test_${KYUUBI_VERSION}_CONNECTION_SPARK_SQL",
Utils.currentUser,
Array(id1))
val id2 = UUID.randomUUID().toString
- val engine2 = new EngineRef(conf.clone, Utils.currentUser, id2, null)
+ val engine2 = new EngineRef(conf.clone, Utils.currentUser, "grp", id2, null)
val engineSpace2 = DiscoveryPaths.makePath(
s"kyuubi_test_${KYUUBI_VERSION}_CONNECTION_SPARK_SQL",
Utils.currentUser,
diff --git a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/AdminCtlSuite.scala b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/AdminCtlSuite.scala
index 10c636924..7a968dca8 100644
--- a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/AdminCtlSuite.scala
+++ b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/AdminCtlSuite.scala
@@ -54,7 +54,7 @@ class AdminCtlSuite extends RestClientTestHelper with TestPrematureExit {
conf.set(KyuubiConf.ENGINE_IDLE_TIMEOUT, 180000L)
conf.set(KyuubiConf.AUTHENTICATION_METHOD, Seq("LDAP", "CUSTOM"))
val user = ldapUser
- val engine = new EngineRef(conf.clone, user, id, null)
+ val engine = new EngineRef(conf.clone, user, "grp", id, null)
val engineSpace = DiscoveryPaths.makePath(
s"kyuubi_test_${KYUUBI_VERSION}_USER_SPARK_SQL",
diff --git a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/AdminRestApiSuite.scala b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/AdminRestApiSuite.scala
index bea20b997..685096dfd 100644
--- a/kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/AdminRestApiSuite.scala
+++ b/kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/AdminRestApiSuite.scala
@@ -47,7 +47,7 @@ class AdminRestApiSuite extends RestClientTestHelper {
conf.set(KyuubiConf.ENGINE_IDLE_TIMEOUT, 180000L)
conf.set(KyuubiConf.AUTHENTICATION_METHOD, Seq("LDAP", "CUSTOM"))
val user = ldapUser
- val engine = new EngineRef(conf.clone, user, id, null)
+ val engine = new EngineRef(conf.clone, user, "grp", id, null)
val engineSpace = DiscoveryPaths.makePath(
s"kyuubi_test_${KYUUBI_VERSION}_USER_SPARK_SQL",