You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ch...@apache.org on 2026/05/15 06:01:49 UTC
(kyuubi) branch master updated: [KYUUBI #7438] Make JSON event logger permission management configurable
This is an automated email from the ASF dual-hosted git repository.
pan3793 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 7282a68075 [KYUUBI #7438] Make JSON event logger permission management configurable
7282a68075 is described below
commit 7282a68075e45927b308d2150c4b72eae66d9527
Author: kongjiexin <ko...@gmail.com>
AuthorDate: Fri May 15 14:01:40 2026 +0800
[KYUUBI #7438] Make JSON event logger permission management configurable
### Why are the changes needed?
The built-in JSON event logger currently creates event log directories and actively sets permissions on directories and log files.
This default behavior is useful for the built-in permission model, but it can conflict with deployments where access control is managed by Apache Ranger, inherited HDFS ACLs, or other external authorization systems. In those environments, an engine user may have permission to create and write event logs, while still not being allowed to call `setPermission` on the target path.
This patch adds `kyuubi.event.json.log.manage.permissions.enabled`, defaulting to `true`, to keep the existing behavior unchanged. When set to `false`, the JSON event logger no longer actively manages directory or file permissions, leaving access control to Ranger, HDFS ACLs, filesystem defaults, or other external permission systems.
This helps multi-tenant Kyuubi deployments use a shared JSON event log root without requiring
each engine user to own or chmod the target HDFS path.
### How was this patch tested?
- [x] Added tests for default JSON event log file permission management.
- [x] Added tests for writing JSON event logs when permission management is disabled.
- [x] Ran:
```bash
build/mvn -pl kyuubi-events -am -DwildcardSuites=org.apache.kyuubi.events.handler.JsonLoggingEventHandlerSuite test
```
### Was this patch authored or co-authored using generative AI tooling?
This patch was prepared with assistance from OpenAI Codex; the contributor reviewed and validated the changes.
Closes #7439 from kongjiexin888-source/json-event-log-permission-management.
Closes #7438
afbd81598 [kongjiexin] Make JSON event logger permission management configurable
Authored-by: kongjiexin <ko...@gmail.com>
Signed-off-by: Cheng Pan <ch...@apache.org>
---
docs/configuration/settings.md | 11 +--
.../org/apache/kyuubi/config/KyuubiConf.scala | 11 +++
.../events/handler/JsonLoggingEventHandler.scala | 22 +++++-
.../handler/JsonLoggingEventHandlerSuite.scala | 85 ++++++++++++++++++++++
4 files changed, 121 insertions(+), 8 deletions(-)
diff --git a/docs/configuration/settings.md b/docs/configuration/settings.md
index bac908c593..73a0d4f617 100644
--- a/docs/configuration/settings.md
+++ b/docs/configuration/settings.md
@@ -232,11 +232,12 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co
### Event
-| Key | Default | Meaning | Type | Since |
-|-----------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------|----------|-------|
-| kyuubi.event.async.pool.keepalive.time | PT1M | Time(ms) that an idle async thread of the async event handler thread pool will wait for a new task to arrive before terminating | duration | 1.7.0 |
-| kyuubi.event.async.pool.size | 8 | Number of threads in the async event handler thread pool | int | 1.7.0 |
-| kyuubi.event.async.pool.wait.queue.size | 100 | Size of the wait queue for the async event handler thread pool | int | 1.7.0 |
+| Key | Default | Meaning | Type | Since |
+|-------------------------------------------------|---------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|--------|
+| kyuubi.event.async.pool.keepalive.time | PT1M | Time(ms) that an idle async thread of the async event handler thread pool will wait for a new task to arrive before terminating | duration | 1.7.0 |
+| kyuubi.event.async.pool.size | 8 | Number of threads in the async event handler thread pool | int | 1.7.0 |
+| kyuubi.event.async.pool.wait.queue.size | 100 | Size of the wait queue for the async event handler thread pool | int | 1.7.0 |
+| kyuubi.event.json.log.managePermissions.enabled | true | Whether the built-in JSON event logger manages the permissions of event log directories and files. When enabled, Kyuubi sets event log directories to 770 and event log files to 660. When disabled, Kyuubi only creates and writes event log paths, leaving permissions to the underlying file system defaults, inherited ACLs, or external authorization systems. | boolean | 1.12.0 |
### Frontend
diff --git a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
index b9762a9792..7d0c5a6301 100644
--- a/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
+++ b/kyuubi-common/src/main/scala/org/apache/kyuubi/config/KyuubiConf.scala
@@ -2649,6 +2649,17 @@ object KyuubiConf {
.stringConf
.createWithDefault("file:///tmp/kyuubi/events")
+ val EVENT_JSON_LOG_MANAGE_PERMISSIONS_ENABLED: ConfigEntry[Boolean] =
+ buildConf("kyuubi.event.json.log.managePermissions.enabled")
+ .doc("Whether the built-in JSON event logger manages the permissions of event log " +
+ "directories and files. When enabled, Kyuubi sets event log directories to 770 and " +
+ "event log files to 660. When disabled, Kyuubi only creates and writes event log " +
+ "paths, leaving permissions to the underlying file system defaults, inherited ACLs, " +
+ "or external authorization systems.")
+ .version("1.12.0")
+ .booleanConf
+ .createWithDefault(true)
+
val SERVER_EVENT_KAFKA_TOPIC: OptionalConfigEntry[String] =
buildConf("kyuubi.backend.server.event.kafka.topic")
.doc("The topic of server events go for the built-in Kafka logger")
diff --git a/kyuubi-events/src/main/scala/org/apache/kyuubi/events/handler/JsonLoggingEventHandler.scala b/kyuubi-events/src/main/scala/org/apache/kyuubi/events/handler/JsonLoggingEventHandler.scala
index 77d80b1521..2140eddc5f 100644
--- a/kyuubi-events/src/main/scala/org/apache/kyuubi/events/handler/JsonLoggingEventHandler.scala
+++ b/kyuubi-events/src/main/scala/org/apache/kyuubi/events/handler/JsonLoggingEventHandler.scala
@@ -29,6 +29,7 @@ import org.apache.hadoop.fs.permission.FsPermission
import org.apache.kyuubi.Logging
import org.apache.kyuubi.config.{ConfigEntry, KyuubiConf}
+import org.apache.kyuubi.config.KyuubiConf.EVENT_JSON_LOG_MANAGE_PERMISSIONS_ENABLED
import org.apache.kyuubi.events.KyuubiEvent
/**
@@ -53,6 +54,7 @@ class JsonLoggingEventHandler(
private var logRoot: URI = _
private var fs: FileSystem = _
private val writers = HashMap.empty[String, Logger]
+ private val manageLogPermissions = kyuubiConf.get(EVENT_JSON_LOG_MANAGE_PERMISSIONS_ENABLED)
initialize(kyuubiConf)
@@ -86,7 +88,7 @@ class JsonLoggingEventHandler(
} else {
new Path(new Path(new Path(logRoot), event.eventType), partitions)
}
- FileSystem.mkdirs(fs, eventPath, JSON_LOG_DIR_PERM)
+ createLogDir(eventPath)
val logFile = new Path(eventPath, logName + ".json")
var hadoopDataStream: FSDataOutputStream = null
val rawStream =
@@ -96,13 +98,27 @@ class JsonLoggingEventHandler(
hadoopDataStream = fs.create(logFile)
hadoopDataStream
}
- fs.setPermission(logFile, JSON_LOG_FILE_PERM)
+ manageLogFilePermission(logFile)
val bStream = new BufferedOutputStream(rawStream)
info(s"Logging kyuubi events to $logFile")
(new PrintWriter(bStream), Option(hadoopDataStream))
})
}
+ private def createLogDir(path: Path): Unit = {
+ if (manageLogPermissions) {
+ FileSystem.mkdirs(fs, path, JSON_LOG_DIR_PERM)
+ } else {
+ fs.mkdirs(path)
+ }
+ }
+
+ private def manageLogFilePermission(path: Path): Unit = {
+ if (manageLogPermissions) {
+ fs.setPermission(path, JSON_LOG_FILE_PERM)
+ }
+ }
+
private def requireLogRootWritable(): Unit = {
val fileStatus = fs.getFileStatus(new Path(logRoot))
if (!fileStatus.isDirectory) {
@@ -113,7 +129,7 @@ class JsonLoggingEventHandler(
private def initialize(conf: KyuubiConf): Unit = synchronized {
logRoot = URI.create(conf.get(logPath))
fs = FileSystem.get(logRoot, hadoopConf)
- FileSystem.mkdirs(fs, new Path(logRoot), JSON_LOG_DIR_PERM)
+ createLogDir(new Path(logRoot))
requireLogRootWritable()
}
}
diff --git a/kyuubi-events/src/test/scala/org/apache/kyuubi/events/handler/JsonLoggingEventHandlerSuite.scala b/kyuubi-events/src/test/scala/org/apache/kyuubi/events/handler/JsonLoggingEventHandlerSuite.scala
new file mode 100644
index 0000000000..0683802178
--- /dev/null
+++ b/kyuubi-events/src/test/scala/org/apache/kyuubi/events/handler/JsonLoggingEventHandlerSuite.scala
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.events.handler
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.{FileSystem, Path}
+
+import org.apache.kyuubi.{KyuubiFunSuite, Utils}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.config.KyuubiConf.{ENGINE_EVENT_JSON_LOG_PATH, EVENT_JSON_LOG_MANAGE_PERMISSIONS_ENABLED}
+import org.apache.kyuubi.events.KyuubiEvent
+
+class JsonLoggingEventHandlerSuite extends KyuubiFunSuite {
+
+ test("manage json event log file permissions by default") {
+ val logRoot = createLogRoot()
+ val conf = KyuubiConf().set(ENGINE_EVENT_JSON_LOG_PATH, logRoot)
+ val handler = new JsonLoggingEventHandler(
+ "test-log",
+ ENGINE_EVENT_JSON_LOG_PATH,
+ new Configuration(),
+ conf)
+
+ try {
+ handler(JsonLogPermissionTestEvent("managed"))
+ } finally {
+ handler.close()
+ }
+
+ val fs = FileSystem.get(new java.net.URI(logRoot), new Configuration())
+ val logFile = new Path(logRoot, "json_log_permission_test/day=2026-05-06/test-log.json")
+ assert(fs.exists(logFile))
+ val expectedPermission = Integer.parseInt("660", 8)
+ val actualPermission = fs.getFileStatus(logFile).getPermission.toShort &
+ Integer.parseInt("777", 8)
+ assert(actualPermission == expectedPermission)
+ }
+
+ test("write json event logs without managing permissions") {
+ val logRoot = createLogRoot()
+ val conf = KyuubiConf()
+ .set(ENGINE_EVENT_JSON_LOG_PATH, logRoot)
+ .set(EVENT_JSON_LOG_MANAGE_PERMISSIONS_ENABLED, false)
+ val handler = new JsonLoggingEventHandler(
+ "test-log",
+ ENGINE_EVENT_JSON_LOG_PATH,
+ new Configuration(),
+ conf)
+
+ try {
+ handler(JsonLogPermissionTestEvent("unmanaged"))
+ } finally {
+ handler.close()
+ }
+
+ val fs = FileSystem.get(new java.net.URI(logRoot), new Configuration())
+ val eventPath = new Path(logRoot, "json_log_permission_test/day=2026-05-06")
+ val logFile = new Path(eventPath, "test-log.json")
+ assert(fs.getFileStatus(eventPath).isDirectory)
+ assert(fs.getFileStatus(logFile).isFile)
+ }
+
+ private def createLogRoot(): String = {
+ Utils.createTempDir("kyuubi-json-log").toUri.toString
+ }
+}
+
+case class JsonLogPermissionTestEvent(message: String) extends KyuubiEvent {
+ override def partitions: Seq[(String, String)] = Seq("day" -> "2026-05-06")
+}