You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "wangzhigang1999 (via GitHub)" <gi...@apache.org> on 2026/05/05 16:01:41 UTC
[PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
wangzhigang1999 opened a new pull request, #7431:
URL: https://github.com/apache/kyuubi/pull/7431
### Why are the changes needed?
Part 4 of 4 for the Data Agent Engine ([umbrella](https://github.com/apache/kyuubi/issues/7379), [KPIP-7373](https://github.com/apache/kyuubi/discussions/7373)).
This PR adds the user-facing surface — REST API, SSE streaming, and the Web UI chat interface — completing the Data Agent feature. Builds on the engine runtime from PR 3 (#7415).
Changes include:
**Server / REST API**
- `DataAgentResource` — `POST /sessions/{handle}/chat` (SSE) and `POST /sessions/{handle}/approve` (JSON)
- `ChatRequest` / `ApprovalRequest` DTOs in `kyuubi-client-api`
- Stream lifecycle: bounded op-submit pool, preflight validation before engine launch, deadline-bounded poll loop, terminal-state-aware error surfacing, named `ping` keepalive
- Request validation: model-name pattern, approval-mode whitelist, requestId charset (prevents engine command injection)
- Bundles SQLite and PostgreSQL JDBC drivers in the engine package (Apache-compatible); MySQL stays test-scope
**Engine streaming**
- `ReasoningDelta` event for streaming chain-of-thought (qwen3, DeepSeek-R1) via the OpenAI-compatible `reasoning_content` delta field
- `AgentRunContext` token accounting split into `accumulated*` (billing) and `last*` (current context size for UIs); provider-reported total forwarded verbatim so `CompactionMiddleware` can size the next prompt correctly
**Web UI**
- New `/data-agent` route with chat view, multi-tab session management, tool-call/approval rendering, reasoning collapse, markdown + sanitized HTML
- Pinia store for chat state, persistent across tab switches
- `@microsoft/fetch-event-source` based SSE client with watchdog, reconnect on visibility, and inline approval flow
### How was this patch tested?
- **Unit tests**: `EventTest` (event-type SSE name parity), web-ui `chat-stream.spec.ts` (SSE parsing, error frame handling)
- **Live LLM integration tests** (require `DATA_AGENT_OPENAI_API_KEY`): full ReAct loop, approval gate, denial flow, compaction trigger — all green against dashscope qwen-plus
- **Manual E2E**: ran the dev server, exercised chat, model override, approval/deny, and reconnect across tab switches against a live engine
### Was this patch authored or co-authored using generative AI tooling?
Partially assisted by Claude Code (Claude Opus 4.7) for code review, test generation, and PR formatting. Core design and implementation are human-authored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#issuecomment-4414955610
@pan3793 I have added complete usage documentation and configuration instructions. Ready for your further review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3468496566
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,18 +118,46 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // For Kyuubi/HiveServer2 backends, pass the session user for proxy-user impersonation —
+ // but only when the URL does not already carry explicit credentials, because URL-supplied
+ // credentials should win (some users pass credentials via session variables).
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
+ boolean isKyuubiBackend = "kyuubi".equals(subprotocol) || "hive2".equals(subprotocol);
+ String hikariUser = null;
+ if (isKyuubiBackend && !hasKyuubiUrlCredentials(jdbcUrl)) {
+ hikariUser = ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
+ }
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ DataSource ds = DataSourceFactory.create(jdbcUrl, hikariUser);
Review Comment:
Yes, since I had never used Kerberos before, I’ve recently been learning it and attempting to fix this issue; however, it’s indeed more complex than anticipated. I agree with temporarily disabling Kerberos authentication to unblock this PR, and we’ll address the Kerberos issue separately in a smaller, dedicated PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3474544835
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,18 +118,46 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // For Kyuubi/HiveServer2 backends, pass the session user for proxy-user impersonation —
+ // but only when the URL does not already carry explicit credentials, because URL-supplied
+ // credentials should win (some users pass credentials via session variables).
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
+ boolean isKyuubiBackend = "kyuubi".equals(subprotocol) || "hive2".equals(subprotocol);
+ String hikariUser = null;
+ if (isKyuubiBackend && !hasKyuubiUrlCredentials(jdbcUrl)) {
+ hikariUser = ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
+ }
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ DataSource ds = DataSourceFactory.create(jdbcUrl, hikariUser);
Review Comment:
many AI apps run agents/tools in a sandbox, maybe we need to design a sandbox approach in the future, for example, run the data agent process in a container
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3298363674
##########
kyuubi-server/web-ui/package.json:
##########
@@ -19,8 +19,12 @@
},
"dependencies": {
"@element-plus/icons-vue": "^2.0.9",
+ "@microsoft/fetch-event-source": "^2.0.1",
+ "@types/dompurify": "^3.0.5",
"date-fns": "^2.29.3",
+ "dompurify": "^3.4.2",
"element-plus": "^2.2.12",
+ "marked": "^16.1.2",
Review Comment:
LICENSE-binary needs to be updated when adding new deps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#issuecomment-4797390679
@pan3793 I disabled Kerberos and added a TODO; I also updated the related configuration instructions and documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3298304125
##########
docs/quick_start/quick_start_with_data_agent.md:
##########
@@ -0,0 +1,211 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements. See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License. You may obtain a copy of the License at
+-
+- http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Data Agent Engine
+
+The Data Agent engine lets users explore and query data with natural language. It is a Kyuubi engine type (`DATA_AGENT`) that runs an LLM-driven [ReAct](https://arxiv.org/abs/2210.03629) agent. The agent reasons about the user's request, calls a small set of SQL tools against a JDBC-accessible backend (Spark, Trino, MySQL, SQLite, ...), and streams its responses to the bundled web UI.
+
+This page covers how to configure the engine, how the web UI behaves at runtime, the approval / risk model that gates SQL tool execution, and a few deployment notes worth knowing.
+
+## At a glance
+
+- **Engine type**: `DATA_AGENT`, share level follows `kyuubi.engine.share.level` like every other engine
+- **LLM provider**: any OpenAI-compatible chat-completion endpoint (OpenAI, Azure OpenAI, vLLM, llama.cpp, or any other provider that exposes the OpenAI chat-completion API)
+- **Datasource**: any JDBC URL the engine can reach. When unset, defaults to ZooKeeper service discovery for ZK-HA deployments, or `jdbc:hive2://localhost:<thrift-port>/default` otherwise
+- **Tools** (built in): `run_select_query` (low-risk, read-only) and `run_mutation_query` (high-risk, mutating). Per-dialect prompt templates ship for Spark / Trino / MySQL / SQLite / Generic
+- **Interface**: the bundled Kyuubi web UI. Pick **Data Agent** from the left navigation after logging in
+
+## Quick start
+
+### 1. Configure the LLM provider
+
+Add the following to `$KYUUBI_HOME/conf/kyuubi-defaults.conf`:
+
+```
+kyuubi.engine.data.agent.provider OPENAI_COMPATIBLE
+
+# OpenAI-compatible endpoint and credentials
+kyuubi.engine.data.agent.openai.endpoint https://api.openai.com/v1
Review Comment:
nit: alignment
```suggestion
kyuubi.engine.data.agent.openai.endpoint https://api.openai.com/v1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3298357282
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala:
##########
@@ -258,15 +259,28 @@ private[kyuubi] class EngineRef(
engineRefId,
extraEngineLog,
defaultEngineName)
+ case CHAT =>
+ new ChatProcessBuilder(appUser, doAsEnabled, conf, engineRefId, extraEngineLog)
case DATA_AGENT =>
if (conf.get(ENGINE_DATA_AGENT_JDBC_URL).isEmpty) {
val haAddresses = conf.get(HA_ADDRESSES)
- if (haAddresses.nonEmpty) {
- val jdbcUrl = s"jdbc:hive2://$haAddresses/default;" +
+ val isZkHa = haAddresses.nonEmpty &&
+ conf.get(HA_CLIENT_CLASS).endsWith("ZookeeperDiscoveryClient")
+ val jdbcUrl = if (isZkHa) {
+ s"jdbc:hive2://$haAddresses/default;" +
s"serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=$serverSpace"
- conf.set(ENGINE_DATA_AGENT_JDBC_URL.key, jdbcUrl)
- info(s"Data Agent JDBC URL not configured, using Kyuubi server via ZK: $jdbcUrl")
+ } else {
+ val port = conf.get(FRONTEND_THRIFT_BINARY_BIND_PORT)
+ if (port == 0) {
+ throw KyuubiSQLException(
+ s"Cannot derive a default Data Agent JDBC URL: " +
+ s"${FRONTEND_THRIFT_BINARY_BIND_PORT.key} is 0 (random). " +
+ s"Set ${ENGINE_DATA_AGENT_JDBC_URL.key} explicitly.")
+ }
+ s"jdbc:hive2://localhost:$port/default"
Review Comment:
try use the real bound address instead of hardcoded localhost
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3298366956
##########
kyuubi-server/web-ui/src/views/data-agent/utils/jdbc.ts:
##########
@@ -0,0 +1,110 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+export const JDBC_TEMPLATES = [
+ {
+ label: 'Spark / Hive (Thrift)',
+ value:
+ 'jdbc:hive2://localhost:10009/default;user=username;password=password',
Review Comment:
same, prefer `jdbc:kyuubi:` over `jdbc:hive2:`, please check all places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#issuecomment-4387736728
@wangzhigang1999 thanks for quickly following up, I will probably finish reviewing this patch this week, do you have a plan to supply the user docs for the data agent engine? if so, you can go ahead to write the docs, it will be the best material for reviewers and users to understand and adopt this new feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3317463172
##########
docs/quick_start/quick_start_with_data_agent.md:
##########
@@ -0,0 +1,211 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements. See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License. You may obtain a copy of the License at
+-
+- http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Data Agent Engine
+
+The Data Agent engine lets users explore and query data with natural language. It is a Kyuubi engine type (`DATA_AGENT`) that runs an LLM-driven [ReAct](https://arxiv.org/abs/2210.03629) agent. The agent reasons about the user's request, calls a small set of SQL tools against a JDBC-accessible backend (Spark, Trino, MySQL, SQLite, ...), and streams its responses to the bundled web UI.
+
+This page covers how to configure the engine, how the web UI behaves at runtime, the approval / risk model that gates SQL tool execution, and a few deployment notes worth knowing.
+
+## At a glance
+
+- **Engine type**: `DATA_AGENT`, share level follows `kyuubi.engine.share.level` like every other engine
+- **LLM provider**: any OpenAI-compatible chat-completion endpoint (OpenAI, Azure OpenAI, vLLM, llama.cpp, or any other provider that exposes the OpenAI chat-completion API)
+- **Datasource**: any JDBC URL the engine can reach. When unset, defaults to ZooKeeper service discovery for ZK-HA deployments, or `jdbc:hive2://localhost:<thrift-port>/default` otherwise
+- **Tools** (built in): `run_select_query` (low-risk, read-only) and `run_mutation_query` (high-risk, mutating). Per-dialect prompt templates ship for Spark / Trino / MySQL / SQLite / Generic
+- **Interface**: the bundled Kyuubi web UI. Pick **Data Agent** from the left navigation after logging in
+
+## Quick start
+
+### 1. Configure the LLM provider
+
+Add the following to `$KYUUBI_HOME/conf/kyuubi-defaults.conf`:
+
+```
+kyuubi.engine.data.agent.provider OPENAI_COMPATIBLE
+
+# OpenAI-compatible endpoint and credentials
+kyuubi.engine.data.agent.openai.endpoint https://api.openai.com/v1
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3317533691
##########
docs/quick_start/quick_start_with_data_agent.md:
##########
@@ -0,0 +1,211 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements. See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License. You may obtain a copy of the License at
+-
+- http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Data Agent Engine
+
+The Data Agent engine lets users explore and query data with natural language. It is a Kyuubi engine type (`DATA_AGENT`) that runs an LLM-driven [ReAct](https://arxiv.org/abs/2210.03629) agent. The agent reasons about the user's request, calls a small set of SQL tools against a JDBC-accessible backend (Spark, Trino, MySQL, SQLite, ...), and streams its responses to the bundled web UI.
+
+This page covers how to configure the engine, how the web UI behaves at runtime, the approval / risk model that gates SQL tool execution, and a few deployment notes worth knowing.
+
+## At a glance
+
+- **Engine type**: `DATA_AGENT`, share level follows `kyuubi.engine.share.level` like every other engine
+- **LLM provider**: any OpenAI-compatible chat-completion endpoint (OpenAI, Azure OpenAI, vLLM, llama.cpp, or any other provider that exposes the OpenAI chat-completion API)
+- **Datasource**: any JDBC URL the engine can reach. When unset, defaults to ZooKeeper service discovery for ZK-HA deployments, or `jdbc:hive2://localhost:<thrift-port>/default` otherwise
+- **Tools** (built in): `run_select_query` (low-risk, read-only) and `run_mutation_query` (high-risk, mutating). Per-dialect prompt templates ship for Spark / Trino / MySQL / SQLite / Generic
+- **Interface**: the bundled Kyuubi web UI. Pick **Data Agent** from the left navigation after logging in
+
+## Quick start
+
+### 1. Configure the LLM provider
+
+Add the following to `$KYUUBI_HOME/conf/kyuubi-defaults.conf`:
+
+```
+kyuubi.engine.data.agent.provider OPENAI_COMPATIBLE
+
+# OpenAI-compatible endpoint and credentials
+kyuubi.engine.data.agent.openai.endpoint https://api.openai.com/v1
+kyuubi.engine.data.agent.openai.api.key sk-xxxxxxxxxxxxxxxxxxxxxxxx
+kyuubi.engine.data.agent.model gpt-4o-mini
+```
+
+Substitute the endpoint, API key, and model ID for whichever OpenAI-compatible provider you use.
+
+### 2. Restart Kyuubi and open the UI
+
+```
+$KYUUBI_HOME/bin/kyuubi restart
+```
+
+Open `http://<kyuubi-host>:10099/ui` in a browser (the port comes from `kyuubi.frontend.rest.bind.port`, default `10099`). Pick **Data Agent** in the left navigation.
+
+### 3. Ask a question
Review Comment:
Yes, I tested the TPC-H connector provided by Kyuubi, which works very well for running demo queries. I have updated this section in the documentation and included two simple TPC-H-based examples.
<img width="2838" height="2112" alt="image" src="https://github.com/user-attachments/assets/abd860f5-c080-44a4-9148-65a58f67c040" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3468499740
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,18 +118,46 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // For Kyuubi/HiveServer2 backends, pass the session user for proxy-user impersonation —
+ // but only when the URL does not already carry explicit credentials, because URL-supplied
+ // credentials should win (some users pass credentials via session variables).
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
+ boolean isKyuubiBackend = "kyuubi".equals(subprotocol) || "hive2".equals(subprotocol);
+ String hikariUser = null;
+ if (isKyuubiBackend && !hasKyuubiUrlCredentials(jdbcUrl)) {
+ hikariUser = ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
+ }
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ DataSource ds = DataSourceFactory.create(jdbcUrl, hikariUser);
Review Comment:
I will add a commit to disable Kerberos as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3298383922
##########
docs/quick_start/quick_start_with_data_agent.md:
##########
@@ -0,0 +1,211 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements. See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License. You may obtain a copy of the License at
+-
+- http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Data Agent Engine
+
+The Data Agent engine lets users explore and query data with natural language. It is a Kyuubi engine type (`DATA_AGENT`) that runs an LLM-driven [ReAct](https://arxiv.org/abs/2210.03629) agent. The agent reasons about the user's request, calls a small set of SQL tools against a JDBC-accessible backend (Spark, Trino, MySQL, SQLite, ...), and streams its responses to the bundled web UI.
+
+This page covers how to configure the engine, how the web UI behaves at runtime, the approval / risk model that gates SQL tool execution, and a few deployment notes worth knowing.
+
+## At a glance
+
+- **Engine type**: `DATA_AGENT`, share level follows `kyuubi.engine.share.level` like every other engine
+- **LLM provider**: any OpenAI-compatible chat-completion endpoint (OpenAI, Azure OpenAI, vLLM, llama.cpp, or any other provider that exposes the OpenAI chat-completion API)
+- **Datasource**: any JDBC URL the engine can reach. When unset, defaults to ZooKeeper service discovery for ZK-HA deployments, or `jdbc:hive2://localhost:<thrift-port>/default` otherwise
+- **Tools** (built in): `run_select_query` (low-risk, read-only) and `run_mutation_query` (high-risk, mutating). Per-dialect prompt templates ship for Spark / Trino / MySQL / SQLite / Generic
+- **Interface**: the bundled Kyuubi web UI. Pick **Data Agent** from the left navigation after logging in
+
+## Quick start
+
+### 1. Configure the LLM provider
+
+Add the following to `$KYUUBI_HOME/conf/kyuubi-defaults.conf`:
+
+```
+kyuubi.engine.data.agent.provider OPENAI_COMPATIBLE
+
+# OpenAI-compatible endpoint and credentials
+kyuubi.engine.data.agent.openai.endpoint https://api.openai.com/v1
+kyuubi.engine.data.agent.openai.api.key sk-xxxxxxxxxxxxxxxxxxxxxxxx
+kyuubi.engine.data.agent.model gpt-4o-mini
+```
+
+Substitute the endpoint, API key, and model ID for whichever OpenAI-compatible provider you use.
+
+### 2. Restart Kyuubi and open the UI
+
+```
+$KYUUBI_HOME/bin/kyuubi restart
+```
+
+Open `http://<kyuubi-host>:10099/ui` in a browser (the port comes from `kyuubi.frontend.rest.bind.port`, default `10099`). Pick **Data Agent** in the left navigation.
+
+### 3. Ask a question
Review Comment:
is there a known dataset and question set that can be used for a demo? for example, kyuubi provides Spark TPC-H connector, which could provide the TPC-H dataset on the fly, is it suitable for the demo purpose?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3298331839
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,15 +118,29 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
-
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // The kyuubi session user is only meaningful when we connect back to Kyuubi/HiveServer2,
+ // where it drives proxy-user impersonation for the downstream engine. For external JDBC
+ // backends (mysql, postgresql, trino, ...) it would shadow URL-supplied credentials and
+ // cause auth failures (e.g. MySQL Connector/J overriding ?user=root with anonymous), so
+ // leave the username unset and let the driver read it from the URL.
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
Review Comment:
we have faced the same question on the JDBC engine, and the assumption may not always true, some users want to pass credentials using a session variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#issuecomment-4387687460
@pan3793 this is the final PR (4/4) for #7379, ready for review whenever you have time. Thanks for reviewing the earlier ones!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3317863680
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,15 +118,29 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
-
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // The kyuubi session user is only meaningful when we connect back to Kyuubi/HiveServer2,
+ // where it drives proxy-user impersonation for the downstream engine. For external JDBC
+ // backends (mysql, postgresql, trino, ...) it would shadow URL-supplied credentials and
+ // cause auth failures (e.g. MySQL Connector/J overriding ?user=root with anonymous), so
+ // leave the username unset and let the driver read it from the URL.
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
Review Comment:
Quick look confirms it — JDBC engine's `propagateCredential` is binary, and Trino's `connection.user` silently falls back to `sessionUser`. Same shape as data-agent. I'll note this in the follow-up issue so it's tracked across engines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#issuecomment-4380982201
<img width="3456" height="1926" alt="image" src="https://github.com/user-attachments/assets/9ad2b83f-d7bf-4012-957a-ae5fccd0befb" />
<img width="3456" height="1926" alt="image" src="https://github.com/user-attachments/assets/96383400-db67-4b40-b3f8-c12166514bf4" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#issuecomment-4393772697
Sure, no problem. I’ll start by running a comprehensive round of tests to cover as many scenarios as possible, and then draft the documentation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 closed pull request #7431: [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI
URL: https://github.com/apache/kyuubi/pull/7431
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3317492415
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,15 +118,29 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
-
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // The kyuubi session user is only meaningful when we connect back to Kyuubi/HiveServer2,
+ // where it drives proxy-user impersonation for the downstream engine. For external JDBC
+ // backends (mysql, postgresql, trino, ...) it would shadow URL-supplied credentials and
+ // cause auth failures (e.g. MySQL Connector/J overriding ?user=root with anonymous), so
+ // leave the username unset and let the driver read it from the URL.
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
Review Comment:
The change here is minimal: for Kyuubi/Hive2 backends, the session user is only injected when the URL does not carry an explicit `;user=` session variable, so `jdbc:kyuubi://host/db;user=alice;password=xxx` now flows through to the driver as-is.
The cleaner answer is explicit `jdbc.username` / `jdbc.password` config keys plus matching UI inputs (which would also keep passwords out of logs / JMX / browser history). I'll open a follow-up issue for that — out of scope here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3317616043
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,15 +118,29 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
-
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // The kyuubi session user is only meaningful when we connect back to Kyuubi/HiveServer2,
+ // where it drives proxy-user impersonation for the downstream engine. For external JDBC
+ // backends (mysql, postgresql, trino, ...) it would shadow URL-supplied credentials and
+ // cause auth failures (e.g. MySQL Connector/J overriding ?user=root with anonymous), so
+ // leave the username unset and let the driver read it from the URL.
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
Review Comment:
I prefer to handle this in another PR to avoid making the current PR too large.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3317638983
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,15 +118,29 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
-
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // The kyuubi session user is only meaningful when we connect back to Kyuubi/HiveServer2,
+ // where it drives proxy-user impersonation for the downstream engine. For external JDBC
+ // backends (mysql, postgresql, trino, ...) it would shadow URL-supplied credentials and
+ // cause auth failures (e.g. MySQL Connector/J overriding ?user=root with anonymous), so
+ // leave the username unset and let the driver read it from the URL.
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
Review Comment:
I think a similar issue exists for the JDBC and Trino engines, worth investigating too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3298383922
##########
docs/quick_start/quick_start_with_data_agent.md:
##########
@@ -0,0 +1,211 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements. See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License. You may obtain a copy of the License at
+-
+- http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Data Agent Engine
+
+The Data Agent engine lets users explore and query data with natural language. It is a Kyuubi engine type (`DATA_AGENT`) that runs an LLM-driven [ReAct](https://arxiv.org/abs/2210.03629) agent. The agent reasons about the user's request, calls a small set of SQL tools against a JDBC-accessible backend (Spark, Trino, MySQL, SQLite, ...), and streams its responses to the bundled web UI.
+
+This page covers how to configure the engine, how the web UI behaves at runtime, the approval / risk model that gates SQL tool execution, and a few deployment notes worth knowing.
+
+## At a glance
+
+- **Engine type**: `DATA_AGENT`, share level follows `kyuubi.engine.share.level` like every other engine
+- **LLM provider**: any OpenAI-compatible chat-completion endpoint (OpenAI, Azure OpenAI, vLLM, llama.cpp, or any other provider that exposes the OpenAI chat-completion API)
+- **Datasource**: any JDBC URL the engine can reach. When unset, defaults to ZooKeeper service discovery for ZK-HA deployments, or `jdbc:hive2://localhost:<thrift-port>/default` otherwise
+- **Tools** (built in): `run_select_query` (low-risk, read-only) and `run_mutation_query` (high-risk, mutating). Per-dialect prompt templates ship for Spark / Trino / MySQL / SQLite / Generic
+- **Interface**: the bundled Kyuubi web UI. Pick **Data Agent** from the left navigation after logging in
+
+## Quick start
+
+### 1. Configure the LLM provider
+
+Add the following to `$KYUUBI_HOME/conf/kyuubi-defaults.conf`:
+
+```
+kyuubi.engine.data.agent.provider OPENAI_COMPATIBLE
+
+# OpenAI-compatible endpoint and credentials
+kyuubi.engine.data.agent.openai.endpoint https://api.openai.com/v1
+kyuubi.engine.data.agent.openai.api.key sk-xxxxxxxxxxxxxxxxxxxxxxxx
+kyuubi.engine.data.agent.model gpt-4o-mini
+```
+
+Substitute the endpoint, API key, and model ID for whichever OpenAI-compatible provider you use.
+
+### 2. Restart Kyuubi and open the UI
+
+```
+$KYUUBI_HOME/bin/kyuubi restart
+```
+
+Open `http://<kyuubi-host>:10099/ui` in a browser (the port comes from `kyuubi.frontend.rest.bind.port`, default `10099`). Pick **Data Agent** in the left navigation.
+
+### 3. Ask a question
Review Comment:
is there a known dataset and question set that can be used for a demo? for example, kyuubi provides Spark TPC-H connector, which could generate the TPC-H dataset on the fly, is it suitable for the demo purpose?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#issuecomment-4829598792
Thank you; it has been merged into the main branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3298306698
##########
docs/quick_start/quick_start_with_data_agent.md:
##########
@@ -0,0 +1,211 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements. See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License. You may obtain a copy of the License at
+-
+- http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Data Agent Engine
+
+The Data Agent engine lets users explore and query data with natural language. It is a Kyuubi engine type (`DATA_AGENT`) that runs an LLM-driven [ReAct](https://arxiv.org/abs/2210.03629) agent. The agent reasons about the user's request, calls a small set of SQL tools against a JDBC-accessible backend (Spark, Trino, MySQL, SQLite, ...), and streams its responses to the bundled web UI.
+
+This page covers how to configure the engine, how the web UI behaves at runtime, the approval / risk model that gates SQL tool execution, and a few deployment notes worth knowing.
+
+## At a glance
+
+- **Engine type**: `DATA_AGENT`, share level follows `kyuubi.engine.share.level` like every other engine
+- **LLM provider**: any OpenAI-compatible chat-completion endpoint (OpenAI, Azure OpenAI, vLLM, llama.cpp, or any other provider that exposes the OpenAI chat-completion API)
+- **Datasource**: any JDBC URL the engine can reach. When unset, defaults to ZooKeeper service discovery for ZK-HA deployments, or `jdbc:hive2://localhost:<thrift-port>/default` otherwise
+- **Tools** (built in): `run_select_query` (low-risk, read-only) and `run_mutation_query` (high-risk, mutating). Per-dialect prompt templates ship for Spark / Trino / MySQL / SQLite / Generic
+- **Interface**: the bundled Kyuubi web UI. Pick **Data Agent** from the left navigation after logging in
+
+## Quick start
+
+### 1. Configure the LLM provider
+
+Add the following to `$KYUUBI_HOME/conf/kyuubi-defaults.conf`:
+
+```
+kyuubi.engine.data.agent.provider OPENAI_COMPATIBLE
+
+# OpenAI-compatible endpoint and credentials
+kyuubi.engine.data.agent.openai.endpoint https://api.openai.com/v1
+kyuubi.engine.data.agent.openai.api.key sk-xxxxxxxxxxxxxxxxxxxxxxxx
+kyuubi.engine.data.agent.model gpt-4o-mini
+```
+
+Substitute the endpoint, API key, and model ID for whichever OpenAI-compatible provider you use.
+
+### 2. Restart Kyuubi and open the UI
Review Comment:
Restart => Start
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3298352567
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/engine/EngineRef.scala:
##########
@@ -258,15 +259,28 @@ private[kyuubi] class EngineRef(
engineRefId,
extraEngineLog,
defaultEngineName)
+ case CHAT =>
+ new ChatProcessBuilder(appUser, doAsEnabled, conf, engineRefId, extraEngineLog)
case DATA_AGENT =>
if (conf.get(ENGINE_DATA_AGENT_JDBC_URL).isEmpty) {
val haAddresses = conf.get(HA_ADDRESSES)
- if (haAddresses.nonEmpty) {
- val jdbcUrl = s"jdbc:hive2://$haAddresses/default;" +
+ val isZkHa = haAddresses.nonEmpty &&
+ conf.get(HA_CLIENT_CLASS).endsWith("ZookeeperDiscoveryClient")
+ val jdbcUrl = if (isZkHa) {
+ s"jdbc:hive2://$haAddresses/default;" +
Review Comment:
prefer `jdbc:kyuubi:` over `jdbc:hive2:` if possible. Some users may add the vanilla Hive JDBC driver to the classpath, and `jdbc:hive2:` may match Hive's one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3317577525
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,15 +118,29 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
-
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // The kyuubi session user is only meaningful when we connect back to Kyuubi/HiveServer2,
+ // where it drives proxy-user impersonation for the downstream engine. For external JDBC
+ // backends (mysql, postgresql, trino, ...) it would shadow URL-supplied credentials and
+ // cause auth failures (e.g. MySQL Connector/J overriding ?user=root with anonymous), so
+ // leave the username unset and let the driver read it from the URL.
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
Review Comment:
Good point. I checked the JDBC engine pattern: it uses `propagateCredential` (default `false`) plus explicit `kyuubi.engine.jdbc.connection.user` / `.password`, passed via JDBC `Properties`.
For this PR I kept the fix minimal: for Kyuubi/Hive2 URLs, the session user is only passed when the URL has no explicit `;user=` session variable. So `jdbc:kyuubi://host/db;user=alice` is left for the driver to handle as-is.
I agree the complete fix should be a follow-up: Data Agent JDBC user/password configs plus a `propagateCredential` opt-in, with UI inputs so passwords do not need to be embedded in URLs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3317635354
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,15 +118,29 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
-
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // The kyuubi session user is only meaningful when we connect back to Kyuubi/HiveServer2,
+ // where it drives proxy-user impersonation for the downstream engine. For external JDBC
+ // backends (mysql, postgresql, trino, ...) it would shadow URL-supplied credentials and
+ // cause auth failures (e.g. MySQL Connector/J overriding ?user=root with anonymous), so
+ // leave the username unset and let the driver read it from the URL.
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
Review Comment:
sgtm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3467121108
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,18 +118,46 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // For Kyuubi/HiveServer2 backends, pass the session user for proxy-user impersonation —
+ // but only when the URL does not already carry explicit credentials, because URL-supplied
+ // credentials should win (some users pass credentials via session variables).
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
+ boolean isKyuubiBackend = "kyuubi".equals(subprotocol) || "hive2".equals(subprotocol);
+ String hikariUser = null;
+ if (isKyuubiBackend && !hasKyuubiUrlCredentials(jdbcUrl)) {
+ hikariUser = ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
+ }
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ DataSource ds = DataSourceFactory.create(jdbcUrl, hikariUser);
Review Comment:
@wangzhigang1999 as a temporary workaround, we can ban Kerberos auth to unblock this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#issuecomment-4484243339
@pan3793 gentle ping on this final Data Agent PR.
Current status:
- CI is green.
- User docs and screenshots were added in 1f0def8.
- Latest branch is mergeable and clean.
- This PR covers the REST/SSE layer and Web UI on top of the already-landed engine/runtime pieces.
Could you take another look when you have time? If the PR is still too large to review comfortably,I can split out docs/UI/server parts further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3317492415
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,15 +118,29 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
-
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // The kyuubi session user is only meaningful when we connect back to Kyuubi/HiveServer2,
+ // where it drives proxy-user impersonation for the downstream engine. For external JDBC
+ // backends (mysql, postgresql, trino, ...) it would shadow URL-supplied credentials and
+ // cause auth failures (e.g. MySQL Connector/J overriding ?user=root with anonymous), so
+ // leave the username unset and let the driver read it from the URL.
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
Review Comment:
The change here is minimal: for Kyuubi/Hive2 backends, the session user is only injected when the URL does not carry an explicit `;user=` session variable, so `jdbc:kyuubi://host/db;user=alice;password=xxx` now flows through to the driver as-is.
The cleaner answer is explicit `jdbc.username` / `jdbc.password` config keys plus matching UI inputs (which would also keep passwords out of logs / JMX / browser history). I'll open a follow-up issue for that — out of scope here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#discussion_r3387089810
##########
externals/kyuubi-data-agent-engine/src/main/java/org/apache/kyuubi/engine/dataagent/provider/chatcompletion/ChatCompletionProvider.java:
##########
@@ -118,18 +118,46 @@ private static DataSource attachJdbcDataSource(
if (jdbcUrl == null) {
return null;
}
- LOG.info("Data Agent JDBC URL configured ({})", jdbcUrl.replaceAll("//.*@", "//<redacted>@"));
-
- String sessionUser =
- ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
+ JdbcDialect dialect = JdbcDialect.fromUrl(jdbcUrl);
+ if (dialect == null) {
+ throw new IllegalArgumentException(
+ "Invalid " + KyuubiConf.ENGINE_DATA_AGENT_JDBC_URL().key());
+ }
+ LOG.info("Data Agent JDBC datasource configured ({})", dialect.datasourceName());
+
+ // For Kyuubi/HiveServer2 backends, pass the session user for proxy-user impersonation —
+ // but only when the URL does not already carry explicit credentials, because URL-supplied
+ // credentials should win (some users pass credentials via session variables).
+ String subprotocol = JdbcDialect.extractSubprotocol(jdbcUrl);
+ boolean isKyuubiBackend = "kyuubi".equals(subprotocol) || "hive2".equals(subprotocol);
+ String hikariUser = null;
+ if (isKyuubiBackend && !hasKyuubiUrlCredentials(jdbcUrl)) {
+ hikariUser = ConfUtils.optionalString(conf, KyuubiReservedKeys.KYUUBI_SESSION_USER_KEY());
+ }
- DataSource ds = DataSourceFactory.create(jdbcUrl, sessionUser);
+ DataSource ds = DataSourceFactory.create(jdbcUrl, hikariUser);
Review Comment:
for kerberized cases, the engine process runs under kyuubi server's host and same user, then the super user's TGT (allows proxying any users) is visible to this process, so this would lead to arbitrary user impersonation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7379][4/4] Data Agent Engine: REST API, SSE streaming, and Web UI [kyuubi]
Posted by "wangzhigang1999 (via GitHub)" <gi...@apache.org>.
wangzhigang1999 commented on PR #7431:
URL: https://github.com/apache/kyuubi/pull/7431#issuecomment-4668512947
@pan3793 gentle follow-up when you have bandwidth. I have addressed the previous review comments, CI is green, and all conversations appear resolved. Please let me know if there is anything else I should update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org