You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by ga...@apache.org on 2026/10/16 09:19:00 UTC
(doris) branch master updated: [improvement](datasource) Include remote JDBC error diagnostics (#68811)
This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 8b96a8b4a47 [improvement](datasource) Include remote JDBC error diagnostics (#68811)
8b96a8b4a47 is described below
commit 8b96a8b4a472ec0e4d24b01baa00664016dd0cfd
Author: Oliveira <43...@users.noreply.github.com>
AuthorDate: Fri Oct 16 17:18:50 2026 +0800
[improvement](datasource) Include remote JDBC error diagnostics (#68811)
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary: JDBC schema lookup, scanning, and write failures
previously reported incomplete diagnostics: remote SQLSTATE and vendor
error codes were lost across FE catalog paths, delegated clients, and BE
Java scanner/writer wrappers, while repeated wrapping could duplicate or
corrupt messages. This change traverses SQLException cause and
nextException graphs with cycle detection and deduplication, carries
diagnostics through a neutral connector SPI, preserves original causes
and Doris error codes, and consistently redacts configured secrets and
JDBC URLs in summaries and JNI stack traces. Existing metadata fallbacks
and connector lifecycle behavior remain intact.
### Release note
JDBC errors now expose separate remote_sqlstate and
remote_vendor_error_code fields while preserving Doris error-code
semantics. The FE connector SPI is now API 13.0 and the JNI plugin SPI
is now API 5.0; external connector and JNI plugins must be rebuilt
against the updated APIs.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [x] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Validation details:
- 59 focused Java unit tests in 14 test classes passed with
`run-fe-ut.sh`.
- Standard ASAN FE/BE build with `build.sh --fe --be -j96` passed,
including Checkstyle and all nine JNI plugin layout checks.
- `test_jdbc_error_diagnostics` passed against MySQL 8 with
`run-regression-test.sh -d external_table_p0/jdbc -s
test_jdbc_error_diagnostics` (one suite, no failures or skips).
- MySQL protocol verification retained Doris ERROR 1105 / errCode=2
alongside remote SQLSTATE 42000 / vendor code 1142, emitted once without
the configured password.
- Behavior changed:
- [ ] No.
- [x] Yes. JDBC diagnostics now retain remote codes and sanitize
messages consistently across FE and BE wrappers.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
.../apache/doris/jdbc/JdbcConnectionTester.java | 11 +-
.../java/org/apache/doris/jdbc/JdbcDataSource.java | 19 +-
.../java/org/apache/doris/jdbc/JdbcJniScanner.java | 21 +-
.../java/org/apache/doris/jdbc/JdbcJniWriter.java | 16 +-
.../apache/doris/jdbc/JdbcTypeHandlerFactory.java | 6 +-
.../org/apache/doris/jdbc/OracleTypeHandler.java | 17 +-
.../jdbc/JdbcConnectionLogDiagnosticsTest.java | 195 ++++++++++++++++
.../org/apache/doris/jdbc/JdbcDiagnosticsTest.java | 197 ++++++++++++++++
.../apache/doris/jdbc/OracleDiagnosticsTest.java | 143 ++++++++++++
.../apache/doris/jni/spi/DiagnosticException.java | 27 +++
.../org/apache/doris/jni/spi/utils/JniUtil.java | 17 +-
.../apache/doris/jni/spi/JniPluginSurfaceTest.java | 2 +-
.../apache/doris/jni/spi/utils/JniUtilTest.java | 41 ++++
.../src/test/resources/jni-plugin-surface.txt | 2 +
.../doris/jni/toolkit/jdbc/JdbcExceptionUtils.java | 236 +++++++++++++++++++
.../doris/jni/toolkit/jdbc/JdbcIOException.java | 37 +++
.../jni/toolkit/jdbc/JdbcExceptionUtilsTest.java | 253 +++++++++++++++++++++
fe/be-java-extensions/pom.xml | 2 +-
fe/fe-connector/fe-connector-jdbc/pom.xml | 7 +
.../doris/connector/jdbc/JdbcDorisConnector.java | 10 +-
.../jdbc/client/JdbcClickHouseConnectorClient.java | 5 +-
.../connector/jdbc/client/JdbcConnectorClient.java | 68 +++---
.../jdbc/client/JdbcDB2ConnectorClient.java | 3 +-
.../jdbc/client/JdbcDiagnosticException.java | 42 ++++
.../jdbc/client/JdbcGbaseConnectorClient.java | 7 +-
.../jdbc/client/JdbcMySQLConnectorClient.java | 16 +-
.../jdbc/client/JdbcOceanBaseConnectorClient.java | 7 +-
.../jdbc/client/JdbcOracleConnectorClient.java | 4 +-
.../jdbc/client/JdbcPostgreSQLConnectorClient.java | 5 +-
.../jdbc/client/JdbcSQLServerConnectorClient.java | 2 +-
.../jdbc/JdbcDorisConnectorDiagnosticsTest.java | 123 ++++++++++
.../jdbc/client/JdbcConnectorDiagnosticsTest.java | 86 +++++++
.../jdbc/client/JdbcDelegatedDiagnosticsTest.java | 162 +++++++++++++
.../doris/connector/spi/DiagnosticException.java | 30 +++
.../connector/spi/ConnectorPluginSurfaceTest.java | 5 +-
.../test/resources/connector-plugin-surface.txt | 2 +
fe/fe-connector/pom.xml | 2 +-
fe/fe-core/pom.xml | 7 +
.../common/util/ExceptionDiagnosticUtils.java | 71 ++++++
.../org/apache/doris/datasource/CatalogIf.java | 8 +-
.../apache/doris/datasource/ExternalCatalog.java | 59 ++++-
.../jdbc/client/JdbcClickHouseClient.java | 8 +-
.../doris/datasource/jdbc/client/JdbcClient.java | 55 +++--
.../jdbc/client/JdbcClientException.java | 52 +++--
.../datasource/jdbc/client/JdbcDB2Client.java | 2 +-
.../datasource/jdbc/client/JdbcGbaseClient.java | 10 +-
.../datasource/jdbc/client/JdbcMySQLClient.java | 22 +-
.../jdbc/client/JdbcOceanBaseClient.java | 2 +-
.../datasource/jdbc/client/JdbcOracleClient.java | 5 +-
.../jdbc/client/JdbcPostgreSQLClient.java | 10 +-
.../streaming/DataSourceConfigValidator.java | 8 +-
.../insert/streaming/StreamingInsertJob.java | 12 +-
.../org/apache/doris/qe/ConnectContextUtil.java | 4 +-
.../java/org/apache/doris/qe/StmtExecutor.java | 20 +-
.../datasource/JdbcCatalogDiagnosticsTest.java | 144 ++++++++++++
.../jdbc/client/JdbcClientExceptionTest.java | 8 +
.../jdbc/client/JdbcLegacyDiagnosticsTest.java | 110 +++++++++
.../doris/qe/JdbcFinalOutputDiagnosticsTest.java | 169 ++++++++++++++
.../jdbc/test_jdbc_error_diagnostics.out | 7 +
.../jdbc/test_jdbc_error_diagnostics.groovy | 93 ++++++++
60 files changed, 2530 insertions(+), 184 deletions(-)
diff --git a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcConnectionTester.java b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcConnectionTester.java
index 96d9286caad..4d00cf5ee63 100644
--- a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcConnectionTester.java
+++ b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcConnectionTester.java
@@ -20,6 +20,8 @@ package org.apache.doris.jdbc;
import org.apache.doris.jni.spi.JniScanner;
import org.apache.doris.jni.spi.vec.ColumnType;
import org.apache.doris.jni.toolkit.jdbc.JdbcDriverUtils;
+import org.apache.doris.jni.toolkit.jdbc.JdbcExceptionUtils;
+import org.apache.doris.jni.toolkit.jdbc.JdbcIOException;
import com.zaxxer.hikari.HikariDataSource;
import org.slf4j.Logger;
@@ -158,9 +160,10 @@ public class JdbcConnectionTester extends JniScanner {
"Failed to test connection: query executed but returned no results.");
}
rs.close();
- LOG.info("JdbcConnectionTester: connection test succeeded for " + jdbcUrl);
+ String safeJdbcUrl = JdbcExceptionUtils.redact(jdbcUrl, jdbcPassword, jdbcUrl);
+ LOG.info("JdbcConnectionTester: connection test succeeded for " + safeJdbcUrl);
} catch (Exception e) {
- throw new IOException("Failed to test JDBC connection: " + e.getMessage(), e);
+ throw new JdbcIOException("Failed to test JDBC connection", e, jdbcPassword, jdbcUrl);
} finally {
Thread.currentThread().setContextClassLoader(oldClassLoader);
}
@@ -184,7 +187,9 @@ public class JdbcConnectionTester extends JniScanner {
conn.close();
}
} catch (SQLException e) {
- LOG.warn("JdbcConnectionTester close error: " + e.getMessage(), e);
+ String diagnostic = JdbcExceptionUtils.format("JdbcConnectionTester close error",
+ e, jdbcPassword, jdbcUrl);
+ LOG.warn("{}", diagnostic);
} finally {
stmt = null;
conn = null;
diff --git a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcDataSource.java b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcDataSource.java
index b4959aef742..0352818d2c9 100644
--- a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcDataSource.java
+++ b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcDataSource.java
@@ -22,8 +22,11 @@ import com.zaxxer.hikari.HikariDataSource;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.util.Collections;
+import java.util.IdentityHashMap;
import java.util.Iterator;
import java.util.Map;
+import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
@@ -101,12 +104,26 @@ public class JdbcDataSource {
+ ", remaining: " + sourcesMap.size());
}
} catch (Exception e) {
- LOG.warn("failed to cleanup jdbc data source", e);
+ LOG.warn("failed to cleanup jdbc data source: {}", cleanupErrorStackTrace(e));
}
}, cleanupInterval, cleanupInterval, TimeUnit.MILLISECONDS);
LOG.info("jdbc datasource cleanup task started, interval: " + cleanupInterval + "ms");
}
+ // Cleanup can fail outside an individual pool, where no credential policy is available.
+ // Keep exception types and locations without rendering driver-supplied messages.
+ static String cleanupErrorStackTrace(Throwable error) {
+ StringBuilder trace = new StringBuilder();
+ Set<Throwable> visited = Collections.newSetFromMap(new IdentityHashMap<>());
+ for (Throwable current = error; current != null && visited.add(current); current = current.getCause()) {
+ trace.append(current.getClass().getName()).append('\n');
+ for (StackTraceElement frame : current.getStackTrace()) {
+ trace.append("\tat ").append(frame).append('\n');
+ }
+ }
+ return trace.toString();
+ }
+
private synchronized void startCleanupTask() {
if (cleanupTask == null || cleanupTask.isCancelled()) {
restartCleanupTask();
diff --git a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniScanner.java b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniScanner.java
index a831d1656e1..7ed832e5ff0 100644
--- a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniScanner.java
+++ b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniScanner.java
@@ -21,6 +21,8 @@ import org.apache.doris.jni.spi.JniScanner;
import org.apache.doris.jni.spi.vec.ColumnType;
import org.apache.doris.jni.spi.vec.ColumnValueConverter;
import org.apache.doris.jni.toolkit.jdbc.JdbcDriverUtils;
+import org.apache.doris.jni.toolkit.jdbc.JdbcExceptionUtils;
+import org.apache.doris.jni.toolkit.jdbc.JdbcIOException;
import com.zaxxer.hikari.HikariDataSource;
import org.slf4j.Logger;
@@ -130,7 +132,7 @@ public class JdbcJniScanner extends JniScanner {
// Select database-specific type handler
String tableType = params.getOrDefault("table_type", "");
- this.typeHandler = JdbcTypeHandlerFactory.create(tableType);
+ this.typeHandler = JdbcTypeHandlerFactory.create(tableType, jdbcPassword, jdbcUrl);
String requiredFields = params.getOrDefault("required_fields", "");
String columnsTypes = params.getOrDefault("columns_types", "");
@@ -211,8 +213,7 @@ public class JdbcJniScanner extends JniScanner {
outputConverters[i] = typeHandler.getOutputConverter(types[i], replaceStr);
}
} catch (Exception e) {
- LOG.warn("JdbcJniScanner " + jdbcUrl + " open failed: " + e.getMessage(), e);
- throw new IOException("JdbcJniScanner open failed: " + e.getMessage(), e);
+ throw new JdbcIOException("JdbcJniScanner open failed", e, jdbcPassword, jdbcUrl);
} finally {
Thread.currentThread().setContextClassLoader(oldClassLoader);
}
@@ -294,8 +295,7 @@ public class JdbcJniScanner extends JniScanner {
readRows += curRows;
return curRows;
} catch (Exception e) {
- LOG.warn("JdbcJniScanner getNext failed: " + e.getMessage(), e);
- throw new IOException("JdbcJniScanner getNext failed: " + e.getMessage(), e);
+ throw new JdbcIOException("JdbcJniScanner getNext failed", e, jdbcPassword, jdbcUrl);
}
}
@@ -307,7 +307,9 @@ public class JdbcJniScanner extends JniScanner {
typeHandler.abortReadConnection(conn, resultSet);
}
} catch (Exception e) {
- LOG.warn("JdbcJniScanner abort connection error: " + e.getMessage(), e);
+ String diagnostic = JdbcExceptionUtils.format("JdbcJniScanner abort connection error",
+ e, jdbcPassword, jdbcUrl);
+ LOG.warn("{}", diagnostic);
}
try {
if (resultSet != null && !resultSet.isClosed()) {
@@ -320,7 +322,9 @@ public class JdbcJniScanner extends JniScanner {
conn.close();
}
} catch (Exception e) {
- LOG.warn("JdbcJniScanner close error: " + e.getMessage(), e);
+ String diagnostic = JdbcExceptionUtils.format("JdbcJniScanner close error",
+ e, jdbcPassword, jdbcUrl);
+ LOG.warn("{}", diagnostic);
} finally {
resultSet = null;
stmt = null;
@@ -387,7 +391,8 @@ public class JdbcJniScanner extends JniScanner {
}
hikariDataSource = ds;
JdbcDataSource.getDataSource().putSource(cacheKey, hikariDataSource);
- LOG.info("JdbcJniScanner: Created connection pool for " + jdbcUrl);
+ String safeJdbcUrl = JdbcExceptionUtils.redact(jdbcUrl, jdbcPassword, jdbcUrl);
+ LOG.info("JdbcJniScanner: Created connection pool for " + safeJdbcUrl);
}
}
}
diff --git a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniWriter.java b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniWriter.java
index 093a423e3c4..9122e98880c 100644
--- a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniWriter.java
+++ b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcJniWriter.java
@@ -22,6 +22,8 @@ import org.apache.doris.jni.spi.vec.ColumnType;
import org.apache.doris.jni.spi.vec.VectorColumn;
import org.apache.doris.jni.spi.vec.VectorTable;
import org.apache.doris.jni.toolkit.jdbc.JdbcDriverUtils;
+import org.apache.doris.jni.toolkit.jdbc.JdbcExceptionUtils;
+import org.apache.doris.jni.toolkit.jdbc.JdbcIOException;
import com.zaxxer.hikari.HikariDataSource;
import org.slf4j.Logger;
@@ -136,7 +138,7 @@ public class JdbcJniWriter extends JniWriter {
LOG.debug("JdbcJniWriter: Preparing insert statement: " + insertSql);
preparedStatement = conn.prepareStatement(insertSql);
} catch (Exception e) {
- throw new IOException("JdbcJniWriter open failed: " + e.getMessage(), e);
+ throw new JdbcIOException("JdbcJniWriter open failed", e, jdbcPassword, jdbcUrl);
} finally {
Thread.currentThread().setContextClassLoader(oldClassLoader);
}
@@ -161,7 +163,7 @@ public class JdbcJniWriter extends JniWriter {
insertTime += System.nanoTime() - startInsert;
writtenRows += numRows;
} catch (SQLException e) {
- throw new IOException("JdbcJniWriter write failed: " + e.getMessage(), e);
+ throw new JdbcIOException("JdbcJniWriter write failed", e, jdbcPassword, jdbcUrl);
}
}
@@ -187,10 +189,11 @@ public class JdbcJniWriter extends JniWriter {
conn.rollback();
}
} catch (SQLException rollbackEx) {
- LOG.warn("JdbcJniWriter rollback on close failure also failed: "
- + rollbackEx.getMessage(), rollbackEx);
+ String diagnostic = JdbcExceptionUtils.format("JdbcJniWriter rollback on close failure also failed",
+ rollbackEx, jdbcPassword, jdbcUrl);
+ LOG.warn("{}", diagnostic);
}
- throw new IOException("JdbcJniWriter close failed: " + e.getMessage(), e);
+ throw new JdbcIOException("JdbcJniWriter close failed", e, jdbcPassword, jdbcUrl);
} finally {
preparedStatement = null;
conn = null;
@@ -374,7 +377,8 @@ public class JdbcJniWriter extends JniWriter {
}
hikariDataSource = ds;
JdbcDataSource.getDataSource().putSource(cacheKey, hikariDataSource);
- LOG.info("JdbcJniWriter: Created connection pool for " + jdbcUrl);
+ String safeJdbcUrl = JdbcExceptionUtils.redact(jdbcUrl, jdbcPassword, jdbcUrl);
+ LOG.info("JdbcJniWriter: Created connection pool for " + safeJdbcUrl);
}
}
}
diff --git a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcTypeHandlerFactory.java b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcTypeHandlerFactory.java
index e1a5f1356c2..549a4215d47 100644
--- a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcTypeHandlerFactory.java
+++ b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/JdbcTypeHandlerFactory.java
@@ -29,6 +29,10 @@ public class JdbcTypeHandlerFactory {
* @return the appropriate type handler
*/
public static JdbcTypeHandler create(String tableType) {
+ return create(tableType, new String[0]);
+ }
+
+ public static JdbcTypeHandler create(String tableType, String... sensitiveValues) {
if (tableType == null || tableType.isEmpty()) {
return new DefaultTypeHandler();
}
@@ -38,7 +42,7 @@ public class JdbcTypeHandlerFactory {
return new MySQLTypeHandler(tableType);
case "ORACLE":
case "OCEANBASE_ORACLE":
- return new OracleTypeHandler();
+ return new OracleTypeHandler(sensitiveValues);
case "POSTGRESQL":
return new PostgreSQLTypeHandler();
case "CLICKHOUSE":
diff --git a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/OracleTypeHandler.java b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/OracleTypeHandler.java
index 8fcca077fc2..28f326d46fb 100644
--- a/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/OracleTypeHandler.java
+++ b/fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/OracleTypeHandler.java
@@ -19,6 +19,7 @@ package org.apache.doris.jdbc;
import org.apache.doris.jni.spi.vec.ColumnType;
import org.apache.doris.jni.spi.vec.ColumnValueConverter;
+import org.apache.doris.jni.toolkit.jdbc.JdbcExceptionUtils;
import com.zaxxer.hikari.HikariDataSource;
import org.slf4j.Logger;
@@ -57,6 +58,16 @@ public class OracleTypeHandler extends DefaultTypeHandler {
java.time.format.DateTimeFormatter.ofPattern("uuuu-MM-dd HH:mm:ss.SSSSSS");
private static final Logger LOG = LoggerFactory.getLogger(OracleTypeHandler.class);
+ private final String[] sensitiveValues;
+
+ public OracleTypeHandler() {
+ this(new String[0]);
+ }
+
+ public OracleTypeHandler(String... sensitiveValues) {
+ this.sensitiveValues = sensitiveValues.clone();
+ }
+
// Whether the JDBC driver supports JDBC 4.1 getObject(int, Class) method.
// Determined at runtime from the driver version. Oracle ojdbc6 (< 12.2.0) does not.
private boolean jdbc41Supported = true;
@@ -81,7 +92,8 @@ public class OracleTypeHandler extends DefaultTypeHandler {
+ ", JDBC 4.1 supported: " + jdbc41Supported);
}
} catch (SQLException e) {
- LOG.warn("Failed to detect Oracle driver version, assuming JDBC 4.1: " + e.getMessage());
+ LOG.warn("Failed to detect Oracle driver version, assuming JDBC 4.1: {}",
+ JdbcExceptionUtils.format("", e, sensitiveValues));
}
versionDetected = true;
}
@@ -271,7 +283,8 @@ public class OracleTypeHandler extends DefaultTypeHandler {
try {
return ((Clob) input).getSubString(1, (int) ((Clob) input).length());
} catch (SQLException e) {
- LOG.error("Failed to get string from clob", e);
+ LOG.error("Failed to get string from clob: {}",
+ JdbcExceptionUtils.format("", e, sensitiveValues));
return null;
}
} else if (input instanceof byte[]) {
diff --git a/fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/JdbcConnectionLogDiagnosticsTest.java b/fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/JdbcConnectionLogDiagnosticsTest.java
new file mode 100644
index 00000000000..c7f70a5ad77
--- /dev/null
+++ b/fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/JdbcConnectionLogDiagnosticsTest.java
@@ -0,0 +1,195 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jdbc;
+
+import com.zaxxer.hikari.HikariDataSource;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.core.LogEvent;
+import org.apache.logging.log4j.core.Logger;
+import org.apache.logging.log4j.core.appender.AbstractAppender;
+import org.apache.logging.log4j.core.config.Property;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import java.lang.reflect.Method;
+import java.sql.Connection;
+import java.sql.Driver;
+import java.sql.DriverPropertyInfo;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.logging.Handler;
+import java.util.logging.LogRecord;
+
+class JdbcConnectionLogDiagnosticsTest {
+ @Test
+ void scannerWriterAndTesterMaskCompleteEscapedSqlServerUrl() throws Exception {
+ String password = "two\";words";
+ String url = "jdbc:sqlserver://host;password={" + password + "};databaseName=db";
+ for (Class<?> type : new Class<?>[] {JdbcJniScanner.class, JdbcJniWriter.class, JdbcConnectionTester.class}) {
+ Map<String, String> params = new HashMap<>();
+ params.put("table_type", "SQLSERVER");
+ params.put("jdbc_url", url);
+ params.put("jdbc_password", password);
+ params.put("catalog_id", "68811");
+ params.put("jdbc_driver_class", StubDriver.class.getName());
+ params.put("jdbc_driver_url", StubDriver.class.getProtectionDomain().getCodeSource()
+ .getLocation().toURI().toString());
+ Object target = type.getConstructor(int.class, Map.class).newInstance(1, params);
+ Method cacheKey = type.getDeclaredMethod("createCacheKey");
+ cacheKey.setAccessible(true);
+ String key = (String) cacheKey.invoke(target);
+ Connection connection = Mockito.mock(Connection.class);
+ PreparedStatement statement = Mockito.mock(PreparedStatement.class);
+ ResultSet result = Mockito.mock(ResultSet.class);
+ Mockito.when(connection.prepareStatement(Mockito.anyString())).thenReturn(statement);
+ Mockito.when(statement.executeQuery()).thenReturn(result);
+ Mockito.when(result.next()).thenReturn(true);
+ ClassLoader original = Thread.currentThread().getContextClassLoader();
+ if (type == JdbcConnectionTester.class) {
+ JdbcDataSource.getDataSource().putSource(key, new HikariDataSource() {
+ @Override
+ public Connection getConnection() {
+ return connection;
+ }
+ });
+ }
+ try (CapturedLog capture = new CapturedLog(type)) {
+ Method initialize = type.getDeclaredMethod(type == JdbcConnectionTester.class
+ ? "openInternal" : "initializeClassLoaderAndDataSource");
+ initialize.setAccessible(true);
+ initialize.invoke(target);
+ Assertions.assertNotNull(JdbcDataSource.getDataSource().getSource(key));
+ Assertions.assertEquals(1, capture.messages.size(), capture.messages.toString());
+ String text = capture.messages.get(0);
+ Assertions.assertTrue(text.contains(type.getSimpleName()), text);
+ Assertions.assertTrue(text.contains("[redacted JDBC URL]"), text);
+ Assertions.assertFalse(text.contains(url), text);
+ Assertions.assertFalse(text.contains(password), text);
+ Assertions.assertFalse(text.contains("words}"), text);
+ } finally {
+ try {
+ // Initialization can fail before the pool is added to the cache.
+ HikariDataSource pool = JdbcDataSource.getDataSource().getSourcesMap().remove(key);
+ if (pool != null) {
+ pool.close();
+ }
+ } finally {
+ Thread.currentThread().setContextClassLoader(original);
+ }
+ }
+ }
+ }
+
+ /** Lets real Hikari configuration load a driver without opening any external connection. */
+ public static class StubDriver implements Driver {
+ @Override
+ public Connection connect(String url, Properties info) throws SQLException {
+ throw new SQLException("This test driver must not open a connection");
+ }
+
+ @Override
+ public boolean acceptsURL(String url) {
+ return url.startsWith("jdbc:sqlserver:");
+ }
+
+ @Override
+ public DriverPropertyInfo[] getPropertyInfo(String url, Properties info) {
+ return new DriverPropertyInfo[0];
+ }
+
+ @Override
+ public int getMajorVersion() {
+ return 1;
+ }
+
+ @Override
+ public int getMinorVersion() {
+ return 0;
+ }
+
+ @Override
+ public boolean jdbcCompliant() {
+ return false;
+ }
+
+ @Override
+ public java.util.logging.Logger getParentLogger() {
+ return java.util.logging.Logger.getGlobal();
+ }
+ }
+
+ // Production uses JUL; tests may select the inherited Log4j SLF4J provider instead.
+ private static class CapturedLog implements AutoCloseable {
+ private final List<String> messages = new CopyOnWriteArrayList<>();
+ private final java.util.logging.Logger jul;
+ private final java.util.logging.Level previousJulLevel;
+ private final Logger log4j;
+ private final Level previousLevel;
+ private final Handler handler = new Handler() {
+ @Override
+ public void publish(LogRecord record) {
+ messages.add(record.getMessage());
+ }
+
+ @Override
+ public void flush() {
+ }
+
+ @Override
+ public void close() {
+ }
+ };
+ private final AbstractAppender appender = new AbstractAppender("jdbc-success-test", null,
+ null, true, Property.EMPTY_ARRAY) {
+ @Override
+ public void append(LogEvent event) {
+ messages.add(event.getMessage().getFormattedMessage());
+ }
+ };
+
+ CapturedLog(Class<?> type) {
+ jul = java.util.logging.Logger.getLogger(type.getName());
+ previousJulLevel = jul.getLevel();
+ log4j = (Logger) LogManager.getLogger(type);
+ previousLevel = log4j.getLevel();
+ jul.setLevel(java.util.logging.Level.ALL);
+ jul.addHandler(handler);
+ appender.start();
+ log4j.addAppender(appender);
+ log4j.setLevel(Level.ALL);
+ }
+
+ @Override
+ public void close() {
+ log4j.removeAppender(appender);
+ log4j.setLevel(previousLevel);
+ appender.stop();
+ jul.removeHandler(handler);
+ jul.setLevel(previousJulLevel);
+ handler.close();
+ }
+ }
+}
diff --git a/fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/JdbcDiagnosticsTest.java b/fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/JdbcDiagnosticsTest.java
new file mode 100644
index 00000000000..a47ae247f33
--- /dev/null
+++ b/fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/JdbcDiagnosticsTest.java
@@ -0,0 +1,197 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jdbc;
+
+import org.apache.doris.jni.spi.utils.JniUtil;
+import org.apache.doris.jni.spi.utils.OffHeap;
+import org.apache.doris.jni.spi.vec.ColumnType;
+import org.apache.doris.jni.spi.vec.VectorTable;
+
+import com.zaxxer.hikari.HikariDataSource;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.lang.reflect.InvocationHandler;
+import java.lang.reflect.Method;
+import java.lang.reflect.Proxy;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Map;
+
+class JdbcDiagnosticsTest {
+ private static SQLException failure() {
+ SQLException error = new SQLException("denied; raw=test-password", "42000", 1142);
+ error.setNextException(new SQLException("connection lost", "08006", 0));
+ return error;
+ }
+
+ @Test
+ void cleanupFailureKeepsTypesAndLocationsWithoutDriverMessages() {
+ SQLException cause = new SQLException("raw=test-password jdbc:mysql://private/db", "08001", 0);
+ RuntimeException error = new RuntimeException("outer raw=test-password", cause);
+ String trace = JdbcDataSource.cleanupErrorStackTrace(error);
+ Assertions.assertTrue(trace.contains("java.lang.RuntimeException"));
+ Assertions.assertTrue(trace.contains("java.sql.SQLException"));
+ Assertions.assertTrue(trace.contains("cleanupFailureKeepsTypesAndLocationsWithoutDriverMessages"));
+ Assertions.assertFalse(trace.contains("test-password"));
+ Assertions.assertFalse(trace.contains("jdbc:mysql"));
+ cause.initCause(error);
+ Assertions.assertEquals(trace, JdbcDataSource.cleanupErrorStackTrace(error));
+ }
+
+ @Test
+ void scannerOpenReportsRemoteCodesThroughTheActualJniFormatter() throws Exception {
+ JdbcJniScanner scanner = new JdbcJniScanner(1, params());
+ SQLException cause = failure();
+ HikariDataSource pool = new HikariDataSource() {
+ @Override
+ public Connection getConnection() throws SQLException {
+ throw cause;
+ }
+ };
+ Method cacheKey = JdbcJniScanner.class.getDeclaredMethod("createCacheKey");
+ cacheKey.setAccessible(true);
+ String key = (String) cacheKey.invoke(scanner);
+ JdbcDataSource.getDataSource().putSource(key, pool);
+ try {
+ assertDiagnostic(Assertions.assertThrows(IOException.class, scanner::open), cause, "open");
+ } finally {
+ JdbcDataSource.getDataSource().getSourcesMap().remove(key);
+ pool.close();
+ }
+ }
+
+ @Test
+ void scannerGetNextReportsStreamingResultSetFailure() throws Exception {
+ JdbcJniScanner scanner = new JdbcJniScanner(1, params());
+ SQLException cause = failure();
+ ResultSet resultSet = proxy(ResultSet.class, (object, method, args) -> {
+ if (method.getName().equals("next")) {
+ throw cause;
+ }
+ throw new UnsupportedOperationException(method.getName());
+ });
+ field(scanner, "resultSet", resultSet);
+ field(scanner, "resultSetOpened", true);
+ field(scanner, "block", new ArrayList<>());
+ assertDiagnostic(Assertions.assertThrows(IOException.class, scanner::getNext), cause, "getNext");
+ }
+
+ @Test
+ void writerOpenReportsConnectionFailure() throws Exception {
+ JdbcJniWriter writer = new JdbcJniWriter(1, params());
+ SQLException cause = failure();
+ HikariDataSource pool = new HikariDataSource() {
+ @Override
+ public Connection getConnection() throws SQLException {
+ throw cause;
+ }
+ };
+ Method cacheKey = JdbcJniWriter.class.getDeclaredMethod("createCacheKey");
+ cacheKey.setAccessible(true);
+ String key = (String) cacheKey.invoke(writer);
+ JdbcDataSource.getDataSource().putSource(key, pool);
+ try {
+ assertDiagnostic(Assertions.assertThrows(IOException.class, writer::open), cause, "open");
+ } finally {
+ JdbcDataSource.getDataSource().getSourcesMap().remove(key);
+ pool.close();
+ }
+ }
+
+ @Test
+ void writerReportsBatchExecutionFailure() throws Exception {
+ JdbcJniWriter writer = new JdbcJniWriter(1, params());
+ SQLException cause = failure();
+ PreparedStatement statement = proxy(PreparedStatement.class, (object, method, args) -> {
+ if (method.getName().equals("executeBatch")) {
+ throw cause;
+ }
+ throw new UnsupportedOperationException(method.getName());
+ });
+ field(writer, "preparedStatement", statement);
+ OffHeap.setTesting();
+ VectorTable table = VectorTable.createWritableTable(new ColumnType[0], new String[0], 1);
+ try {
+ assertDiagnostic(Assertions.assertThrows(IOException.class, () -> writer.writeInternal(table)),
+ cause, "write");
+ } finally {
+ table.close();
+ }
+ }
+
+ @Test
+ void writerReportsCommitFailureAndStillAttemptsRollback() throws Exception {
+ Map<String, String> params = params();
+ params.put("use_transaction", "true");
+ JdbcJniWriter writer = new JdbcJniWriter(1, params);
+ SQLException cause = failure();
+ int[] rollbacks = {0};
+ Connection connection = proxy(Connection.class, (object, method, args) -> {
+ switch (method.getName()) {
+ case "isClosed": return false;
+ case "commit": throw cause;
+ case "rollback":
+ rollbacks[0]++;
+ return null;
+ default: throw new UnsupportedOperationException(method.getName());
+ }
+ });
+ field(writer, "conn", connection);
+ assertDiagnostic(Assertions.assertThrows(IOException.class, writer::close), cause, "close");
+ Assertions.assertEquals(1, rollbacks[0]);
+ }
+
+ private static void assertDiagnostic(IOException error, SQLException cause, String operation) {
+ Assertions.assertSame(cause, error.getCause());
+ String terminal = JniUtil.throwableToString(error);
+ Assertions.assertTrue(terminal.contains(operation), terminal);
+ Assertions.assertTrue(terminal.contains("remote_sqlstate=42000"), terminal);
+ Assertions.assertTrue(terminal.contains("remote_vendor_error_code=1142"), terminal);
+ Assertions.assertTrue(terminal.contains("remote_sqlstate=08006"), terminal);
+ Assertions.assertEquals(terminal.indexOf("remote_sqlstate=42000"),
+ terminal.lastIndexOf("remote_sqlstate=42000"));
+ Assertions.assertFalse(terminal.contains("test-password"), terminal);
+ Assertions.assertFalse(JniUtil.throwableToStackTrace(error).contains("test-password"));
+ }
+
+ private static Map<String, String> params() {
+ Map<String, String> params = new HashMap<>();
+ params.put("jdbc_url", "jdbc:mysql://localhost/diagnostics");
+ params.put("jdbc_password", "test-password");
+ params.put("jdbc_driver_url", "file:///nonexistent/diagnostics-driver.jar");
+ params.put("table_type", "MYSQL");
+ return params;
+ }
+
+ private static void field(Object target, String name, Object value) throws Exception {
+ Field field = target.getClass().getDeclaredField(name);
+ field.setAccessible(true);
+ field.set(target, value);
+ }
+
+ private static <T> T proxy(Class<T> type, InvocationHandler handler) {
+ return type.cast(Proxy.newProxyInstance(type.getClassLoader(), new Class<?>[] {type}, handler));
+ }
+}
diff --git a/fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/OracleDiagnosticsTest.java b/fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/OracleDiagnosticsTest.java
new file mode 100644
index 00000000000..a14766137c1
--- /dev/null
+++ b/fe/be-java-extensions/jdbc-scanner/src/test/java/org/apache/doris/jdbc/OracleDiagnosticsTest.java
@@ -0,0 +1,143 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jdbc;
+
+import org.apache.doris.jni.spi.vec.ColumnType;
+import org.apache.doris.jni.spi.vec.ColumnValueConverter;
+
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.core.LogEvent;
+import org.apache.logging.log4j.core.Logger;
+import org.apache.logging.log4j.core.appender.AbstractAppender;
+import org.apache.logging.log4j.core.config.Property;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Proxy;
+import java.sql.Clob;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CopyOnWriteArrayList;
+import java.util.logging.Handler;
+import java.util.logging.LogRecord;
+
+class OracleDiagnosticsTest {
+ @Test
+ void scannerSuppliesSecretsToDriverMetadataAndClobDiagnosticLogs() throws Exception {
+ String password = "test-password";
+ String url = "jdbc:oracle:thin:alice/url-secret@host:1521:db";
+ Map<String, String> params = new HashMap<>();
+ params.put("table_type", "ORACLE");
+ params.put("jdbc_url", url);
+ params.put("jdbc_password", password);
+ JdbcJniScanner scanner = new JdbcJniScanner(1, params);
+ Field field = JdbcJniScanner.class.getDeclaredField("typeHandler");
+ field.setAccessible(true);
+ JdbcTypeHandler typeHandler = (JdbcTypeHandler) field.get(scanner);
+ SQLException raw = new SQLException("raw=" + password + " url=" + url, "08001", 0);
+ PreparedStatement statement = (PreparedStatement) Proxy.newProxyInstance(
+ PreparedStatement.class.getClassLoader(), new Class<?>[] {PreparedStatement.class},
+ (object, method, args) -> null);
+ Connection[] connectionHolder = new Connection[1];
+ Connection connection = (Connection) Proxy.newProxyInstance(OracleDiagnosticsTest.class.getClassLoader(),
+ new Class<?>[] {Connection.class}, (object, method, args) -> {
+ switch (method.getName()) {
+ case "getMetaData": throw raw;
+ case "prepareStatement": return statement;
+ case "unwrap": {
+ Class<?> target = (Class<?>) args[0];
+ return target == Connection.class
+ ? connectionHolder[0]
+ : Proxy.newProxyInstance(target.getClassLoader(), new Class<?>[] {target},
+ (proxy, proxyMethod, proxyArgs) -> null);
+ }
+ default: throw new UnsupportedOperationException(method.getName());
+ }
+ });
+ connectionHolder[0] = connection;
+ Clob clob = (Clob) Proxy.newProxyInstance(Clob.class.getClassLoader(), new Class<?>[] {Clob.class},
+ (object, method, args) -> {
+ throw raw;
+ });
+ List<String> messages = new CopyOnWriteArrayList<>();
+ List<Throwable> rawCauses = new CopyOnWriteArrayList<>();
+ // Production uses JUL; tests may select the inherited Log4j SLF4J provider instead.
+ Handler julHandler = new Handler() {
+ @Override
+ public void publish(LogRecord record) {
+ messages.add(record.getMessage());
+ if (record.getThrown() != null) {
+ rawCauses.add(record.getThrown());
+ }
+ }
+
+ @Override
+ public void flush() {
+ }
+
+ @Override
+ public void close() {
+ }
+ };
+ AbstractAppender appender = new AbstractAppender("oracle-diagnostics-test", null,
+ null, true, Property.EMPTY_ARRAY) {
+ @Override
+ public void append(LogEvent event) {
+ messages.add(event.getMessage().getFormattedMessage());
+ if (event.getThrown() != null) {
+ rawCauses.add(event.getThrown());
+ }
+ }
+ };
+ java.util.logging.Logger jul = java.util.logging.Logger.getLogger(OracleTypeHandler.class.getName());
+ java.util.logging.Level previousJulLevel = jul.getLevel();
+ Logger log4j = (Logger) LogManager.getLogger(OracleTypeHandler.class);
+ Level previousLevel = log4j.getLevel();
+ jul.setLevel(java.util.logging.Level.ALL);
+ jul.addHandler(julHandler);
+ appender.start();
+ log4j.addAppender(appender);
+ log4j.setLevel(Level.ALL);
+ try {
+ Assertions.assertSame(statement, typeHandler.initializeStatement(connection, "SELECT 1", 1));
+ ColumnValueConverter converter = typeHandler.getOutputConverter(
+ new ColumnType("value", ColumnType.Type.STRING), "");
+ Assertions.assertArrayEquals(new Object[] {null}, converter.convert(new Object[] {clob}));
+ Assertions.assertEquals(2, messages.size());
+ Assertions.assertTrue(rawCauses.isEmpty());
+ for (String message : messages) {
+ Assertions.assertTrue(message.contains("remote_sqlstate=08001"), message);
+ Assertions.assertFalse(message.contains(password), message);
+ Assertions.assertFalse(message.contains("url-secret"), message);
+ }
+ } finally {
+ log4j.removeAppender(appender);
+ log4j.setLevel(previousLevel);
+ appender.stop();
+ jul.removeHandler(julHandler);
+ jul.setLevel(previousJulLevel);
+ julHandler.close();
+ }
+ }
+}
diff --git a/fe/be-java-extensions/jni-spi/src/main/java/org/apache/doris/jni/spi/DiagnosticException.java b/fe/be-java-extensions/jni-spi/src/main/java/org/apache/doris/jni/spi/DiagnosticException.java
new file mode 100644
index 00000000000..a696646b7ad
--- /dev/null
+++ b/fe/be-java-extensions/jni-spi/src/main/java/org/apache/doris/jni/spi/DiagnosticException.java
@@ -0,0 +1,27 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jni.spi;
+
+/**
+ * An exception whose message already contains its complete, sanitized diagnostics.
+ * The JNI bridge must not append raw cause messages to it. Implementations keep the original
+ * cause for Java callers and provide a sanitized stack trace for the BE log.
+ */
+public interface DiagnosticException {
+ String getDiagnosticStackTrace();
+}
diff --git a/fe/be-java-extensions/jni-spi/src/main/java/org/apache/doris/jni/spi/utils/JniUtil.java b/fe/be-java-extensions/jni-spi/src/main/java/org/apache/doris/jni/spi/utils/JniUtil.java
index 8d52896e98b..cb18da2a226 100644
--- a/fe/be-java-extensions/jni-spi/src/main/java/org/apache/doris/jni/spi/utils/JniUtil.java
+++ b/fe/be-java-extensions/jni-spi/src/main/java/org/apache/doris/jni/spi/utils/JniUtil.java
@@ -17,9 +17,14 @@
package org.apache.doris.jni.spi.utils;
+import org.apache.doris.jni.spi.DiagnosticException;
+
import java.io.PrintWriter;
import java.io.StringWriter;
import java.io.Writer;
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.Set;
/**
* Turns a Java exception into the text BE reports to the user.
@@ -37,9 +42,14 @@ public class JniUtil {
public static String throwableToString(Throwable t) {
StringWriter output = new StringWriter();
output.write(String.format("%s: %s", t.getClass().getSimpleName(), t.getMessage()));
- // Follow the chain of exception causes and print them as well.
+ if (t instanceof DiagnosticException) {
+ return output.toString();
+ }
+ // A cause graph may be cyclic; emit each exception only once.
+ Set<Throwable> visited = Collections.newSetFromMap(new IdentityHashMap<Throwable, Boolean>());
+ visited.add(t);
Throwable cause = t;
- while ((cause = cause.getCause()) != null) {
+ while ((cause = cause.getCause()) != null && visited.add(cause)) {
output.write(String.format(" | CAUSED BY: %s: %s",
cause.getClass().getSimpleName(), cause.getMessage()));
}
@@ -48,6 +58,9 @@ public class JniUtil {
/** Full stack trace, for the BE log. */
public static String throwableToStackTrace(Throwable t) {
+ if (t instanceof DiagnosticException) {
+ return ((DiagnosticException) t).getDiagnosticStackTrace();
+ }
Writer output = new StringWriter();
t.printStackTrace(new PrintWriter(output));
return output.toString();
diff --git a/fe/be-java-extensions/jni-spi/src/test/java/org/apache/doris/jni/spi/JniPluginSurfaceTest.java b/fe/be-java-extensions/jni-spi/src/test/java/org/apache/doris/jni/spi/JniPluginSurfaceTest.java
index 40ca0c97b74..e5d0d70274a 100644
--- a/fe/be-java-extensions/jni-spi/src/test/java/org/apache/doris/jni/spi/JniPluginSurfaceTest.java
+++ b/fe/be-java-extensions/jni-spi/src/test/java/org/apache/doris/jni/spi/JniPluginSurfaceTest.java
@@ -85,7 +85,7 @@ public class JniPluginSurfaceTest {
*/
@Test
public void jniPluginApiMajorTracksTheRecordedSurfaceChange() {
- Assertions.assertEquals("5.0", SpiVersion.version());
+ Assertions.assertEquals("6.0", SpiVersion.version());
}
/** The package the frozen surface is, entirely: everything shipped in doris-jni-spi.jar. */
diff --git a/fe/be-java-extensions/jni-spi/src/test/java/org/apache/doris/jni/spi/utils/JniUtilTest.java b/fe/be-java-extensions/jni-spi/src/test/java/org/apache/doris/jni/spi/utils/JniUtilTest.java
new file mode 100644
index 00000000000..5a0b75a84ba
--- /dev/null
+++ b/fe/be-java-extensions/jni-spi/src/test/java/org/apache/doris/jni/spi/utils/JniUtilTest.java
@@ -0,0 +1,41 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jni.spi.utils;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.time.Duration;
+
+class JniUtilTest {
+ @Test
+ void preservesOrdinaryExceptionSummaries() {
+ RuntimeException cause = new RuntimeException("root");
+ Assertions.assertEquals("Exception: context | CAUSED BY: RuntimeException: root",
+ JniUtil.throwableToString(new Exception("context", cause)));
+ }
+
+ @Test
+ void terminatesOnOrdinaryCauseCycles() {
+ Exception first = new Exception("one");
+ Exception second = new Exception("two", first);
+ first.initCause(second);
+ Assertions.assertTimeoutPreemptively(Duration.ofSeconds(2), () -> Assertions.assertEquals(
+ "Exception: one | CAUSED BY: Exception: two", JniUtil.throwableToString(first)));
+ }
+}
diff --git a/fe/be-java-extensions/jni-spi/src/test/resources/jni-plugin-surface.txt b/fe/be-java-extensions/jni-spi/src/test/resources/jni-plugin-surface.txt
index 32390064618..ecbd0d28804 100644
--- a/fe/be-java-extensions/jni-spi/src/test/resources/jni-plugin-surface.txt
+++ b/fe/be-java-extensions/jni-spi/src/test/resources/jni-plugin-surface.txt
@@ -12,6 +12,8 @@
# when fields, constructors, nested types and modifiers were added, and once when generic types,
# throws clauses, each type's own declaration and compile-time constant VALUES were added while
# java.lang.Enum's own members were dropped. The API was untouched both times.
+org.apache.doris.jni.spi.DiagnosticException#<type> interface [public,abstract]
+org.apache.doris.jni.spi.DiagnosticException#getDiagnosticStackTrace():java.lang.String [public,abstract]
org.apache.doris.jni.spi.DorisPlugin#<type> interface [public,abstract]
org.apache.doris.jni.spi.DorisPlugin#getScannerFactories():java.lang.Iterable<org.apache.doris.jni.spi.JniScannerFactory> [public]
org.apache.doris.jni.spi.DorisPlugin#getUdfExecutorFactories():java.lang.Iterable<org.apache.doris.jni.spi.UdfExecutorFactory> [public]
diff --git a/fe/be-java-extensions/plugin-toolkit/src/main/java/org/apache/doris/jni/toolkit/jdbc/JdbcExceptionUtils.java b/fe/be-java-extensions/plugin-toolkit/src/main/java/org/apache/doris/jni/toolkit/jdbc/JdbcExceptionUtils.java
new file mode 100644
index 00000000000..32e98c78147
--- /dev/null
+++ b/fe/be-java-extensions/plugin-toolkit/src/main/java/org/apache/doris/jni/toolkit/jdbc/JdbcExceptionUtils.java
@@ -0,0 +1,236 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jni.toolkit.jdbc;
+
+import java.io.PrintWriter;
+import java.io.StringWriter;
+import java.sql.SQLException;
+import java.util.ArrayDeque;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Comparator;
+import java.util.Deque;
+import java.util.IdentityHashMap;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+/** JDBC diagnostics shared by FE metadata clients and BE readers/writers. */
+public final class JdbcExceptionUtils {
+ private static final Pattern REMOTE_DIAGNOSTIC = Pattern.compile(
+ "(\\[remote_sqlstate=(?:[A-Za-z0-9]{5}|unknown), remote_vendor_error_code=-?[0-9]+, message=)"
+ + "((?:\\[redacted JDBC URL\\]|\\\\.|[^\\\\\\]])*+)(\\])", Pattern.DOTALL);
+ private static final Pattern JDBC_URL = Pattern.compile("(?i)jdbc:[^\\s\"'<>]+");
+ private static final Pattern CREDENTIAL = Pattern.compile(
+ "(?i)([\"']?(?:password|passwd|pwd|user|username|token|secret|access[_-]?key)[\"']?)(\\s*[=:]\\s*)"
+ + "(\\[redacted JDBC URL\\]|\"(?:\\\\.|[^\"\\\\])*+\"|'(?:\\\\.|[^'\\\\])*+'|[^\\s;&,]+)");
+
+ private JdbcExceptionUtils() {
+ }
+
+ /**
+ * Collects both JDBC chains, with identity-based cycle detection. Identical diagnostics are
+ * emitted once, but errors with the same message and different remote codes stay distinct.
+ * Non-SQL exceptions retain their messages without inventing remote error codes.
+ */
+ public static String format(String context, Throwable error, String... sensitiveValues) {
+ return format(context, error, true, sensitiveValues);
+ }
+
+ /** Appends only SQL diagnostics, preserving non-SQL exception messages and formatting. */
+ public static String appendSqlDiagnostics(String context, Throwable error, String... sensitiveValues) {
+ return format(context, error, false, sensitiveValues);
+ }
+
+ private static String format(String context, Throwable error, boolean includeMessages, String[] sensitiveValues) {
+ // RuntimeException constructors allow an absent cause; retain that existing JDBC API behavior.
+ if (error == null) {
+ return redact(context, sensitiveValues);
+ }
+ Set<Throwable> visited = Collections.newSetFromMap(new IdentityHashMap<Throwable, Boolean>());
+ Set<String> messages = new LinkedHashSet<>();
+ Set<String> sqlMessages = new LinkedHashSet<>();
+ Set<String> diagnostics = new LinkedHashSet<>();
+ Deque<Throwable> pending = new ArrayDeque<>();
+ pending.add(error);
+ while (!pending.isEmpty()) {
+ Throwable current = pending.removeFirst();
+ if (!visited.add(current)) {
+ continue;
+ }
+ String message = redact(current.getMessage(), sensitiveValues);
+ if (current instanceof SQLException) {
+ SQLException sqlError = (SQLException) current;
+ String sqlState = sqlError.getSQLState();
+ sqlState = sqlState == null || sqlState.trim().isEmpty() ? "unknown" : sqlState;
+ diagnostics.add("[remote_sqlstate=" + sqlState
+ + ", remote_vendor_error_code=" + sqlError.getErrorCode()
+ + ", message=" + escapeDiagnosticMessage(message) + "]");
+ sqlMessages.add(message);
+ if (sqlError.getNextException() != null) {
+ pending.addLast(sqlError.getNextException());
+ }
+ } else if (includeMessages && !message.isEmpty()) {
+ messages.add(message);
+ }
+ if (current.getCause() != null) {
+ pending.addLast(current.getCause());
+ }
+ }
+ messages.removeAll(sqlMessages);
+ String safeContext = redact(context, sensitiveValues);
+ // An outer formatted wrapper already carries its nested wrapper's context and diagnostics.
+ List<String> wrappers = new ArrayList<>(messages);
+ messages.removeIf(message -> REMOTE_DIAGNOSTIC.matcher(message).find()
+ && (safeContext.contains(message) || wrappers.stream().anyMatch(
+ outer -> !outer.equals(message) && outer.contains(message))));
+ diagnostics.removeIf(diagnostic -> safeContext.contains(diagnostic)
+ || messages.stream().anyMatch(message -> message.contains(diagnostic)));
+ messages.remove(safeContext);
+ messages.addAll(diagnostics);
+ String details = String.join(" | ", messages);
+ String result = safeContext.isEmpty() ? details : safeContext + (details.isEmpty() ? "" : ": " + details);
+ // Sibling wrappers may also share frames without containing one another's messages.
+ Matcher frames = REMOTE_DIAGNOSTIC.matcher(result);
+ Set<String> emitted = new LinkedHashSet<>();
+ StringBuffer unique = new StringBuffer();
+ while (frames.find()) {
+ String frame = emitted.add(frames.group()) ? frames.group() : "";
+ frames.appendReplacement(unique, Matcher.quoteReplacement(frame));
+ }
+ return frames.appendTail(unique).toString();
+ }
+
+ /** Redacts configured secrets and JDBC URLs before diagnostic text crosses an output boundary. */
+ public static String redact(String text, String... sensitiveValues) {
+ if (text == null) {
+ return "";
+ }
+ // Remote codes are scalar diagnostics, not credential text. In particular a short
+ // numeric password must not change SQLSTATE or the vendor code on repeated formatting.
+ Matcher codes = REMOTE_DIAGNOSTIC.matcher(text);
+ StringBuilder result = new StringBuilder();
+ int start = 0;
+ while (codes.find()) {
+ result.append(redactText(text.substring(start, codes.start()), sensitiveValues));
+ result.append(codes.group(1));
+ result.append(escapeDiagnosticMessage(redactText(unescapeDiagnosticMessage(codes.group(2)),
+ sensitiveValues)));
+ result.append(codes.group(3));
+ start = codes.end();
+ }
+ return result.append(redactText(text.substring(start), sensitiveValues)).toString();
+ }
+
+ // Escape message delimiters so arbitrary driver text cannot terminate a diagnostic frame.
+ // Keep the canonical URL mask intact for readability and repeated redaction.
+ private static String escapeDiagnosticMessage(String message) {
+ return transformDiagnosticMessage(message, true);
+ }
+
+ private static String unescapeDiagnosticMessage(String message) {
+ return transformDiagnosticMessage(message, false);
+ }
+
+ private static String transformDiagnosticMessage(String message, boolean escape) {
+ StringBuilder result = new StringBuilder();
+ for (int i = 0; i < message.length(); i++) {
+ if (message.startsWith("[redacted JDBC URL]", i)) {
+ result.append("[redacted JDBC URL]");
+ i += "[redacted JDBC URL]".length() - 1;
+ } else {
+ char character = message.charAt(i);
+ if (escape && (character == '\\' || character == ']')) {
+ result.append('\\');
+ } else if (!escape && character == '\\') {
+ character = message.charAt(++i);
+ }
+ result.append(character);
+ }
+ }
+ return result.toString();
+ }
+
+ private static String redactText(String text, String[] sensitiveValues) {
+ // Collect all spans against the original text. Rewriting a password first can destroy
+ // a URL/key; rewriting a property first can split a configured password containing ','.
+ List<Redaction> spans = new ArrayList<>();
+ Matcher urls = JDBC_URL.matcher(text);
+ while (urls.find()) {
+ spans.add(new Redaction(urls.start(), urls.end(), true));
+ }
+ Matcher credentials = CREDENTIAL.matcher(text);
+ while (credentials.find()) {
+ String value = credentials.group(3);
+ if (!value.equals("***") && !value.equals("[redacted JDBC URL]")) {
+ spans.add(new Redaction(credentials.start(3), credentials.end(3), false));
+ }
+ }
+ String alternatives = Stream.concat(Stream.of("***", "[redacted JDBC URL]"),
+ Arrays.stream(sensitiveValues).filter(value -> value != null && !value.isEmpty()))
+ .distinct().sorted(Comparator.comparingInt(String::length).reversed())
+ .map(Pattern::quote).collect(Collectors.joining("|"));
+ Matcher secrets = Pattern.compile(alternatives).matcher(text);
+ while (secrets.find()) {
+ String match = secrets.group();
+ // Existing masks stay opaque, even when a password is '*' or contains their text.
+ if (!match.equals("***") && !match.equals("[redacted JDBC URL]")) {
+ spans.add(new Redaction(secrets.start(), secrets.end(), false));
+ }
+ }
+ spans.sort(Comparator.comparingInt(span -> span.start));
+ StringBuilder masked = new StringBuilder();
+ int cursor = 0;
+ for (int i = 0; i < spans.size(); i++) {
+ Redaction span = spans.get(i);
+ int end = span.end;
+ boolean jdbcUrl = span.jdbcUrl;
+ while (i + 1 < spans.size() && spans.get(i + 1).start <= end) {
+ Redaction overlapping = spans.get(++i);
+ end = Math.max(end, overlapping.end);
+ jdbcUrl |= overlapping.jdbcUrl;
+ }
+ masked.append(text, cursor, span.start).append(jdbcUrl ? "[redacted JDBC URL]" : "***");
+ cursor = end;
+ }
+ return masked.append(text, cursor, text.length()).toString();
+ }
+
+ private static final class Redaction {
+ private final int start;
+ private final int end;
+ private final boolean jdbcUrl;
+
+ private Redaction(int start, int end, boolean jdbcUrl) {
+ this.start = start;
+ this.end = end;
+ this.jdbcUrl = jdbcUrl;
+ }
+ }
+
+ public static String stackTrace(Throwable error, String... sensitiveValues) {
+ StringWriter output = new StringWriter();
+ error.printStackTrace(new PrintWriter(output));
+ return redact(output.toString(), sensitiveValues);
+ }
+}
diff --git a/fe/be-java-extensions/plugin-toolkit/src/main/java/org/apache/doris/jni/toolkit/jdbc/JdbcIOException.java b/fe/be-java-extensions/plugin-toolkit/src/main/java/org/apache/doris/jni/toolkit/jdbc/JdbcIOException.java
new file mode 100644
index 00000000000..7143143d730
--- /dev/null
+++ b/fe/be-java-extensions/plugin-toolkit/src/main/java/org/apache/doris/jni/toolkit/jdbc/JdbcIOException.java
@@ -0,0 +1,37 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jni.toolkit.jdbc;
+
+import org.apache.doris.jni.spi.DiagnosticException;
+
+import java.io.IOException;
+
+/** Keeps the original cause while exposing complete, sanitized JDBC diagnostics through JNI. */
+public final class JdbcIOException extends IOException implements DiagnosticException {
+ private final String[] sensitiveValues;
+
+ public JdbcIOException(String context, Throwable cause, String... sensitiveValues) {
+ super(JdbcExceptionUtils.format(context, cause, sensitiveValues), cause);
+ this.sensitiveValues = sensitiveValues.clone();
+ }
+
+ @Override
+ public String getDiagnosticStackTrace() {
+ return JdbcExceptionUtils.stackTrace(this, sensitiveValues);
+ }
+}
diff --git a/fe/be-java-extensions/plugin-toolkit/src/test/java/org/apache/doris/jni/toolkit/jdbc/JdbcExceptionUtilsTest.java b/fe/be-java-extensions/plugin-toolkit/src/test/java/org/apache/doris/jni/toolkit/jdbc/JdbcExceptionUtilsTest.java
new file mode 100644
index 00000000000..64290b3252c
--- /dev/null
+++ b/fe/be-java-extensions/plugin-toolkit/src/test/java/org/apache/doris/jni/toolkit/jdbc/JdbcExceptionUtilsTest.java
@@ -0,0 +1,253 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.jni.toolkit.jdbc;
+
+import org.apache.doris.jni.spi.utils.JniUtil;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.sql.SQLException;
+import java.time.Duration;
+
+class JdbcExceptionUtilsTest {
+ @Test
+ void reportsRemoteCodesSeparatelyFromTheOperationContext() {
+ SQLException cause = new SQLException("bad syntax", "42000", 1064);
+ JdbcIOException error = new JdbcIOException("scan open failed", cause);
+ Assertions.assertEquals("scan open failed: [remote_sqlstate=42000, "
+ + "remote_vendor_error_code=1064, message=bad syntax]", error.getMessage());
+ Assertions.assertSame(cause, error.getCause());
+ Assertions.assertEquals("JdbcIOException: " + error.getMessage(), JniUtil.throwableToString(error));
+ }
+
+ @Test
+ void followsBothChainsAndKeepsDifferentCodesWithTheSameMessage() {
+ SQLException first = new SQLException("denied", "42000", 1142);
+ SQLException second = new SQLException("denied", "28000", 1045);
+ first.setNextException(second);
+ SQLException duplicate = new SQLException("denied", "42000", 1142);
+ second.initCause(duplicate);
+ String result = JdbcExceptionUtils.format("schema", new RuntimeException("denied", first));
+ Assertions.assertEquals("schema: [remote_sqlstate=42000, remote_vendor_error_code=1142, message=denied]"
+ + " | [remote_sqlstate=28000, remote_vendor_error_code=1045, message=denied]", result);
+ }
+
+ @Test
+ void handlesNullEmptyAndBlankSqlStateAndKeepsZeroVendorCode() {
+ for (String state : new String[] {null, "", " "}) {
+ String result = JdbcExceptionUtils.format("read", new SQLException(null, state, 0));
+ Assertions.assertEquals("read: [remote_sqlstate=unknown, remote_vendor_error_code=0, message=]", result);
+ }
+ }
+
+ @Test
+ void terminatesOnCauseAndNextExceptionCycles() {
+ SQLException first = new SQLException("one", "08001", 1);
+ SQLException second = new SQLException("two", "08006", 2);
+ first.setNextException(second);
+ second.setNextException(first);
+ RuntimeException wrapper = new RuntimeException("pool failed", first);
+ first.initCause(wrapper);
+ Assertions.assertTimeoutPreemptively(Duration.ofSeconds(2), () -> {
+ String result = JdbcExceptionUtils.format("connect", wrapper);
+ Assertions.assertEquals("connect: pool failed | [remote_sqlstate=08001, "
+ + "remote_vendor_error_code=1, message=one] | [remote_sqlstate=08006, "
+ + "remote_vendor_error_code=2, message=two]", result);
+ Assertions.assertTrue(JniUtil.throwableToString(new JdbcIOException("connect", wrapper))
+ .contains("remote_sqlstate=08006"));
+ });
+ }
+
+ @Test
+ void absentCauseRetainsContextWithoutRemoteCodes() {
+ Assertions.assertEquals("open", JdbcExceptionUtils.format("open", null));
+ Assertions.assertEquals("open", JdbcExceptionUtils.appendSqlDiagnostics("open", null));
+ Assertions.assertEquals("", JdbcExceptionUtils.format("", null));
+ }
+
+ @Test
+ void doesNotInventRemoteCodesForNonSqlExceptions() {
+ RuntimeException cause = new RuntimeException("driver missing");
+ Assertions.assertEquals("open: driver missing", JdbcExceptionUtils.format("open", cause));
+ Assertions.assertEquals("open", JdbcExceptionUtils.appendSqlDiagnostics("open", cause));
+ }
+
+ @Test
+ void doesNotAppendDiagnosticsAlreadyPresentInTheContext() {
+ SQLException cause = new SQLException("duplicate key", "23000", 1062);
+ String once = JdbcExceptionUtils.format("write", cause);
+ Assertions.assertEquals(once, JdbcExceptionUtils.appendSqlDiagnostics(once, cause));
+ }
+
+ @Test
+ void redactsSecretsFromBothJniSummaryAndStackTraceWithoutChangingTheCause() {
+ String password = "a$special\\password";
+ String url = "jdbc:mysql://user:other-secret@localhost/db?password=url-secret";
+ SQLException cause = new SQLException("password=label-secret; " + url + " raw=" + password, "28000", 1045);
+ JdbcIOException error = new JdbcIOException("connection failed", cause, password, url);
+ for (String rendered : new String[] {JniUtil.throwableToString(error), JniUtil.throwableToStackTrace(error)}) {
+ for (String secret : new String[] {password, "other-secret", "url-secret", "label-secret", url}) {
+ Assertions.assertFalse(rendered.contains(secret), rendered);
+ }
+ Assertions.assertTrue(rendered.contains("remote_sqlstate=28000"), rendered);
+ }
+ Assertions.assertSame(cause, error.getCause());
+ Assertions.assertTrue(cause.getMessage().contains(password));
+ }
+
+ @Test
+ void doesNotDuplicateDiagnosticsFromAnAlreadyFormattedCause() {
+ SQLException sql = new SQLException("denied", "42000", 1142);
+ RuntimeException connection = new RuntimeException(JdbcExceptionUtils.format("connect", sql), sql);
+ Assertions.assertEquals("schema: connect: [remote_sqlstate=42000, "
+ + "remote_vendor_error_code=1142, message=denied]",
+ JdbcExceptionUtils.format("schema", connection));
+ }
+
+ @Test
+ void nestedFormattedWrappersKeepEachRemoteFrameOnce() {
+ SQLException sql = new SQLException("denied raw=test-secret", "28000", 1045);
+ sql.setNextException(new SQLException("denied raw=test-secret", "42000", 1142));
+ Throwable current = sql;
+ String expected = "";
+ for (String context : new String[] {"connect", "initialize", "validate", "streaming job"}) {
+ String formatted = JdbcExceptionUtils.format(context, current, "test-secret");
+ expected = context + ": " + expected;
+ if (current == sql) {
+ expected = formatted;
+ }
+ Assertions.assertEquals(expected, formatted);
+ Assertions.assertEquals(1, formatted.split("remote_sqlstate=28000", -1).length - 1);
+ Assertions.assertEquals(1, formatted.split("remote_sqlstate=42000", -1).length - 1);
+ Assertions.assertFalse(formatted.contains("test-secret"));
+ current = new RuntimeException(formatted, current);
+ }
+ }
+
+ @Test
+ void sharedFramesInDifferentWrappersDoNotHideDistinctRemoteErrors() {
+ SQLException sql = new SQLException("denied", "28000", 1045);
+ String frame = JdbcExceptionUtils.format("", sql);
+ RuntimeException inner = new RuntimeException("borrow: " + frame, sql);
+ RuntimeException outer = new RuntimeException("initialize: " + frame, inner);
+ String formatted = JdbcExceptionUtils.format("job", outer);
+ Assertions.assertEquals(1, formatted.split("remote_sqlstate=28000", -1).length - 1);
+ Assertions.assertTrue(formatted.contains("initialize:"));
+ Assertions.assertTrue(formatted.contains("borrow:"));
+ }
+
+ @Test
+ void shortNumericPasswordsDoNotCorruptRemoteCodes() {
+ for (String password : new String[] {"42", "0", "1064"}) {
+ SQLException cause = new SQLException("raw=" + password, "42000", 1064);
+ String once = JdbcExceptionUtils.format("schema", cause, password);
+ Assertions.assertTrue(once.contains("remote_sqlstate=42000, remote_vendor_error_code=1064"));
+ Assertions.assertTrue(once.contains("message=raw=***"));
+ Assertions.assertEquals(once, JdbcExceptionUtils.appendSqlDiagnostics(once, cause, password));
+ Assertions.assertTrue(JniUtil.throwableToStackTrace(new JdbcIOException("schema", cause, password))
+ .contains("remote_sqlstate=42000, remote_vendor_error_code=1064"));
+ }
+ }
+
+ @Test
+ void redactsUnknownJdbcUrlsAndQuotedCredentialProperties() {
+ Assertions.assertEquals("[redacted JDBC URL] password=***; user=*** token=***",
+ JdbcExceptionUtils.redact("jdbc:postgresql://host/db?password=p password='two words'; user=alice token=xyz"));
+ }
+
+ @Test
+ void overlappingPasswordsCannotDisableStructuralRedaction() {
+ String url = "jdbc:oracle:thin:alice/url-secret@host:1521:db";
+ for (String password : new String[] {"jdbc", "oracle", "password", "*", "redacted"}) {
+ String text = url + " password=another-secret raw=" + password;
+ String safe = JdbcExceptionUtils.redact(text, password, url);
+ Assertions.assertFalse(safe.contains("url-secret"), safe);
+ Assertions.assertFalse(safe.contains("another-secret"), safe);
+ Assertions.assertEquals(safe, JdbcExceptionUtils.redact(safe, password, url));
+ }
+ Assertions.assertEquals("\"password\":***", JdbcExceptionUtils.redact("\"password\":\"another-secret\""));
+ }
+
+ @Test
+ void maskCharactersDoNotDuplicateDiagnosticsWhenRewrapped() {
+ for (String password : new String[] {"*", "**", "***", "a***b", "]", "[", "message", ",", "="}) {
+ SQLException cause = new SQLException("raw=" + password, "42000", 1142);
+ String once = JdbcExceptionUtils.format("connect", cause, password);
+ RuntimeException wrapper = new RuntimeException(once, cause);
+ String twice = JdbcExceptionUtils.format("schema", wrapper, password);
+ Assertions.assertEquals("schema: " + once, twice);
+ Assertions.assertEquals(once, JdbcExceptionUtils.appendSqlDiagnostics(once, cause, password));
+ }
+ }
+
+ @Test
+ void credentialLabelsAndDiagnosticBracketsStayStableWhenRewrapped() {
+ for (String message : new String[] {"password=remote-secret", "password='remote-secret'",
+ "password=abc,def", "password=abc]def"}) {
+ SQLException cause = new SQLException(message, "28000", 1045);
+ String once = JdbcExceptionUtils.format("connect", cause, "abc,def", "abc]def");
+ Assertions.assertEquals("schema: " + once, JdbcExceptionUtils.format("schema",
+ new RuntimeException(once, cause), "abc,def", "abc]def"));
+ Assertions.assertFalse(once.contains("remote-secret"));
+ Assertions.assertFalse(once.contains("def"));
+ }
+ }
+
+ @Test
+ void nestedBracketsAndUrlsStayStableAcrossRepeatedFormatting() {
+ for (String password : new String[] {"redacted", "]", "[", "\\", "0"}) {
+ SQLException first = new SQLException("connect jdbc:mysql://host/db [unbalanced] ] | raw=" + password,
+ "28000", 1045);
+ first.setNextException(new SQLException("other [message] \\ path", "08001", 0));
+ String once = JdbcExceptionUtils.format("connect", first, password);
+ Assertions.assertEquals(once, JdbcExceptionUtils.redact(once, password));
+ Assertions.assertEquals("schema: " + once, JdbcExceptionUtils.format("schema",
+ new RuntimeException(once, first), password));
+ Assertions.assertEquals(once, JdbcExceptionUtils.appendSqlDiagnostics(once, first, password));
+ Assertions.assertTrue(once.contains("[redacted JDBC URL]"), once);
+ }
+ }
+
+ @Test
+ void longMessagesDoNotConsumeStackDuringRedactionAndRewrapping() {
+ String message = String.join("", java.util.Collections.nCopies(20000, "text ] \\ [redacted JDBC URL] "));
+ SQLException cause = new SQLException(message, "42000", 1064);
+ String once = JdbcExceptionUtils.format("connect", cause, "redacted");
+ Assertions.assertEquals(once, JdbcExceptionUtils.redact(once, "redacted"));
+ Assertions.assertEquals("schema: " + once, JdbcExceptionUtils.format("schema",
+ new RuntimeException(once, cause), "redacted"));
+ Assertions.assertTrue(JdbcExceptionUtils.stackTrace(new JdbcIOException("connect", cause, "redacted"),
+ "redacted").contains(once));
+ }
+
+ @Test
+ void escapedQuotedCredentialsAndUrlValuesAreRedactedStably() {
+ for (String text : new String[] {"password=\"first\\\"remaining-secret\"", "password='first\\'remaining-secret'",
+ "password=jdbc:mysql://host/db?password=url-secret", "password=[redacted JDBC URL]"}) {
+ String safe = JdbcExceptionUtils.redact(text);
+ Assertions.assertFalse(safe.contains("remaining-secret"), safe);
+ Assertions.assertFalse(safe.contains("url-secret"), safe);
+ Assertions.assertEquals(safe, JdbcExceptionUtils.redact(safe));
+ SQLException cause = new SQLException(text, "28000", 1045);
+ String once = JdbcExceptionUtils.format("connect", cause);
+ Assertions.assertEquals("schema: " + once, JdbcExceptionUtils.format("schema",
+ new RuntimeException(once, cause)));
+ }
+ }
+}
diff --git a/fe/be-java-extensions/pom.xml b/fe/be-java-extensions/pom.xml
index 2176d691df3..1b488449a55 100644
--- a/fe/be-java-extensions/pom.xml
+++ b/fe/be-java-extensions/pom.xml
@@ -73,7 +73,7 @@ under the License.
surface - adding an interface or a method counts, not only removing or changing one. That
definition is what makes a differing minor safe to ignore.
-->
- <jni.plugin.api.version>5.0</jni.plugin.api.version>
+ <jni.plugin.api.version>6.0</jni.plugin.api.version>
</properties>
<build>
diff --git a/fe/fe-connector/fe-connector-jdbc/pom.xml b/fe/fe-connector/fe-connector-jdbc/pom.xml
index 80532cc7131..8712536def8 100644
--- a/fe/fe-connector/fe-connector-jdbc/pom.xml
+++ b/fe/fe-connector/fe-connector-jdbc/pom.xml
@@ -40,6 +40,13 @@ under the License.
</description>
<dependencies>
+ <!-- Shared, dependency-free Java 8 JDBC diagnostics, also used by the BE plugin. -->
+ <dependency>
+ <groupId>org.apache.doris</groupId>
+ <artifactId>plugin-toolkit</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>fe-connector-spi</artifactId>
diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcDorisConnector.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcDorisConnector.java
index 24db60145ba..2e6e602001b 100644
--- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcDorisConnector.java
+++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/JdbcDorisConnector.java
@@ -29,6 +29,7 @@ import org.apache.doris.connector.spi.DorisConnectorException;
import org.apache.doris.connector.spi.scan.ConnectorScanPlanProvider;
import org.apache.doris.connector.spi.write.ConnectorWritePlanProvider;
import org.apache.doris.foundation.security.JdbcDriverUrlSecurity;
+import org.apache.doris.jni.toolkit.jdbc.JdbcExceptionUtils;
import org.apache.doris.thrift.TJdbcTable;
import org.apache.doris.thrift.TOdbcTableType;
import org.apache.doris.thrift.TTableDescriptor;
@@ -163,8 +164,11 @@ public class JdbcDorisConnector implements Connector {
return ConnectorTestResult.success(
"Connected successfully, found " + dbs.size() + " databases");
} catch (Exception e) {
- LOG.warn("JDBC connection test failed", e);
- return ConnectorTestResult.failure("JDBC connection failed: " + e.getMessage());
+ String diagnostic = JdbcExceptionUtils.format("JDBC connection failed", e,
+ props.getPassword(), props.getJdbcUrl());
+ String diagnosticTrace = JdbcExceptionUtils.stackTrace(e, props.getPassword(), props.getJdbcUrl());
+ LOG.warn("JDBC connection test failed: {}", diagnosticTrace);
+ return ConnectorTestResult.failure(diagnostic);
}
}
@@ -203,7 +207,7 @@ public class JdbcDorisConnector implements Connector {
boolean enableMappingVarbinary = props.isEnableMappingVarbinary();
boolean enableMappingTimestampTz = props.isEnableMappingTimestampTz();
- LOG.info("Creating JDBC connector client for dbType={}, url={}", dbType, jdbcUrl);
+ LOG.info("Creating JDBC connector client for dbType={}", dbType);
return JdbcConnectorClient.create(
dbType, context.getCatalogName(), jdbcUrl, user, password,
driverUrl, driverClass,
diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcClickHouseConnectorClient.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcClickHouseConnectorClient.java
index ddaa82ecbcf..6990b3074d1 100644
--- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcClickHouseConnectorClient.java
+++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcClickHouseConnectorClient.java
@@ -19,7 +19,6 @@ package org.apache.doris.connector.jdbc.client;
import org.apache.doris.connector.jdbc.JdbcDbType;
import org.apache.doris.connector.spi.ConnectorType;
-import org.apache.doris.connector.spi.DorisConnectorException;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -81,7 +80,7 @@ public class JdbcClickHouseConnectorClient extends JdbcConnectorClient {
}
}
} catch (SQLException e) {
- throw new DorisConnectorException("Failed to get database name list from ClickHouse", e);
+ throw jdbcException("Failed to get database name list from ClickHouse", e);
} finally {
closeResources(rs, conn);
}
@@ -105,7 +104,7 @@ public class JdbcClickHouseConnectorClient extends JdbcConnectorClient {
}
consumer.accept(rs);
} catch (SQLException e) {
- throw new DorisConnectorException("Failed to process table", e);
+ throw jdbcException("Failed to process table", e);
} finally {
closeResources(rs, conn);
}
diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcConnectorClient.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcConnectorClient.java
index edc61da0428..845d03f442c 100644
--- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcConnectorClient.java
+++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcConnectorClient.java
@@ -20,6 +20,7 @@ package org.apache.doris.connector.jdbc.client;
import org.apache.doris.connector.jdbc.JdbcDbType;
import org.apache.doris.connector.spi.ConnectorType;
import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.jni.toolkit.jdbc.JdbcExceptionUtils;
import com.zaxxer.hikari.HikariDataSource;
import org.apache.logging.log4j.LogManager;
@@ -87,6 +88,7 @@ public abstract class JdbcConnectorClient implements Closeable {
protected final boolean enableMappingTimestampTz;
protected ClassLoader classLoader;
protected HikariDataSource dataSource;
+ private String jdbcPassword = "";
/**
* Factory method to create the correct client subclass for the given DB type.
@@ -225,6 +227,7 @@ public abstract class JdbcConnectorClient implements Closeable {
private void initializeDataSource(String url, String user, String password,
String driverClass, int poolMin, int poolMax, int maxWait, int maxLife) {
+ jdbcPassword = password;
ClassLoader old = Thread.currentThread().getContextClassLoader();
try {
Thread.currentThread().setContextClassLoader(this.classLoader);
@@ -249,7 +252,7 @@ public abstract class JdbcConnectorClient implements Closeable {
LOG.info("JdbcConnectorClient set PoolMin={}, PoolMax={}, MaxWait={}, MaxLife={}",
poolMin, poolMax, maxWait, maxLife);
} catch (Exception e) {
- throw new DorisConnectorException("Failed to initialize JDBC data source: " + e.getMessage(), e);
+ throw jdbcException("Failed to initialize JDBC data source", e);
} finally {
Thread.currentThread().setContextClassLoader(old);
}
@@ -264,7 +267,7 @@ public abstract class JdbcConnectorClient implements Closeable {
URL url = new URL(resolveDriverUrl(driverUrl));
this.classLoader = getOrCreateDriverClassLoader(url);
} catch (MalformedURLException e) {
- throw new DorisConnectorException(
+ throw jdbcException(
"Failed to load JDBC driver from path: " + driverUrl, e);
}
}
@@ -335,9 +338,9 @@ public abstract class JdbcConnectorClient implements Closeable {
Thread.currentThread().setContextClassLoader(this.classLoader);
return dataSource.getConnection();
} catch (Exception e) {
- throw new DorisConnectorException(
+ throw jdbcException(
"Catalog '" + catalogName + "' cannot connect to JDBC: "
- + getAllExceptionMessages(e), e);
+ + "connection failed", e);
} finally {
Thread.currentThread().setContextClassLoader(old);
}
@@ -355,7 +358,8 @@ public abstract class JdbcConnectorClient implements Closeable {
((Connection) r).close();
}
} catch (SQLException e) {
- LOG.warn("Failed to close resource: {}", e.getMessage(), e);
+ String diagnostic = JdbcExceptionUtils.format("Failed to close resource", e, jdbcPassword, jdbcUrl);
+ LOG.warn("{}", diagnostic);
}
}
}
@@ -383,7 +387,7 @@ public abstract class JdbcConnectorClient implements Closeable {
}
}
} catch (SQLException e) {
- throw new DorisConnectorException("Failed to get database name list from JDBC", e);
+ throw jdbcException("Failed to get database name list from JDBC", e);
} finally {
closeResources(rs, conn);
}
@@ -411,7 +415,7 @@ public abstract class JdbcConnectorClient implements Closeable {
names.add(rs.getString("TABLE_NAME"));
}
} catch (SQLException e) {
- throw new DorisConnectorException(
+ throw jdbcException(
"Failed to list tables for remote database: " + remoteDbName, e);
}
});
@@ -427,7 +431,7 @@ public abstract class JdbcConnectorClient implements Closeable {
exists[0] = true;
}
} catch (SQLException e) {
- throw new DorisConnectorException(
+ throw jdbcException(
"Failed to check table existence: " + remoteDbName + "." + remoteTableName, e);
}
});
@@ -447,9 +451,8 @@ public abstract class JdbcConnectorClient implements Closeable {
schema.add(new JdbcFieldInfo(rs));
}
} catch (SQLException e) {
- throw new DorisConnectorException(
- "Failed to get JDBC columns info for " + remoteDbName + "." + remoteTableName
- + ": " + getRootCauseMessage(e), e);
+ throw jdbcException(
+ "Failed to get JDBC columns info for " + remoteDbName + "." + remoteTableName, e);
} finally {
closeResources(rs, conn);
}
@@ -471,9 +474,8 @@ public abstract class JdbcConnectorClient implements Closeable {
primaryKeys.add(rs.getString("COLUMN_NAME"));
}
} catch (SQLException e) {
- throw new DorisConnectorException(
- "Failed to get primary keys for " + remoteDbName + "." + remoteTableName
- + ": " + getRootCauseMessage(e), e);
+ throw jdbcException(
+ "Failed to get primary keys for " + remoteDbName + "." + remoteTableName, e);
} finally {
closeResources(rs, conn);
}
@@ -510,7 +512,7 @@ public abstract class JdbcConnectorClient implements Closeable {
LOG.debug("Finished executing DML stmt: {}, effected rows: {}", origStmt, effectedRows);
}
} catch (SQLException e) {
- throw new DorisConnectorException("Failed to execute stmt: " + e.getMessage(), e);
+ throw jdbcException("Failed to execute stmt", e);
} finally {
closeResources(stmt, conn);
}
@@ -535,7 +537,7 @@ public abstract class JdbcConnectorClient implements Closeable {
columns.add(new JdbcFieldInfo(metaData, i));
}
} catch (SQLException e) {
- throw new DorisConnectorException("Failed to get columns from query: " + query, e);
+ throw jdbcException("Failed to get columns from query: " + query, e);
} finally {
closeResources(pstmt, conn);
}
@@ -551,7 +553,7 @@ public abstract class JdbcConnectorClient implements Closeable {
conn = getConnection();
return conn.getMetaData().getDriverVersion();
} catch (SQLException e) {
- throw new DorisConnectorException("Failed to get JDBC driver version", e);
+ throw jdbcException("Failed to get JDBC driver version", e);
} finally {
closeResources(conn);
}
@@ -591,7 +593,7 @@ public abstract class JdbcConnectorClient implements Closeable {
consumer.accept(customRs);
}
} catch (SQLException e) {
- throw new DorisConnectorException("Failed to process table", e);
+ throw jdbcException("Failed to process table", e);
} finally {
closeResources(customRs, stmt, standardRs, conn);
}
@@ -695,26 +697,18 @@ public abstract class JdbcConnectorClient implements Closeable {
return map;
}
- private static String getAllExceptionMessages(Throwable throwable) {
- StringBuilder sb = new StringBuilder();
- while (throwable != null) {
- String msg = throwable.getMessage();
- if (msg != null && !msg.isEmpty()) {
- if (sb.length() > 0) {
- sb.append(" | Caused by: ");
- }
- sb.append(msg);
- }
- throwable = throwable.getCause();
- }
- return sb.toString();
+ /** Shares an existing pool without losing the owner's diagnostic credentials. */
+ protected void shareDataSourceWith(JdbcConnectorClient delegate) {
+ delegate.classLoader = classLoader;
+ delegate.dataSource = dataSource;
+ delegate.jdbcPassword = jdbcPassword;
}
- private static String getRootCauseMessage(Throwable throwable) {
- Throwable root = throwable;
- while (root.getCause() != null) {
- root = root.getCause();
- }
- return root.getMessage();
+ protected String jdbcDiagnosticMessage(Throwable cause) {
+ return JdbcExceptionUtils.format("", cause, jdbcPassword, jdbcUrl);
+ }
+
+ protected DorisConnectorException jdbcException(String context, Throwable cause) {
+ return new JdbcDiagnosticException(context, cause, jdbcPassword, jdbcUrl);
}
}
diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcDB2ConnectorClient.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcDB2ConnectorClient.java
index 2c6155c997f..b0c1cba623a 100644
--- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcDB2ConnectorClient.java
+++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcDB2ConnectorClient.java
@@ -19,7 +19,6 @@ package org.apache.doris.connector.jdbc.client;
import org.apache.doris.connector.jdbc.JdbcDbType;
import org.apache.doris.connector.spi.ConnectorType;
-import org.apache.doris.connector.spi.DorisConnectorException;
import java.sql.Connection;
import java.sql.ResultSet;
@@ -70,7 +69,7 @@ public class JdbcDB2ConnectorClient extends JdbcConnectorClient {
}
}
} catch (SQLException e) {
- throw new DorisConnectorException("Failed to get database name list from DB2", e);
+ throw jdbcException("Failed to get database name list from DB2", e);
} finally {
closeResources(rs, conn);
}
diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcDiagnosticException.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcDiagnosticException.java
new file mode 100644
index 00000000000..260411c0dd1
--- /dev/null
+++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcDiagnosticException.java
@@ -0,0 +1,42 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.jdbc.client;
+
+import org.apache.doris.connector.spi.DiagnosticException;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.jni.toolkit.jdbc.JdbcExceptionUtils;
+
+/** Connector-owned, sanitized JDBC diagnostics with the original driver cause retained. */
+public final class JdbcDiagnosticException extends DorisConnectorException implements DiagnosticException {
+ private final String[] sensitiveValues;
+
+ public JdbcDiagnosticException(String context, Throwable cause, String... sensitiveValues) {
+ super(JdbcExceptionUtils.format(context, cause, sensitiveValues), cause);
+ this.sensitiveValues = sensitiveValues.clone();
+ }
+
+ @Override
+ public String getDiagnosticMessage() {
+ return getMessage();
+ }
+
+ @Override
+ public String getDiagnosticStackTrace(Throwable error) {
+ return JdbcExceptionUtils.stackTrace(error, sensitiveValues);
+ }
+}
diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcGbaseConnectorClient.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcGbaseConnectorClient.java
index ac4084d70e8..4db32728a4c 100644
--- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcGbaseConnectorClient.java
+++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcGbaseConnectorClient.java
@@ -19,7 +19,6 @@ package org.apache.doris.connector.jdbc.client;
import org.apache.doris.connector.jdbc.JdbcDbType;
import org.apache.doris.connector.spi.ConnectorType;
-import org.apache.doris.connector.spi.DorisConnectorException;
import java.sql.Connection;
import java.sql.DatabaseMetaData;
@@ -68,7 +67,7 @@ public class JdbcGbaseConnectorClient extends JdbcConnectorClient {
}
}
} catch (SQLException e) {
- throw new DorisConnectorException("Failed to get database name list from GBase", e);
+ throw jdbcException("Failed to get database name list from GBase", e);
} finally {
closeResources(rs, conn);
}
@@ -86,7 +85,7 @@ public class JdbcGbaseConnectorClient extends JdbcConnectorClient {
rs = meta.getTables(remoteDbName, null, remoteTableName, tableTypes);
consumer.accept(rs);
} catch (SQLException e) {
- throw new DorisConnectorException("Failed to process table", e);
+ throw jdbcException("Failed to process table", e);
} finally {
closeResources(rs, conn);
}
@@ -123,7 +122,7 @@ public class JdbcGbaseConnectorClient extends JdbcConnectorClient {
schema.add(new JdbcFieldInfo(rs));
}
} catch (SQLException e) {
- throw new DorisConnectorException(
+ throw jdbcException(
"Failed to get JDBC columns info for " + remoteDbName + "." + remoteTableName, e);
} finally {
closeResources(rs, conn);
diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcMySQLConnectorClient.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcMySQLConnectorClient.java
index 5e8e6ae6594..255ddb7c3a8 100644
--- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcMySQLConnectorClient.java
+++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcMySQLConnectorClient.java
@@ -88,7 +88,7 @@ public class JdbcMySQLConnectorClient extends JdbcConnectorClient {
}
}
} catch (Exception e) {
- LOG.warn("Failed to detect if remote MySQL is Doris: {}", e.getMessage());
+ LOG.warn("Failed to detect if remote MySQL is Doris: {}", jdbcDiagnosticMessage(e));
} finally {
closeResources(rs, stmt, conn);
}
@@ -133,7 +133,7 @@ public class JdbcMySQLConnectorClient extends JdbcConnectorClient {
}
}
} catch (SQLException e) {
- throw new DorisConnectorException("Failed to get database name list from MySQL", e);
+ throw jdbcException("Failed to get database name list from MySQL", e);
} finally {
closeResources(rs, conn);
}
@@ -151,7 +151,7 @@ public class JdbcMySQLConnectorClient extends JdbcConnectorClient {
rs = databaseMetaData.getTables(remoteDbName, null, remoteTableName, tableTypes);
consumer.accept(rs);
} catch (SQLException e) {
- throw new DorisConnectorException("Failed to process table", e);
+ throw jdbcException("Failed to process table", e);
} finally {
closeResources(rs, conn);
}
@@ -192,7 +192,7 @@ public class JdbcMySQLConnectorClient extends JdbcConnectorClient {
}
return "";
} catch (SQLException e) {
- throw new DorisConnectorException(
+ throw jdbcException(
"Failed to get table comment for " + remoteDbName + "." + remoteTableName, e);
} finally {
closeResources(rs, ps, conn);
@@ -218,7 +218,7 @@ public class JdbcMySQLConnectorClient extends JdbcConnectorClient {
}
}
} catch (Exception e) {
- LOG.warn("Failed to get row count for {}.{}: {}", dbName, tableName, e.getMessage());
+ LOG.warn("Failed to get row count for {}.{}: {}", dbName, tableName, jdbcDiagnosticMessage(e));
}
return -1;
}
@@ -235,7 +235,7 @@ public class JdbcMySQLConnectorClient extends JdbcConnectorClient {
primaryKeys.add(rs.getString("COLUMN_NAME"));
}
} catch (SQLException e) {
- throw new DorisConnectorException(
+ throw jdbcException(
"Failed to get primary keys for " + remoteDbName + "." + remoteTableName, e);
} finally {
closeResources(rs, conn);
@@ -267,7 +267,7 @@ public class JdbcMySQLConnectorClient extends JdbcConnectorClient {
}
}
} catch (SQLException e) {
- throw new DorisConnectorException(
+ throw jdbcException(
"Failed to get JDBC columns info for " + remoteDbName + "." + remoteTableName, e);
} finally {
closeResources(rs, conn);
@@ -290,7 +290,7 @@ public class JdbcMySQLConnectorClient extends JdbcConnectorClient {
fieldToType.put(rs.getString("Field"), rs.getString("Type"));
}
} catch (SQLException e) {
- throw new DorisConnectorException(
+ throw jdbcException(
"Failed to get column data types for " + remoteDbName + "." + remoteTableName, e);
} finally {
closeResources(rs, stmt, conn);
diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcOceanBaseConnectorClient.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcOceanBaseConnectorClient.java
index ca041d54f7b..fa051cc1a88 100644
--- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcOceanBaseConnectorClient.java
+++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcOceanBaseConnectorClient.java
@@ -96,7 +96,8 @@ public class JdbcOceanBaseConnectorClient extends JdbcConnectorClient {
enableMappingVarbinary, enableMappingTimestampTz);
}
} catch (SQLException e) {
- LOG.warn("Failed to detect OceanBase compatibility mode, defaulting to MySQL mode", e);
+ LOG.warn("Failed to detect OceanBase compatibility mode, defaulting to MySQL mode: {}",
+ jdbcDiagnosticMessage(e));
result = new JdbcMySQLConnectorClient(
catalogName, JdbcDbType.MYSQL, jdbcUrl,
onlySpecifiedDatabase, includeDatabaseMap, excludeDatabaseMap,
@@ -104,9 +105,7 @@ public class JdbcOceanBaseConnectorClient extends JdbcConnectorClient {
} finally {
closeResources(rs, stmt, conn);
}
- // Share the class loader and data source so the delegate can get connections
- result.classLoader = this.classLoader;
- result.dataSource = this.dataSource;
+ shareDataSourceWith(result);
return result;
}
diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcOracleConnectorClient.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcOracleConnectorClient.java
index fee35965f35..eeacb4a289f 100644
--- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcOracleConnectorClient.java
+++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcOracleConnectorClient.java
@@ -115,7 +115,7 @@ public class JdbcOracleConnectorClient extends JdbcConnectorClient {
return new String[] {rs.getString("TABLE_OWNER"), rs.getString("TABLE_NAME")};
}
} catch (SQLException e) {
- LOG.debug("Failed to resolve synonym {}.{}: {}", schema, tableName, e.getMessage());
+ LOG.debug("Failed to resolve synonym {}.{}: {}", schema, tableName, jdbcDiagnosticMessage(e));
} finally {
closeResources(rs, stmt, conn);
}
@@ -210,7 +210,7 @@ public class JdbcOracleConnectorClient extends JdbcConnectorClient {
}
}
} catch (Exception e) {
- LOG.warn("Failed to get row count for {}.{}: {}", dbName, tableName, e.getMessage());
+ LOG.warn("Failed to get row count for {}.{}: {}", dbName, tableName, jdbcDiagnosticMessage(e));
}
return -1;
}
diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClient.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClient.java
index 9abbe40029b..1c72110703b 100644
--- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClient.java
+++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcPostgreSQLConnectorClient.java
@@ -19,7 +19,6 @@ package org.apache.doris.connector.jdbc.client;
import org.apache.doris.connector.jdbc.JdbcDbType;
import org.apache.doris.connector.spi.ConnectorType;
-import org.apache.doris.connector.spi.DorisConnectorException;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -91,7 +90,7 @@ public class JdbcPostgreSQLConnectorClient extends JdbcConnectorClient {
}
}
} catch (SQLException e) {
- throw new DorisConnectorException(
+ throw jdbcException(
"Failed to get JDBC columns info for " + remoteDbName + "." + remoteTableName, e);
} finally {
closeResources(rs, conn);
@@ -280,7 +279,7 @@ public class JdbcPostgreSQLConnectorClient extends JdbcConnectorClient {
}
}
} catch (Exception e) {
- LOG.warn("Failed to get row count for {}.{}: {}", dbName, tableName, e.getMessage());
+ LOG.warn("Failed to get row count for {}.{}: {}", dbName, tableName, jdbcDiagnosticMessage(e));
}
return -1;
}
diff --git a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcSQLServerConnectorClient.java b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcSQLServerConnectorClient.java
index f9f115beefd..d968652d4e0 100644
--- a/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcSQLServerConnectorClient.java
+++ b/fe/fe-connector/fe-connector-jdbc/src/main/java/org/apache/doris/connector/jdbc/client/JdbcSQLServerConnectorClient.java
@@ -248,7 +248,7 @@ public class JdbcSQLServerConnectorClient extends JdbcConnectorClient {
}
}
} catch (Exception e) {
- LOG.warn("Failed to get row count for {}.{}: {}", dbName, tableName, e.getMessage());
+ LOG.warn("Failed to get row count for {}.{}: {}", dbName, tableName, jdbcDiagnosticMessage(e));
}
return -1;
}
diff --git a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcDorisConnectorDiagnosticsTest.java b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcDorisConnectorDiagnosticsTest.java
new file mode 100644
index 00000000000..fffeddbec9a
--- /dev/null
+++ b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcDorisConnectorDiagnosticsTest.java
@@ -0,0 +1,123 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.jdbc;
+
+import org.apache.doris.connector.jdbc.client.JdbcConnectorClient;
+import org.apache.doris.connector.jdbc.client.JdbcFieldInfo;
+import org.apache.doris.connector.spi.ConnectorContext;
+import org.apache.doris.connector.spi.ConnectorTestResult;
+import org.apache.doris.connector.spi.ConnectorType;
+
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.core.LogEvent;
+import org.apache.logging.log4j.core.Logger;
+import org.apache.logging.log4j.core.appender.AbstractAppender;
+import org.apache.logging.log4j.core.config.Property;
+import org.apache.logging.log4j.core.layout.PatternLayout;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+import java.sql.SQLException;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.CopyOnWriteArrayList;
+
+class JdbcDorisConnectorDiagnosticsTest {
+ @Test
+ void clientCreationAndConnectionFailureLogsRedactUrlsAndOriginalCauses() throws Exception {
+ String password = "test-password";
+ String url = "jdbc:mysql://localhost/remote?password=url-secret";
+ Map<String, String> props = new HashMap<>();
+ props.put("jdbc_url", url);
+ props.put("password", password);
+ props.put("driver_class", "java.lang.Object");
+ ConnectorContext context = new ConnectorContext() {
+ @Override
+ public String getCatalogName() {
+ return "diagnostics";
+ }
+
+ @Override
+ public long getCatalogId() {
+ return 1;
+ }
+
+ @Override
+ public Map<String, String> getEnvironment() {
+ return Collections.emptyMap();
+ }
+ };
+ JdbcDorisConnector connector = new JdbcDorisConnector(props, context);
+ SQLException raw = new SQLException("raw=" + password + " url=" + url, "08001", 0);
+ JdbcConnectorClient failingClient = new JdbcConnectorClient("diagnostics", JdbcDbType.MYSQL, url, false,
+ Collections.emptyMap(), Collections.emptyMap(), false, false) {
+ @Override
+ public List<String> getDatabaseNameList() {
+ throw jdbcException("load databases", raw);
+ }
+
+ @Override
+ public ConnectorType jdbcTypeToConnectorType(JdbcFieldInfo field) {
+ throw new UnsupportedOperationException();
+ }
+ };
+ List<LogEvent> events = new CopyOnWriteArrayList<>();
+ AbstractAppender appender = new AbstractAppender("jdbc-diagnostics-test", null,
+ PatternLayout.createDefaultLayout(), true, Property.EMPTY_ARRAY) {
+ @Override
+ public void append(LogEvent event) {
+ events.add(event.toImmutable());
+ }
+ };
+ Logger logger = (Logger) LogManager.getLogger(JdbcDorisConnector.class);
+ appender.start();
+ Level previousLevel = logger.getLevel();
+ logger.addAppender(appender);
+ logger.setLevel(Level.INFO);
+ try {
+ // This fails before any remote IO, after logging the client creation event.
+ Assertions.assertFalse(connector.testConnection(null).isSuccess());
+ Field client = JdbcDorisConnector.class.getDeclaredField("client");
+ client.setAccessible(true);
+ client.set(connector, failingClient);
+ ConnectorTestResult result = connector.testConnection(null);
+ Assertions.assertFalse(result.isSuccess());
+ Assertions.assertTrue(result.getMessage().contains("remote_sqlstate=08001"));
+ Assertions.assertFalse(result.getMessage().contains(password));
+ Assertions.assertTrue(events.stream().anyMatch(event -> event.getMessage().getFormattedMessage()
+ .contains("Creating JDBC connector client")));
+ Assertions.assertTrue(events.stream().anyMatch(event -> event.getMessage().getFormattedMessage()
+ .contains("remote_sqlstate=08001")));
+ for (LogEvent event : events) {
+ Assertions.assertNull(event.getThrown(), "A raw cause would bypass diagnostic sanitization");
+ String message = event.getMessage().getFormattedMessage();
+ Assertions.assertFalse(message.contains(password), message);
+ Assertions.assertFalse(message.contains("url-secret"), message);
+ Assertions.assertFalse(message.contains(url), message);
+ }
+ } finally {
+ logger.removeAppender(appender);
+ logger.setLevel(previousLevel);
+ appender.stop();
+ }
+ }
+}
diff --git a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcConnectorDiagnosticsTest.java b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcConnectorDiagnosticsTest.java
new file mode 100644
index 00000000000..48f0ea7a7df
--- /dev/null
+++ b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcConnectorDiagnosticsTest.java
@@ -0,0 +1,86 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.jdbc.client;
+
+import org.apache.doris.connector.jdbc.JdbcDbType;
+import org.apache.doris.connector.spi.ConnectorType;
+import org.apache.doris.connector.spi.DiagnosticException;
+import org.apache.doris.connector.spi.DorisConnectorException;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Proxy;
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.SQLException;
+import java.util.Collections;
+
+class JdbcConnectorDiagnosticsTest {
+ @Test
+ void schemaFailureIncludesAllRemoteDiagnosticsAndKeepsTheOriginalCause() {
+ SQLException cause = new SQLException("columns denied", "42000", 1142);
+ cause.setNextException(new SQLException("connection failed", "08001", 0));
+ DatabaseMetaData metadata = proxy(DatabaseMetaData.class, (object, method, args) -> {
+ if (method.getName().equals("getSearchStringEscape")) {
+ return "";
+ }
+ if (method.getName().equals("getColumns")) {
+ throw cause;
+ }
+ throw new UnsupportedOperationException(method.getName());
+ });
+ Connection connection = proxy(Connection.class, (object, method, args) -> {
+ switch (method.getName()) {
+ case "getMetaData": return metadata;
+ case "getCatalog": return "remote";
+ case "close": return null;
+ default: throw new UnsupportedOperationException(method.getName());
+ }
+ });
+ JdbcConnectorClient client = new JdbcConnectorClient("diagnostics", JdbcDbType.MYSQL,
+ "jdbc:mysql://localhost/remote?password=url-secret", false,
+ Collections.emptyMap(), Collections.emptyMap(), false, false) {
+ @Override
+ public Connection getConnection() {
+ return connection;
+ }
+
+ @Override
+ public ConnectorType jdbcTypeToConnectorType(JdbcFieldInfo field) {
+ throw new UnsupportedOperationException();
+ }
+ };
+ DorisConnectorException error = Assertions.assertThrows(DorisConnectorException.class,
+ () -> client.getJdbcColumnsInfo("remote", "diagnostics_table"));
+ Assertions.assertSame(cause, error.getCause());
+ DiagnosticException diagnostic = (DiagnosticException) error;
+ String trace = diagnostic.getDiagnosticStackTrace(new RuntimeException("catalog failed", error));
+ Assertions.assertFalse(trace.contains("url-secret"), trace);
+ Assertions.assertTrue(trace.contains("catalog failed"), trace);
+ Assertions.assertTrue(error.getMessage().contains("remote_sqlstate=42000"));
+ Assertions.assertTrue(error.getMessage().contains("remote_vendor_error_code=1142"));
+ Assertions.assertTrue(error.getMessage().contains("remote_sqlstate=08001"));
+ Assertions.assertTrue(error.getMessage().contains("remote.diagnostics_table"));
+ Assertions.assertFalse(error.getMessage().contains("url-secret"));
+ }
+
+ private static <T> T proxy(Class<T> type, java.lang.reflect.InvocationHandler handler) {
+ return type.cast(Proxy.newProxyInstance(type.getClassLoader(), new Class<?>[] {type}, handler));
+ }
+}
diff --git a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcDelegatedDiagnosticsTest.java b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcDelegatedDiagnosticsTest.java
new file mode 100644
index 00000000000..e65ac9fb94b
--- /dev/null
+++ b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/client/JdbcDelegatedDiagnosticsTest.java
@@ -0,0 +1,162 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.jdbc.client;
+
+import org.apache.doris.connector.jdbc.JdbcDbType;
+import org.apache.doris.connector.spi.DiagnosticException;
+import org.apache.doris.connector.spi.DorisConnectorException;
+
+import com.zaxxer.hikari.HikariDataSource;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.core.LogEvent;
+import org.apache.logging.log4j.core.Logger;
+import org.apache.logging.log4j.core.appender.AbstractAppender;
+import org.apache.logging.log4j.core.config.Property;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Proxy;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.concurrent.CopyOnWriteArrayList;
+
+class JdbcDelegatedDiagnosticsTest {
+ private static final String PASSWORD = "test-password";
+ private static final String URL = "jdbc:oceanbase://localhost/remote?password=url-secret";
+
+ @Test
+ void oceanBaseFallbackSharesPasswordWithTheActualDelegateAndSanitizesItsLog() throws Exception {
+ SQLException raw = new SQLException("raw=" + PASSWORD + " url=" + URL, "42000", 1142);
+ Connection first = (Connection) Proxy.newProxyInstance(Connection.class.getClassLoader(),
+ new Class<?>[] {Connection.class}, (object, method, args) -> {
+ switch (method.getName()) {
+ case "createStatement": throw raw;
+ case "close": return null;
+ default: throw new UnsupportedOperationException(method.getName());
+ }
+ });
+ int[] calls = {0};
+ HikariDataSource pool = new HikariDataSource() {
+ @Override
+ public Connection getConnection() throws SQLException {
+ if (calls[0]++ == 0) {
+ return first;
+ }
+ throw raw;
+ }
+ };
+ JdbcOceanBaseConnectorClient client = new JdbcOceanBaseConnectorClient("diagnostics", JdbcDbType.OCEANBASE,
+ URL, false, Collections.emptyMap(), Collections.emptyMap(), false, false);
+ configure(client, pool);
+ List<LogEvent> events = new CopyOnWriteArrayList<>();
+ AbstractAppender appender = recordingAppender(events);
+ Logger logger = (Logger) LogManager.getLogger(JdbcOceanBaseConnectorClient.class);
+ Level previousLevel = logger.getLevel();
+ logger.addAppender(appender);
+ logger.setLevel(Level.INFO);
+ try {
+ DorisConnectorException error = Assertions.assertThrows(DorisConnectorException.class,
+ () -> client.getPrimaryKeys("remote", "diagnostics_table"));
+ Assertions.assertSame(raw, error.getCause());
+ Assertions.assertTrue(error.getMessage().contains("remote_sqlstate=42000"));
+ Assertions.assertFalse(error.getMessage().contains(PASSWORD), error.getMessage());
+ Assertions.assertFalse(((DiagnosticException) error).getDiagnosticStackTrace(error).contains(PASSWORD));
+ assertSafeLogs(events);
+ } finally {
+ logger.removeAppender(appender);
+ logger.setLevel(previousLevel);
+ appender.stop();
+ pool.close();
+ }
+ }
+
+ @Test
+ void optionalRowCountFailuresKeepTheirExistingFallbackAndSanitizeLogs() throws Exception {
+ HikariDataSource pool = new HikariDataSource() {
+ @Override
+ public Connection getConnection() throws SQLException {
+ throw new SQLException("raw=" + PASSWORD + " url=" + URL, "08001", 0);
+ }
+ };
+ List<JdbcConnectorClient> clients = Arrays.asList(
+ new JdbcMySQLConnectorClient("diagnostics", JdbcDbType.MYSQL, URL, false,
+ Collections.emptyMap(), Collections.emptyMap(), false, false),
+ new JdbcOracleConnectorClient("diagnostics", JdbcDbType.ORACLE, URL, false,
+ Collections.emptyMap(), Collections.emptyMap(), false, false),
+ new JdbcPostgreSQLConnectorClient("diagnostics", JdbcDbType.POSTGRESQL, URL, false,
+ Collections.emptyMap(), Collections.emptyMap(), false, false),
+ new JdbcSQLServerConnectorClient("diagnostics", JdbcDbType.SQLSERVER, URL, false,
+ Collections.emptyMap(), Collections.emptyMap(), false, false));
+ try {
+ for (JdbcConnectorClient client : clients) {
+ configure(client, pool);
+ List<LogEvent> events = new CopyOnWriteArrayList<>();
+ AbstractAppender appender = recordingAppender(events);
+ Logger logger = (Logger) LogManager.getLogger(client.getClass());
+ Level previousLevel = logger.getLevel();
+ logger.addAppender(appender);
+ logger.setLevel(Level.INFO);
+ try {
+ Assertions.assertEquals(-1, client.getRowCount("remote", "diagnostics_table"));
+ assertSafeLogs(events);
+ } finally {
+ logger.removeAppender(appender);
+ logger.setLevel(previousLevel);
+ appender.stop();
+ }
+ }
+ } finally {
+ pool.close();
+ }
+ }
+
+ private static void configure(JdbcConnectorClient client, HikariDataSource pool) throws Exception {
+ client.dataSource = pool;
+ Field password = JdbcConnectorClient.class.getDeclaredField("jdbcPassword");
+ password.setAccessible(true);
+ password.set(client, PASSWORD);
+ }
+
+ private static AbstractAppender recordingAppender(List<LogEvent> events) {
+ AbstractAppender appender = new AbstractAppender("jdbc-client-diagnostics-test", null,
+ null, true, Property.EMPTY_ARRAY) {
+ @Override
+ public void append(LogEvent event) {
+ events.add(event.toImmutable());
+ }
+ };
+ appender.start();
+ return appender;
+ }
+
+ private static void assertSafeLogs(List<LogEvent> events) {
+ Assertions.assertFalse(events.isEmpty());
+ for (LogEvent event : events) {
+ Assertions.assertNull(event.getThrown());
+ String text = event.getMessage().getFormattedMessage();
+ Assertions.assertTrue(text.contains("remote_sqlstate="), text);
+ Assertions.assertFalse(text.contains(PASSWORD), text);
+ Assertions.assertFalse(text.contains("url-secret"), text);
+ }
+ }
+}
diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/DiagnosticException.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/DiagnosticException.java
new file mode 100644
index 00000000000..419c5bcb906
--- /dev/null
+++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/DiagnosticException.java
@@ -0,0 +1,30 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.connector.spi;
+
+/**
+ * Opt-in diagnostic policy for exceptions whose original causes must be retained but must not
+ * be rendered directly. The connector owns sanitization; the engine uses these outputs without
+ * inspecting connector properties. This interface is shared parent-first across plugin loaders.
+ */
+public interface DiagnosticException {
+ String getDiagnosticMessage();
+
+ /** Formats the complete output-boundary error, including wrappers around this exception. */
+ String getDiagnosticStackTrace(Throwable error);
+}
diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
index f0fa7ddbc41..7a28ab6aa69 100644
--- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
+++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorPluginSurfaceTest.java
@@ -88,12 +88,13 @@ public class ConnectorPluginSurfaceTest {
Assertions.assertNotNull(in, "missing connector plugin API version resource");
version.load(in);
}
- // Major 17 makes the scoped request the canonical streaming entry point.
- Assertions.assertEquals("17.0", version.getProperty("api.version"));
+ // Major 18 adds DiagnosticException to the major 17 scoped streaming surface.
+ Assertions.assertEquals("18.0", version.getProperty("api.version"));
}
/** Root entry points plus provider/handle types returned to connector plugins. */
private static final List<Class<?>> FROZEN_TYPES = Arrays.asList(
+ DiagnosticException.class,
ConnectorProvider.class,
ConnectorContext.class,
ConnectorSession.class,
diff --git a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
index d0eaa168f45..aa67d2b9468 100644
--- a/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
+++ b/fe/fe-connector/fe-connector-spi/src/test/resources/connector-plugin-surface.txt
@@ -112,6 +112,8 @@ org.apache.doris.connector.spi.ConnectorTableSchema#getTableName():java.lang.Str
org.apache.doris.connector.spi.ConnectorTableSchema#getWriteMetadataIdentity():java.lang.String
org.apache.doris.connector.spi.ConnectorTableSchema#hashCode():int
org.apache.doris.connector.spi.ConnectorTableSchema#toString():java.lang.String
+org.apache.doris.connector.spi.DiagnosticException#getDiagnosticMessage():java.lang.String
+org.apache.doris.connector.spi.DiagnosticException#getDiagnosticStackTrace(java.lang.Throwable):java.lang.String
org.apache.doris.connector.spi.handle.ConnectorColumnHandle#equals(java.lang.Object):boolean
org.apache.doris.connector.spi.handle.ConnectorColumnHandle#hashCode():int
org.apache.doris.connector.spi.handle.ConnectorColumnHandle#withProjectedFieldIds(java.util.Set):org.apache.doris.connector.spi.handle.ConnectorColumnHandle
diff --git a/fe/fe-connector/pom.xml b/fe/fe-connector/pom.xml
index daa012ed574..eab1885c677 100644
--- a/fe/fe-connector/pom.xml
+++ b/fe/fe-connector/pom.xml
@@ -55,7 +55,7 @@ under the License.
of the latter two means bumping this property as well (and fe-extension-spi means bumping
all five families).
-->
- <connector.plugin.api.version>17.0</connector.plugin.api.version>
+ <connector.plugin.api.version>18.0</connector.plugin.api.version>
</properties>
<modules>
diff --git a/fe/fe-core/pom.xml b/fe/fe-core/pom.xml
index 0e2006e53a1..a488b5d809b 100644
--- a/fe/fe-core/pom.xml
+++ b/fe/fe-core/pom.xml
@@ -155,6 +155,13 @@ under the License.
</exclusion>
</exclusions>
</dependency>
+ <!-- Shared, dependency-free Java 8 JDBC diagnostics, also used by the BE plugin. -->
+ <dependency>
+ <groupId>org.apache.doris</groupId>
+ <artifactId>plugin-toolkit</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
<!--
Completes the Jersey 1.x stack on the FE's own classpath. jersey-core arrives here
transitively already, and both com.sun. and javax. are mandatory parent-first for every
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/ExceptionDiagnosticUtils.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/ExceptionDiagnosticUtils.java
new file mode 100644
index 00000000000..001b0d6a232
--- /dev/null
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/ExceptionDiagnosticUtils.java
@@ -0,0 +1,71 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.util;
+
+import org.apache.doris.common.NereidsException;
+import org.apache.doris.connector.spi.DiagnosticException;
+
+import org.apache.commons.lang3.exception.ExceptionUtils;
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.Logger;
+
+import java.util.Collections;
+import java.util.IdentityHashMap;
+import java.util.Set;
+import javax.annotation.Nullable;
+
+/** Applies connector-owned diagnostic policies at FE exception output boundaries. */
+public final class ExceptionDiagnosticUtils {
+ private ExceptionDiagnosticUtils() {
+ }
+
+ public static String message(Throwable error) {
+ DiagnosticException diagnostic = findDiagnosticException(error);
+ return diagnostic == null ? ExceptionUtils.getRootCauseMessage(error) : diagnostic.getDiagnosticMessage();
+ }
+
+ @Nullable
+ public static String sanitizedStackTrace(Throwable error) {
+ DiagnosticException diagnostic = findDiagnosticException(error);
+ return diagnostic == null ? null : diagnostic.getDiagnosticStackTrace(error);
+ }
+
+ /** Keep ordinary Throwable logging while applying connector policies to the full outer exception. */
+ public static void log(Logger logger, Level level, String message, Object argument, Throwable error) {
+ String trace = sanitizedStackTrace(error);
+ if (trace == null) {
+ logger.log(level, message, argument, error);
+ } else {
+ logger.log(level, message + "\n{}", argument, trace);
+ }
+ }
+
+ @Nullable
+ private static DiagnosticException findDiagnosticException(Throwable error) {
+ Set<Throwable> visited = Collections.newSetFromMap(new IdentityHashMap<>());
+ for (Throwable current = error; current != null && visited.add(current);) {
+ if (current instanceof DiagnosticException
+ && ((DiagnosticException) current).getDiagnosticMessage() != null) {
+ return (DiagnosticException) current;
+ }
+ current = current instanceof NereidsException
+ ? ((NereidsException) current).getException() : current.getCause();
+ }
+ return null;
+ }
+}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java
index e62620dd626..13260a4a912 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java
@@ -32,6 +32,7 @@ import org.apache.doris.common.DdlException;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.MetaNotFoundException;
import org.apache.doris.common.UserException;
+import org.apache.doris.common.util.ExceptionDiagnosticUtils;
import org.apache.doris.datasource.log.CatalogLog;
import org.apache.doris.nereids.trees.plans.commands.info.AddPartitionFieldOp;
import org.apache.doris.nereids.trees.plans.commands.info.CreateTableInfo;
@@ -83,7 +84,12 @@ public interface CatalogIf<T extends DatabaseIf> {
try {
return getDbNames();
} catch (Exception e) {
- LOG.warn("failed to get db names in catalog {}", getName(), e);
+ String diagnosticTrace = ExceptionDiagnosticUtils.sanitizedStackTrace(e);
+ if (diagnosticTrace != null) {
+ LOG.warn("failed to get db names in catalog {}: {}", getName(), diagnosticTrace);
+ } else {
+ LOG.warn("failed to get db names in catalog {}", getName(), e);
+ }
return Lists.newArrayList();
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
index 59c76f487b5..9073222dab8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
@@ -37,6 +37,7 @@ import org.apache.doris.common.Pair;
import org.apache.doris.common.ThreadPoolManager;
import org.apache.doris.common.UserException;
import org.apache.doris.common.Version;
+import org.apache.doris.common.util.ExceptionDiagnosticUtils;
import org.apache.doris.common.util.Util;
import org.apache.doris.connector.cache.CacheSpec;
import org.apache.doris.connector.cache.MetaCacheBudgetManager;
@@ -66,7 +67,6 @@ import com.google.common.collect.Sets;
import com.google.gson.annotations.SerializedName;
import org.apache.commons.lang3.NotImplementedException;
import org.apache.commons.lang3.StringUtils;
-import org.apache.commons.lang3.exception.ExceptionUtils;
import org.apache.commons.lang3.math.NumberUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -350,8 +350,13 @@ public abstract class ExternalCatalog
this.errorMsg = "";
}
} catch (Exception e) {
- LOG.warn("failed to init catalog {}:{}", name, id, e);
- this.errorMsg = ExceptionUtils.getRootCauseMessage(e);
+ this.errorMsg = initErrorMessage(e);
+ String diagnosticTrace = initErrorStackTrace(e);
+ if (diagnosticTrace != null) {
+ LOG.warn("failed to init catalog {}:{}: {}", name, id, diagnosticTrace);
+ } else {
+ LOG.warn("failed to init catalog {}:{}", name, id, e);
+ }
throw new RuntimeException("Failed to init catalog: " + name + ", error: " + this.errorMsg, e);
} finally {
isInitializing = false;
@@ -368,7 +373,16 @@ public abstract class ExternalCatalog
* {@code alter catalog ... set properties} triggers {@link #resetToUninitialized(boolean)}.
*/
protected void recordDeferredInitError(Throwable t) {
- this.errorMsg = ExceptionUtils.getRootCauseMessage(t);
+ this.errorMsg = initErrorMessage(t);
+ }
+
+ public static String initErrorMessage(Throwable error) {
+ return ExceptionDiagnosticUtils.message(error);
+ }
+
+ @Nullable
+ public static String initErrorStackTrace(Throwable error) {
+ return ExceptionDiagnosticUtils.sanitizedStackTrace(error);
}
protected final void initLocalObjects() {
@@ -740,7 +754,12 @@ public abstract class ExternalCatalog
try {
return getDbNames();
} catch (Exception e) {
- LOG.warn("failed to get db names in catalog {}", getName(), e);
+ String diagnosticTrace = initErrorStackTrace(e);
+ if (diagnosticTrace != null) {
+ LOG.warn("failed to get db names in catalog {}: {}", getName(), diagnosticTrace);
+ } else {
+ LOG.warn("failed to get db names in catalog {}", getName(), e);
+ }
return Lists.newArrayList();
}
} else {
@@ -763,7 +782,12 @@ public abstract class ExternalCatalog
try {
makeSureInitialized();
} catch (Exception e) {
- LOG.warn("failed to get db {} in catalog {}", dbName, name, e);
+ String diagnosticTrace = initErrorStackTrace(e);
+ if (diagnosticTrace != null) {
+ LOG.warn("failed to get db {} in catalog {}: {}", dbName, name, diagnosticTrace);
+ } else {
+ LOG.warn("failed to get db {} in catalog {}", dbName, name, e);
+ }
return null;
}
@@ -845,7 +869,12 @@ public abstract class ExternalCatalog
try {
makeSureInitialized();
} catch (Exception e) {
- LOG.warn("failed to get db {} in catalog {}", dbId, name, e);
+ String diagnosticTrace = initErrorStackTrace(e);
+ if (diagnosticTrace != null) {
+ LOG.warn("failed to get db {} in catalog {}: {}", dbId, name, diagnosticTrace);
+ } else {
+ LOG.warn("failed to get db {} in catalog {}", dbId, name, e);
+ }
return null;
}
@@ -1086,13 +1115,25 @@ public abstract class ExternalCatalog
throw e; // Rethrow to let the caller handle this critical issue
} else {
// Any errors other than name conflicts, we default to not finding the database
- LOG.warn("Failed to check db {} exist in remote system, ignore it.", localDbName, e);
+ String diagnosticTrace = initErrorStackTrace(e);
+ if (diagnosticTrace != null) {
+ LOG.warn("Failed to check db {} exist in remote system, ignore it: {}",
+ localDbName, diagnosticTrace);
+ } else {
+ LOG.warn("Failed to check db {} exist in remote system, ignore it.", localDbName, e);
+ }
return null;
}
} catch (Exception e) {
// If connection failed, it will throw exception.
// ignore it and treat it as not exist.
- LOG.warn("Failed to check db {} exist in remote system, ignore it.", localDbName, e);
+ String diagnosticTrace = initErrorStackTrace(e);
+ if (diagnosticTrace != null) {
+ LOG.warn("Failed to check db {} exist in remote system, ignore it: {}",
+ localDbName, diagnosticTrace);
+ } else {
+ LOG.warn("Failed to check db {} exist in remote system, ignore it.", localDbName, e);
+ }
return null;
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClickHouseClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClickHouseClient.java
index 712b0d243fa..3618ffcaa53 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClickHouseClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClickHouseClient.java
@@ -43,7 +43,7 @@ public class JdbcClickHouseClient extends JdbcClient {
this.databaseTermIsCatalog = isDatabaseTermCatalog(
databaseMetaData, databaseMetaData.getDriverVersion());
} catch (SQLException e) {
- throw new JdbcClientException("Failed to initialize JdbcClickHouseClient: %s", e.getMessage());
+ throw jdbcException("Failed to initialize JdbcClickHouseClient", e);
}
}
@@ -71,7 +71,7 @@ public class JdbcClickHouseClient extends JdbcClient {
}
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get database name list from jdbc", e);
+ throw jdbcException("failed to get database name list from jdbc", e);
} finally {
close(rs, conn);
}
@@ -93,7 +93,7 @@ public class JdbcClickHouseClient extends JdbcClient {
}
resultSetConsumer.accept(rs);
} catch (SQLException e) {
- throw new JdbcClientException("Failed to process table", e);
+ throw jdbcException("Failed to process table", e);
} finally {
close(rs, conn);
}
@@ -245,7 +245,7 @@ public class JdbcClickHouseClient extends JdbcClient {
try (Connection conn = getConnection()) {
return conn.getMetaData().getDriverVersion();
} catch (SQLException e) {
- throw new JdbcClientException("Failed to get jdbc driver version", e);
+ throw jdbcException("Failed to get jdbc driver version", e);
}
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java
index 67ca4d5d9cf..994f949a17f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClient.java
@@ -23,8 +23,8 @@ import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.Type;
import org.apache.doris.cloud.security.SecurityChecker;
import org.apache.doris.common.DdlException;
-import org.apache.doris.common.util.Util;
import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema;
+import org.apache.doris.jni.toolkit.jdbc.JdbcExceptionUtils;
import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
@@ -62,6 +62,8 @@ public abstract class JdbcClient {
private String catalogName;
protected String dbType;
protected String jdbcUser;
+ private String jdbcPassword;
+ private String diagnosticJdbcUrl;
protected ClassLoader classLoader = null;
protected HikariDataSource dataSource = null;
protected boolean isOnlySpecifiedDatabase;
@@ -103,6 +105,8 @@ public abstract class JdbcClient {
setJdbcDriverSystemProperties();
this.catalogName = jdbcClientConfig.getCatalog();
this.jdbcUser = jdbcClientConfig.getUser();
+ this.jdbcPassword = jdbcClientConfig.getPassword();
+ this.diagnosticJdbcUrl = jdbcClientConfig.getJdbcUrl();
this.isOnlySpecifiedDatabase = Boolean.parseBoolean(jdbcClientConfig.getOnlySpecifiedDatabase());
this.includeDatabaseMap =
Optional.ofNullable(jdbcClientConfig.getIncludeDatabaseMap()).orElse(Collections.emptyMap());
@@ -153,7 +157,7 @@ public abstract class JdbcClient {
URL url = new URL(JdbcResource.getFullDriverUrl(config.getDriverUrl()));
classLoaderMap.remove(url);
// Prompt user to verify driver validity and retry
- throw new JdbcClientException(
+ throw jdbcException(
String.format("Failed to load driver class `%s`. "
+ "Please check that the driver JAR is valid and retry.",
config.getDriverClass()), e);
@@ -161,7 +165,7 @@ public abstract class JdbcClient {
// ignore invalid URL when cleaning cache
}
}
- throw new JdbcClientException(e.getMessage(), e);
+ throw jdbcException(e.getMessage(), e);
} finally {
Thread.currentThread().setContextClassLoader(oldClassLoader);
}
@@ -187,7 +191,8 @@ public abstract class JdbcClient {
try {
return JdbcResource.parseDbType(jdbcUrl);
} catch (DdlException e) {
- throw new JdbcClientException("Failed to parse db type from jdbcUrl: " + jdbcUrl, e);
+ throw new JdbcClientException(e,
+ JdbcExceptionUtils.format("Failed to parse db type from jdbcUrl", e, jdbcUrl), jdbcUrl);
}
}
@@ -204,9 +209,8 @@ public abstract class JdbcClient {
conn = dataSource.getConnection();
} catch (Exception e) {
String errorMessage = String.format(
- "Catalog `%s` can not connect to jdbc due to error: %s",
- this.getCatalogName(), JdbcClientException.getAllExceptionMessages(e));
- throw new JdbcClientException(errorMessage, e);
+ "Catalog `%s` can not connect to jdbc due to error", this.getCatalogName());
+ throw jdbcException(errorMessage, e);
} finally {
Thread.currentThread().setContextClassLoader(oldClassLoader);
}
@@ -225,7 +229,7 @@ public abstract class JdbcClient {
((Connection) resource).close();
}
} catch (SQLException e) {
- LOG.warn("Failed to close resource: {}", e.getMessage(), e);
+ LOG.warn("Failed to close resource: {}", jdbcDiagnosticMessage(e));
}
}
}
@@ -247,7 +251,7 @@ public abstract class JdbcClient {
LOG.debug("finished to execute dml stmt: {}, effected rows: {}", origStmt, effectedRows);
}
} catch (SQLException e) {
- throw new JdbcClientException("Failed to execute stmt. error: " + e.getMessage(), e);
+ throw jdbcException("Failed to execute stmt", e);
} finally {
close(stmt, conn);
}
@@ -278,7 +282,7 @@ public abstract class JdbcClient {
}
}
} catch (SQLException e) {
- throw new JdbcClientException("Failed to get columns from query: %s", e, query);
+ throw jdbcException("Failed to get columns from query: %s", e, query);
} finally {
close(pstmt, conn);
}
@@ -322,7 +326,7 @@ public abstract class JdbcClient {
}
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get database name list from jdbc", e);
+ throw jdbcException("failed to get database name list from jdbc", e);
} finally {
close(rs, conn);
}
@@ -353,7 +357,7 @@ public abstract class JdbcClient {
remoteTablesNames.add(rs.getString("TABLE_NAME"));
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get all tables for remote database: `%s`", e, remoteDbName);
+ throw jdbcException("failed to get all tables for remote database: `%s`", e, remoteDbName);
}
});
return remoteTablesNames;
@@ -375,7 +379,7 @@ public abstract class JdbcClient {
isExist[0] = true;
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to judge if table exist for table %s in db %s",
+ throw jdbcException("failed to judge if table exist for table %s in db %s",
e, remoteTableName, remoteDbName);
}
});
@@ -398,8 +402,8 @@ public abstract class JdbcClient {
tableSchema.add(new JdbcFieldSchema(rs));
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get jdbc columns info for remote table `%s.%s`: %s",
- remoteDbName, remoteTableName, Util.getRootCauseMessage(e));
+ throw jdbcException("failed to get jdbc columns info for remote table `%s.%s`",
+ e, remoteDbName, remoteTableName);
} finally {
close(rs, conn);
}
@@ -434,8 +438,8 @@ public abstract class JdbcClient {
primaryKeys.add(fieldName);
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get jdbc primary key info for remote table `%s.%s`: %s",
- remoteDbName, remoteTableName, Util.getRootCauseMessage(e));
+ throw jdbcException("failed to get jdbc primary key info for remote table `%s.%s`",
+ e, remoteDbName, remoteTableName);
} finally {
close(rs, conn);
}
@@ -475,7 +479,7 @@ public abstract class JdbcClient {
resultSetConsumer.accept(customRs);
}
} catch (SQLException e) {
- throw new JdbcClientException("Failed to process table", e);
+ throw jdbcException("Failed to process table", e);
} finally {
close(customRs, stmt, standardRs, conn);
}
@@ -562,7 +566,7 @@ public abstract class JdbcClient {
"Failed to test connection in FE: query executed but returned no results.");
}
} catch (SQLException e) {
- throw new JdbcClientException("Failed to test connection in FE: " + e.getMessage(), e);
+ throw jdbcException("Failed to test connection in FE", e);
} finally {
close(rs, stmt, conn);
}
@@ -578,9 +582,20 @@ public abstract class JdbcClient {
conn = getConnection();
return conn.getMetaData().getDriverVersion();
} catch (SQLException e) {
- throw new JdbcClientException("Failed to get jdbc driver version", e);
+ throw jdbcException("Failed to get jdbc driver version", e);
} finally {
close(conn);
}
}
+
+ protected JdbcClientException jdbcException(String context, Throwable cause, Object... args) {
+ return new JdbcClientException(cause, JdbcExceptionUtils.format(
+ JdbcClientException.formatMessage(context, args), cause, jdbcPassword, diagnosticJdbcUrl),
+ jdbcPassword, diagnosticJdbcUrl);
+ }
+
+ protected String jdbcDiagnosticMessage(Throwable cause) {
+ return JdbcExceptionUtils.format("", cause, jdbcPassword, diagnosticJdbcUrl);
+ }
+
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClientException.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClientException.java
index b07662459da..a39977944ea 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClientException.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcClientException.java
@@ -17,16 +17,43 @@
package org.apache.doris.datasource.jdbc.client;
-public class JdbcClientException extends RuntimeException {
+import org.apache.doris.connector.spi.DiagnosticException;
+import org.apache.doris.jni.toolkit.jdbc.JdbcExceptionUtils;
+
+public class JdbcClientException extends RuntimeException implements DiagnosticException {
+ private final String[] sensitiveValues;
+
public JdbcClientException(String format, Throwable cause, Object... msg) {
- super(formatMessage(format, msg), cause);
+ super(JdbcExceptionUtils.appendSqlDiagnostics(formatMessage(format, msg), cause), cause);
+ sensitiveValues = new String[0];
}
public JdbcClientException(String format, Object... msg) {
- super(formatMessage(format, msg));
+ super(JdbcExceptionUtils.redact(formatMessage(format, msg)));
+ sensitiveValues = new String[0];
+ }
+
+ JdbcClientException(Throwable cause, String diagnosticMessage, String... sensitiveValues) {
+ super(diagnosticMessage, cause);
+ this.sensitiveValues = sensitiveValues.clone();
+ }
+
+ public static JdbcClientException diagnostic(Throwable cause, String context, String... sensitiveValues) {
+ return new JdbcClientException(cause, JdbcExceptionUtils.format(context, cause, sensitiveValues),
+ sensitiveValues);
+ }
+
+ @Override
+ public String getDiagnosticMessage() {
+ return getMessage();
+ }
+
+ @Override
+ public String getDiagnosticStackTrace(Throwable error) {
+ return JdbcExceptionUtils.stackTrace(error, sensitiveValues);
}
- private static String formatMessage(String format, Object... msg) {
+ static String formatMessage(String format, Object... msg) {
if (msg == null || msg.length == 0) {
return format;
} else {
@@ -50,17 +77,10 @@ public class JdbcClientException extends RuntimeException {
}
public static String getAllExceptionMessages(Throwable throwable) {
- StringBuilder sb = new StringBuilder();
- while (throwable != null) {
- String message = throwable.getMessage();
- if (message != null && !message.isEmpty()) {
- if (sb.length() > 0) {
- sb.append(" | Caused by: ");
- }
- sb.append(message);
- }
- throwable = throwable.getCause();
- }
- return sb.toString();
+ return JdbcExceptionUtils.format("", throwable);
+ }
+
+ public static String getAllExceptionMessages(Throwable throwable, String... sensitiveValues) {
+ return JdbcExceptionUtils.format("", throwable, sensitiveValues);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcDB2Client.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcDB2Client.java
index 2c989f4ecd3..31e167c4c04 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcDB2Client.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcDB2Client.java
@@ -55,7 +55,7 @@ public class JdbcDB2Client extends JdbcClient {
}
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get database name list from jdbc", e);
+ throw jdbcException("failed to get database name list from jdbc", e);
} finally {
close(rs, conn);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcGbaseClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcGbaseClient.java
index 6121ef2dbfc..6f69f4b897e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcGbaseClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcGbaseClient.java
@@ -17,9 +17,9 @@
package org.apache.doris.datasource.jdbc.client;
+
import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.Type;
-import org.apache.doris.common.util.Util;
import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema;
import com.google.common.collect.Lists;
@@ -55,7 +55,7 @@ public class JdbcGbaseClient extends JdbcClient {
}
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get database name list from jdbc", e);
+ throw jdbcException("failed to get database name list from jdbc", e);
} finally {
close(rs, conn);
}
@@ -73,7 +73,7 @@ public class JdbcGbaseClient extends JdbcClient {
rs = databaseMetaData.getTables(remoteDbName, null, remoteTableName, tableTypes);
resultSetConsumer.accept(rs);
} catch (SQLException e) {
- throw new JdbcClientException("Failed to process table", e);
+ throw jdbcException("Failed to process table", e);
} finally {
close(rs, conn);
}
@@ -100,8 +100,8 @@ public class JdbcGbaseClient extends JdbcClient {
tableSchema.add(field);
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get jdbc columns info for remote table `%s.%s`: %s",
- remoteDbName, remoteTableName, Util.getRootCauseMessage(e));
+ throw jdbcException("failed to get jdbc columns info for remote table `%s.%s`",
+ e, remoteDbName, remoteTableName);
} finally {
close(rs, conn);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClient.java
index 36e5b49b381..bdc00baa5d5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcMySQLClient.java
@@ -17,10 +17,10 @@
package org.apache.doris.datasource.jdbc.client;
+
import org.apache.doris.catalog.ArrayType;
import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.Type;
-import org.apache.doris.common.util.Util;
import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema;
import com.google.common.base.Preconditions;
@@ -71,7 +71,7 @@ public class JdbcMySQLClient extends JdbcClient {
}
} catch (SQLException | JdbcClientException e) {
closeClient();
- throw new JdbcClientException("Failed to initialize JdbcMySQLClient: %s", e.getMessage());
+ throw jdbcException("Failed to initialize JdbcMySQLClient", e);
} finally {
close(rs, stmt, conn);
}
@@ -111,7 +111,7 @@ public class JdbcMySQLClient extends JdbcClient {
tableCommentBuilder.add(Strings.nullToEmpty(rs.getString("REMARKS")));
}
} catch (SQLException e) {
- throw new JdbcClientException(
+ throw jdbcException(
"failed to get table's comment for remote database: `%s`, remote table: `%s`",
e, remoteDbName, remoteTableName);
}
@@ -145,7 +145,7 @@ public class JdbcMySQLClient extends JdbcClient {
}
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get database name list from jdbc", e);
+ throw jdbcException("failed to get database name list from jdbc", e);
} finally {
close(rs, conn);
}
@@ -163,7 +163,7 @@ public class JdbcMySQLClient extends JdbcClient {
rs = databaseMetaData.getTables(remoteDbName, null, remoteTableName, tableTypes);
resultSetConsumer.accept(rs);
} catch (SQLException e) {
- throw new JdbcClientException("Failed to process table", e);
+ throw jdbcException("Failed to process table", e);
} finally {
close(rs, conn);
}
@@ -204,8 +204,8 @@ public class JdbcMySQLClient extends JdbcClient {
tableSchema.add(field);
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get jdbc columns info for remote table `%s.%s`: %s",
- remoteDbName, remoteTableName, Util.getRootCauseMessage(e));
+ throw jdbcException("failed to get jdbc columns info for remote table `%s.%s`",
+ e, remoteDbName, remoteTableName);
} finally {
close(rs, conn);
}
@@ -226,8 +226,8 @@ public class JdbcMySQLClient extends JdbcClient {
primaryKeys.put(rs.getShort("KEY_SEQ"), rs.getString("COLUMN_NAME"));
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get jdbc primary key info for remote table `%s.%s`: %s",
- remoteDbName, remoteTableName, Util.getRootCauseMessage(e));
+ throw jdbcException("failed to get jdbc primary key info for remote table `%s.%s`",
+ e, remoteDbName, remoteTableName);
} finally {
close(rs, conn);
}
@@ -403,8 +403,8 @@ public class JdbcMySQLClient extends JdbcClient {
fieldToType.put(fieldName, typeName);
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get jdbc columns info for remote table `%s.%s`: %s",
- remoteDbName, remoteTableName, Util.getRootCauseMessage(e));
+ throw jdbcException("failed to get jdbc columns info for remote table `%s.%s`",
+ e, remoteDbName, remoteTableName);
} finally {
close(resultSet, stmt, conn);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOceanBaseClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOceanBaseClient.java
index 8dda7721335..bacdc80fa81 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOceanBaseClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOceanBaseClient.java
@@ -54,7 +54,7 @@ public class JdbcOceanBaseClient extends JdbcClient {
throw new JdbcClientException("Failed to determine OceanBase compatibility mode");
}
} catch (SQLException e) {
- throw new JdbcClientException("Failed to initialize JdbcOceanBaseClient: %s", e.getMessage());
+ throw jdbcException("Failed to initialize JdbcOceanBaseClient", e);
} finally {
close(rs, stmt, conn);
closeClient();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOracleClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOracleClient.java
index 5cbb94e20db..433772ecd13 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOracleClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcOracleClient.java
@@ -17,9 +17,9 @@
package org.apache.doris.datasource.jdbc.client;
+
import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.Type;
-import org.apache.doris.common.util.Util;
import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema;
import com.google.common.base.Strings;
@@ -99,8 +99,7 @@ public class JdbcOracleClient extends JdbcClient {
}
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get table name list from jdbc for table %s:%s", e, remoteTableName,
- Util.getRootCauseMessage(e));
+ throw jdbcException("failed to get table name list from jdbc for table %s", e, remoteTableName);
} finally {
close(rs, conn, stmt, isSynonymRs, synonymInfoRs);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java
index b3cebe60824..57a8212632c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/client/JdbcPostgreSQLClient.java
@@ -17,10 +17,10 @@
package org.apache.doris.datasource.jdbc.client;
+
import org.apache.doris.catalog.ArrayType;
import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.Type;
-import org.apache.doris.common.util.Util;
import org.apache.doris.datasource.jdbc.util.JdbcFieldSchema;
import com.google.common.collect.Lists;
@@ -86,14 +86,14 @@ public class JdbcPostgreSQLClient extends JdbcClient {
}
} catch (SQLException ex) {
LOG.warn("Failed to get array dimensions for column {}: {}",
- columnName, Util.getRootCauseMessage(ex));
+ columnName, jdbcDiagnosticMessage(ex));
} finally {
close(arrayRs, null);
if (pstmt != null) {
try {
pstmt.close();
} catch (SQLException ex) {
- LOG.warn("Failed to close prepared statement: {}", Util.getRootCauseMessage(ex));
+ LOG.warn("Failed to close prepared statement: {}", jdbcDiagnosticMessage(ex));
}
}
}
@@ -101,8 +101,8 @@ public class JdbcPostgreSQLClient extends JdbcClient {
tableSchema.add(new JdbcFieldSchema(rs, arrayDimensions));
}
} catch (SQLException e) {
- throw new JdbcClientException("failed to get jdbc columns info for remote table `%s.%s`: %s",
- remoteDbName, remoteTableName, Util.getRootCauseMessage(e));
+ throw jdbcException("failed to get jdbc columns info for remote table `%s.%s`",
+ e, remoteDbName, remoteTableName);
} finally {
close(rs, conn);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java
index de453aec99e..6db16f5d960 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/DataSourceConfigValidator.java
@@ -183,9 +183,11 @@ public class DataSourceConfigValidator {
} catch (JobException e) {
throw e;
} catch (Exception e) {
- throw new JobException(
- "Failed to validate OceanBase compatibility mode: "
- + JdbcClientException.getAllExceptionMessages(e), e);
+ JdbcClientException diagnostic = JdbcClientException.diagnostic(e,
+ "Failed to validate OceanBase compatibility mode",
+ sourceProperties.get(DataSourceConfigKeys.PASSWORD),
+ sourceProperties.get(DataSourceConfigKeys.JDBC_URL));
+ throw new JobException(diagnostic.getMessage(), diagnostic);
} finally {
jdbcClient.closeClient();
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertJob.java b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertJob.java
index 028fa871767..0b99218be95 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/job/extensions/insert/streaming/StreamingInsertJob.java
@@ -32,8 +32,10 @@ import org.apache.doris.common.InternalErrorCode;
import org.apache.doris.common.Pair;
import org.apache.doris.common.UserException;
import org.apache.doris.common.io.Text;
+import org.apache.doris.common.util.ExceptionDiagnosticUtils;
import org.apache.doris.common.util.TimeUtils;
import org.apache.doris.datasource.InternalCatalog;
+import org.apache.doris.datasource.jdbc.client.JdbcClientException;
import org.apache.doris.job.base.AbstractJob;
import org.apache.doris.job.base.JobExecutionConfiguration;
import org.apache.doris.job.base.TimerDefinition;
@@ -92,6 +94,7 @@ import lombok.Setter;
import lombok.extern.log4j.Log4j2;
import org.apache.commons.collections4.CollectionUtils;
import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.Level;
import java.io.DataOutput;
import java.io.IOException;
@@ -278,8 +281,13 @@ public class StreamingInsertJob extends AbstractJob<StreamingJobSchedulerTask, M
this.offsetProvider = createOffsetProvider(getConvertedSourceProperties());
this.offsetProvider.initOnCreate(this.syncTables);
} catch (Exception ex) {
- log.warn("init streaming job for {} failed", dataSourceType, ex);
- throw new RuntimeException(ex.getMessage(), ex);
+ JdbcClientException diagnostic = JdbcClientException.diagnostic(ex,
+ "init streaming job for " + dataSourceType + " failed",
+ sourceProperties.get(DataSourceConfigKeys.PASSWORD),
+ sourceProperties.get(DataSourceConfigKeys.JDBC_URL));
+ ExceptionDiagnosticUtils.log(log, Level.WARN, "init streaming job for {} failed",
+ dataSourceType, diagnostic);
+ throw new RuntimeException(diagnostic.getMessage(), diagnostic);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContextUtil.java b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContextUtil.java
index 55dd81147ae..6cd1f26a5da 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContextUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContextUtil.java
@@ -24,7 +24,7 @@ import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.Pair;
-import org.apache.doris.common.util.Util;
+import org.apache.doris.common.util.ExceptionDiagnosticUtils;
import org.apache.doris.nereids.StatementContext;
import java.util.Map;
@@ -90,7 +90,7 @@ public class ConnectContextUtil {
} catch (DdlException e) {
return Optional.of(Pair.of(e.getMysqlErrorCode(), e.getMessage()));
} catch (Throwable t) {
- return Optional.of(Pair.of(ErrorCode.ERR_INTERNAL_ERROR, Util.getRootCauseMessage(t)));
+ return Optional.of(Pair.of(ErrorCode.ERR_INTERNAL_ERROR, ExceptionDiagnosticUtils.message(t)));
}
ctx.getState().setOk();
return Optional.empty();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
index c0f9fad616a..c2a0f689b4b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java
@@ -58,6 +58,7 @@ import org.apache.doris.common.profile.SummaryProfile.SummaryBuilder;
import org.apache.doris.common.util.DebugPointUtil;
import org.apache.doris.common.util.DebugPointUtil.DebugPoint;
import org.apache.doris.common.util.DebugUtil;
+import org.apache.doris.common.util.ExceptionDiagnosticUtils;
import org.apache.doris.common.util.NetUtils;
import org.apache.doris.common.util.TimeUtils;
import org.apache.doris.common.util.UniqueIdUtils;
@@ -150,6 +151,7 @@ import com.google.common.collect.Sets;
import com.google.protobuf.ByteString;
import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.lang3.StringUtils;
+import org.apache.logging.log4j.Level;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.thrift.TSerializer;
@@ -788,7 +790,8 @@ public class StmtExecutor {
context.getQueryIdentifier(), e);
throw new UserException(e.getMessage());
}
- LOG.warn("Analyze failed. {}", context.getQueryIdentifier(), e);
+ ExceptionDiagnosticUtils.log(LOG, Level.WARN, "Analyze failed. {}",
+ context.getQueryIdentifier(), e);
// Planning wraps the window rejection in NereidsException/AnalysisException.
// NereidsException(Exception) keeps its wrapped exception outside Throwable.cause.
Throwable cause = e instanceof NereidsException
@@ -801,7 +804,8 @@ public class StmtExecutor {
}
return;
} catch (Exception e) {
- LOG.warn("Nereids execute failed. {}", context.getQueryIdentifier(), e);
+ ExceptionDiagnosticUtils.log(LOG, Level.WARN, "Nereids execute failed. {}",
+ context.getQueryIdentifier(), e);
context.getState().setError(e.getMessage());
throw e;
}
@@ -978,7 +982,8 @@ public class StmtExecutor {
((Command) logicalPlan).run(context, this);
} catch (QueryStateException e) {
if (LOG.isDebugEnabled()) {
- LOG.debug("Command({}) process failed.", getStmtForLogging(originStmt.originStmt), e);
+ ExceptionDiagnosticUtils.log(LOG, Level.DEBUG, "Command({}) process failed.",
+ getStmtForLogging(originStmt.originStmt), e);
}
context.setState(e.getQueryState());
throw new NereidsException("Command(" + getStmtForLogging(originStmt.originStmt)
@@ -987,7 +992,8 @@ public class StmtExecutor {
} catch (UserException e) {
// Return message to info client what happened.
if (LOG.isDebugEnabled()) {
- LOG.debug("Command({}) process failed.", getStmtForLogging(originStmt.originStmt), e);
+ ExceptionDiagnosticUtils.log(LOG, Level.DEBUG, "Command({}) process failed.",
+ getStmtForLogging(originStmt.originStmt), e);
}
if (Config.isCloudMode() && SystemInfoService.needRetryWithReplan(e.getDetailMessage())) {
// For errors in SystemInfoService.NEED_REPLAN_ERRORS,
@@ -1000,7 +1006,8 @@ public class StmtExecutor {
new AnalysisException(e.getMessage(), e));
} catch (Exception | Error e) {
// Maybe our bug
- LOG.info("Command({}) process failed.", getStmtForLogging(originStmt.originStmt), e);
+ ExceptionDiagnosticUtils.log(LOG, Level.INFO, "Command({}) process failed.",
+ getStmtForLogging(originStmt.originStmt), e);
context.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR, e.getMessage());
throw new NereidsException("Command (" + getStmtForLogging(originStmt.originStmt)
+ ") process failed.",
@@ -1052,7 +1059,8 @@ public class StmtExecutor {
planner.plan(parsedStmt, context.getSessionVariable().toThrift());
checkBlockRulesByScan(planner);
} catch (Exception e) {
- LOG.warn("Nereids plan query failed:\n{}", getStmtForLogging(originStmt.originStmt), e);
+ ExceptionDiagnosticUtils.log(LOG, Level.WARN, "Nereids plan query failed:\n{}",
+ getStmtForLogging(originStmt.originStmt), e);
throw new NereidsException(new AnalysisException(e.getMessage(), e));
}
profile.getSummaryProfile().setQueryPlanFinishTime(TimeUtils.getStartTimeMs());
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/JdbcCatalogDiagnosticsTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/JdbcCatalogDiagnosticsTest.java
new file mode 100644
index 00000000000..67be8e732f6
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/JdbcCatalogDiagnosticsTest.java
@@ -0,0 +1,144 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource;
+
+import org.apache.doris.connector.spi.DiagnosticException;
+import org.apache.doris.connector.spi.DorisConnectorException;
+import org.apache.doris.datasource.jdbc.client.JdbcClientException;
+import org.apache.doris.jni.toolkit.jdbc.JdbcExceptionUtils;
+
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.core.LogEvent;
+import org.apache.logging.log4j.core.Logger;
+import org.apache.logging.log4j.core.appender.AbstractAppender;
+import org.apache.logging.log4j.core.config.Property;
+import org.apache.logging.log4j.core.layout.PatternLayout;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import java.sql.SQLException;
+
+class JdbcCatalogDiagnosticsTest {
+ @Test
+ void catalogStatusPreservesSafeConnectorAndLegacyDiagnosticsThroughCacheWrappers() {
+ SQLException raw = new SQLException("denied password=test-password", "42000", 1142);
+ String safe = JdbcExceptionUtils.format("load databases", raw, "test-password");
+ JdbcClientException legacy = new JdbcClientException(safe);
+ legacy.initCause(raw);
+ Throwable[] wrappers = {new SafeConnectorException(safe, raw), legacy};
+ for (Throwable wrapper : wrappers) {
+ String status = ExternalCatalog.initErrorMessage(new RuntimeException("cache failed", wrapper));
+ Assertions.assertTrue(status.contains("remote_sqlstate=42000"));
+ Assertions.assertTrue(status.contains("remote_vendor_error_code=1142"));
+ Assertions.assertFalse(status.contains("test-password"));
+ }
+ }
+
+ @Test
+ void nonSqlJdbcFailuresKeepSanitizedContext() {
+ RuntimeException raw = new RuntimeException("driver failure password=test-password");
+ String safe = JdbcExceptionUtils.format("open JDBC client", raw, "test-password");
+ Assertions.assertEquals(safe, ExternalCatalog.initErrorMessage(new SafeConnectorException(safe, raw)));
+ JdbcClientException legacy = new JdbcClientException(safe);
+ legacy.initCause(raw);
+ Assertions.assertEquals(safe, ExternalCatalog.initErrorMessage(legacy));
+ }
+
+ @Test
+ void connectorWithoutMessageStillReportsItsRootCause() {
+ Assertions.assertEquals("IllegalStateException: root failure", ExternalCatalog.initErrorMessage(
+ new DorisConnectorException(null, new IllegalStateException("root failure"))));
+ }
+
+ @Test
+ void nonJdbcConnectorStillReportsItsSpecificRootCause() {
+ Assertions.assertEquals("IllegalStateException: remote IO failure", ExternalCatalog.initErrorMessage(
+ new DorisConnectorException("listDatabases failed", new IllegalStateException("remote IO failure"))));
+ }
+
+ @Test
+ void unrelatedCatalogErrorsKeepTheirExistingRootCauseMessage() {
+ Assertions.assertEquals("IllegalStateException: root failure", ExternalCatalog.initErrorMessage(
+ new RuntimeException("wrapper", new IllegalStateException("root failure"))));
+ }
+
+ @Test
+ void diagnosticPolicySanitizesTheCompleteCatalogWrapperTrace() {
+ SQLException cause = new SQLException("raw=test-password", "08001", 0);
+ String safe = JdbcExceptionUtils.format("load databases", cause, "test-password");
+ Throwable outer = new RuntimeException("outer raw=test-password", new SafeConnectorException(safe, cause));
+ String trace = ExternalCatalog.initErrorStackTrace(outer);
+ Assertions.assertNotNull(trace);
+ Assertions.assertTrue(trace.contains("outer raw=***"));
+ Assertions.assertTrue(trace.contains("remote_sqlstate=08001"));
+ Assertions.assertFalse(trace.contains("test-password"));
+ Assertions.assertNull(ExternalCatalog.initErrorStackTrace(new IllegalStateException("ordinary failure")));
+ }
+
+ @Test
+ void swallowedCatalogFailureRetainsItsCompleteLogTrace() {
+ CatalogIf<?> catalog = Mockito.mock(CatalogIf.class);
+ RuntimeException error = new RuntimeException("catalog wrapper", new IllegalStateException("remote IO failure"));
+ Mockito.when(catalog.getDbNames()).thenThrow(error);
+ Mockito.when(catalog.getDbNamesOrEmpty()).thenCallRealMethod();
+ StringBuilder logs = new StringBuilder();
+ AbstractAppender appender = new AbstractAppender("catalog-diagnostics-test", null,
+ PatternLayout.newBuilder().withPattern("%m%n%ex").build(), false, Property.EMPTY_ARRAY) {
+ @Override
+ public void append(LogEvent event) {
+ logs.append(getLayout().toSerializable(event));
+ }
+ };
+ Logger logger = (Logger) LogManager.getLogger(CatalogIf.class);
+ Level originalLevel = logger.getLevel();
+ appender.start();
+ logger.addAppender(appender);
+ logger.setLevel(Level.WARN);
+ try {
+ Assertions.assertTrue(catalog.getDbNamesOrEmpty().isEmpty());
+ } finally {
+ logger.removeAppender(appender);
+ logger.setLevel(originalLevel);
+ appender.stop();
+ }
+ Assertions.assertTrue(logs.toString().contains("catalog wrapper"), logs.toString());
+ Assertions.assertTrue(logs.toString().contains("Caused by: java.lang.IllegalStateException: remote IO failure"),
+ logs.toString());
+ Assertions.assertTrue(logs.toString().contains("swallowedCatalogFailureRetainsItsCompleteLogTrace"),
+ logs.toString());
+ }
+
+ private static class SafeConnectorException extends DorisConnectorException implements DiagnosticException {
+ SafeConnectorException(String message, Throwable cause) {
+ super(message, cause);
+ }
+
+ @Override
+ public String getDiagnosticMessage() {
+ return getMessage();
+ }
+
+ @Override
+ public String getDiagnosticStackTrace(Throwable error) {
+ return JdbcExceptionUtils.stackTrace(error, "test-password");
+ }
+ }
+
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcClientExceptionTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcClientExceptionTest.java
index 040d4e9109f..713b07b0ce1 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcClientExceptionTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcClientExceptionTest.java
@@ -65,6 +65,14 @@ public class JdbcClientExceptionTest {
Assertions.assertNull(exception.getCause());
}
+ @Test
+ public void testExceptionWithAbsentCause() {
+ JdbcClientException exception = new JdbcClientException("context", (Throwable) null);
+ Assertions.assertEquals("context", exception.getMessage());
+ Assertions.assertNull(exception.getCause());
+ Assertions.assertEquals("", JdbcClientException.getAllExceptionMessages(null));
+ }
+
@Test
public void testExceptionWithCause() {
String message = "Database connection failed.";
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcLegacyDiagnosticsTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcLegacyDiagnosticsTest.java
new file mode 100644
index 00000000000..b07eac70519
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/client/JdbcLegacyDiagnosticsTest.java
@@ -0,0 +1,110 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.jdbc.client;
+
+import com.zaxxer.hikari.HikariDataSource;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.mockito.Mockito;
+
+import java.lang.reflect.Field;
+import java.lang.reflect.Proxy;
+import java.sql.Connection;
+import java.sql.DatabaseMetaData;
+import java.sql.SQLException;
+
+class JdbcLegacyDiagnosticsTest {
+ @Test
+ void schemaFailureKeepsCauseAndRedactsConfiguredSecrets() throws Exception {
+ SQLException cause = new SQLException("columns denied raw=test-password", "42000", 1142);
+ cause.setNextException(new SQLException("connection failed", "08001", 0));
+ DatabaseMetaData metadata = proxy(DatabaseMetaData.class, (object, method, args) -> {
+ if (method.getName().equals("getSearchStringEscape")) {
+ return "";
+ }
+ if (method.getName().equals("getColumns")) {
+ throw cause;
+ }
+ throw new UnsupportedOperationException(method.getName());
+ });
+ Connection connection = proxy(Connection.class, (object, method, args) -> {
+ switch (method.getName()) {
+ case "getMetaData": return metadata;
+ case "getCatalog": return "remote";
+ case "close": return null;
+ default: throw new UnsupportedOperationException(method.getName());
+ }
+ });
+ JdbcClient client = Mockito.mock(JdbcClient.class, Mockito.CALLS_REAL_METHODS);
+ Mockito.doReturn(connection).when(client).getConnection();
+ set(client, "jdbcPassword", "test-password");
+ set(client, "diagnosticJdbcUrl", "jdbc:mysql://localhost/remote?password=url-secret");
+ JdbcClientException error = Assertions.assertThrows(JdbcClientException.class,
+ () -> client.getJdbcColumnsInfo("remote", "diagnostics_table"));
+ Assertions.assertSame(cause, error.getCause());
+ Assertions.assertTrue(error.getMessage().contains("remote_sqlstate=42000"));
+ Assertions.assertTrue(error.getMessage().contains("remote_vendor_error_code=1142"));
+ Assertions.assertTrue(error.getMessage().contains("remote_sqlstate=08001"));
+ Assertions.assertTrue(error.getMessage().contains("remote.diagnostics_table"));
+ Assertions.assertFalse(error.getMessage().contains("test-password"));
+ }
+
+ @Test
+ void connectionFailureIncludesRemoteCodesAndRedactsUrl() throws Exception {
+ String url = "jdbc:mysql://localhost/db?password=url-secret";
+ SQLException cause = new SQLException("connect " + url + " raw=test-password", "08001", 0);
+ JdbcClient client = Mockito.mock(JdbcClient.class, Mockito.CALLS_REAL_METHODS);
+ HikariDataSource dataSource = Mockito.mock(HikariDataSource.class);
+ Mockito.when(dataSource.getConnection()).thenThrow(cause);
+ client.dataSource = dataSource;
+ set(client, "jdbcPassword", "test-password");
+ set(client, "diagnosticJdbcUrl", url);
+ JdbcClientException error = Assertions.assertThrows(JdbcClientException.class, client::getConnection);
+ Assertions.assertSame(cause, error.getCause());
+ Assertions.assertTrue(error.getMessage().contains("can not connect to jdbc due to error:"));
+ Assertions.assertTrue(error.getMessage().contains("remote_sqlstate=08001, remote_vendor_error_code=0"));
+ Assertions.assertFalse(error.getMessage().contains("test-password"));
+ Assertions.assertFalse(error.getMessage().contains("url-secret"));
+ String trace = error.getDiagnosticStackTrace(new RuntimeException("outer wrapper", error));
+ Assertions.assertTrue(trace.contains("outer wrapper"));
+ Assertions.assertTrue(trace.contains("remote_sqlstate=08001, remote_vendor_error_code=0"));
+ Assertions.assertFalse(trace.contains("test-password"));
+ Assertions.assertFalse(trace.contains("url-secret"));
+ }
+
+ @Test
+ void invalidUrlKeepsLegacyPrefixAndRedactsUrl() {
+ String url = "jdbc:unsupported://localhost/db?password=url-secret";
+ JdbcClientException error = Assertions.assertThrows(JdbcClientException.class,
+ () -> JdbcClient.parseDbType(url));
+ Assertions.assertTrue(error.getMessage().startsWith("Failed to parse db type from jdbcUrl:"));
+ Assertions.assertNotNull(error.getCause());
+ Assertions.assertFalse(error.getMessage().contains(url));
+ Assertions.assertFalse(error.getMessage().contains("url-secret"));
+ }
+
+ private static void set(JdbcClient client, String name, String value) throws Exception {
+ Field field = JdbcClient.class.getDeclaredField(name);
+ field.setAccessible(true);
+ field.set(client, value);
+ }
+
+ private static <T> T proxy(Class<T> type, java.lang.reflect.InvocationHandler handler) {
+ return type.cast(Proxy.newProxyInstance(type.getClassLoader(), new Class<?>[] {type}, handler));
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/qe/JdbcFinalOutputDiagnosticsTest.java b/fe/fe-core/src/test/java/org/apache/doris/qe/JdbcFinalOutputDiagnosticsTest.java
new file mode 100644
index 00000000000..574e9ff2e11
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/qe/JdbcFinalOutputDiagnosticsTest.java
@@ -0,0 +1,169 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.qe;
+
+import org.apache.doris.common.NereidsException;
+import org.apache.doris.common.jmockit.Deencapsulation;
+import org.apache.doris.common.util.ExceptionDiagnosticUtils;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.datasource.jdbc.client.JdbcClient;
+import org.apache.doris.datasource.jdbc.client.JdbcClientException;
+import org.apache.doris.job.base.JobExecutionConfiguration;
+import org.apache.doris.job.base.TimerDefinition;
+import org.apache.doris.job.cdc.DataSourceConfigKeys;
+import org.apache.doris.job.common.DataSourceType;
+import org.apache.doris.job.extensions.insert.streaming.StreamingInsertJob;
+import org.apache.doris.job.util.StreamingJobUtils;
+import org.apache.doris.utframe.TestWithFeService;
+
+import org.apache.logging.log4j.Level;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.core.LogEvent;
+import org.apache.logging.log4j.core.Logger;
+import org.apache.logging.log4j.core.appender.AbstractAppender;
+import org.apache.logging.log4j.core.config.Property;
+import org.apache.logging.log4j.core.layout.PatternLayout;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
+
+import java.lang.reflect.InvocationTargetException;
+import java.lang.reflect.Method;
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.util.HashMap;
+import java.util.Map;
+
+class JdbcFinalOutputDiagnosticsTest extends TestWithFeService {
+ private static final String PASSWORD = "boundary-test-secret";
+ private static final String URL = "jdbc:mysql://private-host:3306/source";
+
+ private SQLException rawFailure() {
+ return new SQLException("denied raw=" + PASSWORD + " endpoint=" + URL, "42000", 1142);
+ }
+
+ private void assertSafe(String text) {
+ Assertions.assertTrue(text.contains("remote_sqlstate=42000"), text);
+ Assertions.assertTrue(text.contains("remote_vendor_error_code=1142"), text);
+ Assertions.assertFalse(text.contains(PASSWORD), text);
+ Assertions.assertFalse(text.contains(URL), text);
+ }
+
+ @Test
+ void showDatabasesFinalLogsApplyDiagnosticPolicy() throws Exception {
+ ConnectContext context = Mockito.spy(connectContext);
+ CatalogIf catalog = Mockito.mock(CatalogIf.class);
+ Mockito.doReturn(catalog).when(context).getCatalog("jdbc_diag");
+ Mockito.when(catalog.getDbNames()).thenThrow(JdbcClientException.diagnostic(
+ rawFailure(), "list databases", PASSWORD, URL));
+ try (CapturedLog capture = new CapturedLog(StmtExecutor.class)) {
+ new StmtExecutor(context, "SHOW DATABASES FROM jdbc_diag").execute();
+ Mockito.verify(catalog).getDbNames();
+ String text = capture.text.toString();
+ Assertions.assertTrue(text.contains("process failed"), text);
+ Assertions.assertTrue(text.contains("Analyze failed"), text);
+ Assertions.assertTrue(text.contains("Caused by:"), text);
+ assertSafe(text);
+ }
+ }
+
+ @Test
+ void streamingInitLogsSanitizeDirectSqlException() throws Exception {
+ StreamingInsertJob job = Deencapsulation.newInstance(StreamingInsertJob.class);
+ Deencapsulation.setField(job, "properties", new HashMap<String, String>());
+ Deencapsulation.setField(job, "dataSourceType", DataSourceType.OCEANBASE);
+ Map<String, String> source = new HashMap<>();
+ source.put(DataSourceConfigKeys.PASSWORD, PASSWORD);
+ source.put(DataSourceConfigKeys.JDBC_URL, URL);
+ source.put(DataSourceConfigKeys.DRIVER_URL, "file:///operator/driver.jar");
+ source.put(DataSourceConfigKeys.DRIVER_CLASS, "com.mysql.cj.jdbc.Driver");
+ source.put(DataSourceConfigKeys.USER, "test-user");
+ source.put(DataSourceConfigKeys.DATABASE, "source");
+ Deencapsulation.setField(job, "sourceProperties", source);
+ JobExecutionConfiguration config = new JobExecutionConfiguration();
+ TimerDefinition timer = new TimerDefinition();
+ timer.setStartTimeMs(System.currentTimeMillis());
+ config.setTimerDefinition(timer);
+ Deencapsulation.setField(job, "jobConfig", config);
+ JdbcClient client = Mockito.mock(JdbcClient.class);
+ Connection connection = Mockito.mock(Connection.class);
+ Mockito.when(client.getConnection()).thenReturn(connection);
+ Mockito.when(connection.createStatement()).thenThrow(rawFailure());
+ Method init = StreamingInsertJob.class.getDeclaredMethod("initSourceJob");
+ init.setAccessible(true);
+ try (MockedStatic<StreamingJobUtils> utils = Mockito.mockStatic(StreamingJobUtils.class);
+ CapturedLog capture = new CapturedLog(StreamingInsertJob.class)) {
+ utils.when(() -> StreamingJobUtils.getJdbcClient(Mockito.eq(DataSourceType.OCEANBASE),
+ Mockito.anyMap())).thenReturn(client);
+ InvocationTargetException failure = Assertions.assertThrows(InvocationTargetException.class,
+ () -> init.invoke(job));
+ Mockito.verify(connection).createStatement();
+ assertSafe(failure.getCause().getMessage());
+ assertSafe(ExceptionDiagnosticUtils.sanitizedStackTrace(
+ new RuntimeException("outer job", failure.getCause())));
+ String text = capture.text.toString();
+ Assertions.assertTrue(text.contains("init streaming job for"), text);
+ Assertions.assertTrue(text.contains("Caused by:"), text);
+ assertSafe(text);
+ }
+ Mockito.verify(client).closeClient();
+ }
+
+ @Test
+ void ordinaryErrorsKeepThrowableAndNereidsStoredExceptionKeepsPolicy() {
+ RuntimeException ordinary = new RuntimeException("ordinary failure");
+ try (CapturedLog capture = new CapturedLog(StmtExecutor.class)) {
+ ExceptionDiagnosticUtils.log(LogManager.getLogger(StmtExecutor.class), Level.WARN,
+ "ordinary {}", "query", ordinary);
+ Assertions.assertTrue(capture.text.toString().contains("at org.apache.doris.qe."),
+ capture.text.toString());
+ }
+ JdbcClientException diagnostic = JdbcClientException.diagnostic(rawFailure(), "remote query", PASSWORD, URL);
+ assertSafe(ExceptionDiagnosticUtils.sanitizedStackTrace(new NereidsException(diagnostic)));
+ }
+
+ private static class CapturedLog implements AutoCloseable {
+ private final StringBuilder text = new StringBuilder();
+ private final Logger logger;
+ private final Level originalLevel;
+ private final AbstractAppender appender;
+
+ CapturedLog(Class<?> type) {
+ logger = (Logger) LogManager.getLogger(type);
+ originalLevel = logger.getLevel();
+ appender = new AbstractAppender("boundary-test", null,
+ PatternLayout.newBuilder().withPattern("%m%n%ex").build(), false, Property.EMPTY_ARRAY) {
+ @Override
+ public void append(LogEvent event) {
+ text.append(getLayout().toSerializable(event));
+ }
+ };
+ appender.start();
+ logger.addAppender(appender);
+ logger.setLevel(Level.DEBUG);
+ }
+
+ @Override
+ public void close() {
+ logger.removeAppender(appender);
+ logger.setLevel(originalLevel);
+ appender.stop();
+ }
+ }
+}
diff --git a/regression-test/data/external_table_p0/jdbc/test_jdbc_error_diagnostics.out b/regression-test/data/external_table_p0/jdbc/test_jdbc_error_diagnostics.out
new file mode 100644
index 00000000000..f2b80ed7b31
--- /dev/null
+++ b/regression-test/data/external_table_p0/jdbc/test_jdbc_error_diagnostics.out
@@ -0,0 +1,7 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !read --
+1
+
+-- !unchanged --
+1
+
diff --git a/regression-test/suites/external_table_p0/jdbc/test_jdbc_error_diagnostics.groovy b/regression-test/suites/external_table_p0/jdbc/test_jdbc_error_diagnostics.groovy
new file mode 100644
index 00000000000..4bddb5af96a
--- /dev/null
+++ b/regression-test/suites/external_table_p0/jdbc/test_jdbc_error_diagnostics.groovy
@@ -0,0 +1,93 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Remote codes stay separate from Doris errCode across FE and BE JDBC failures.
+suite("test_jdbc_error_diagnostics", "p0,external") {
+ if (!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableJdbcTest"))) {
+ return
+ }
+ String host = context.config.otherConfigs.get("externalEnvIp")
+ String port = context.config.otherConfigs.get("mysql_57_port")
+ String driverUrl = context.config.otherConfigs.get("jdbcDiagnosticsDriverUrl")
+ if (driverUrl == null) {
+ driverUrl = "https://${getS3BucketName()}.${getS3Endpoint()}/regression/jdbc_driver/mysql-connector-j-8.4.0.jar"
+ }
+ String adminUser = context.config.otherConfigs.get("jdbcDiagnosticsUser") ?: "root"
+ String adminPassword = context.config.otherConfigs.get("jdbcDiagnosticsPassword") ?: "123456"
+ String jdbcUrl = "jdbc:mysql://${host}:${port}/doris_test?useSSL=false&allowPublicKeyRetrieval=true&connectTimeout=1000&socketTimeout=5000"
+
+ ["jdbc_diag_admin", "jdbc_diag_reader", "jdbc_diag_bad_auth", "jdbc_diag_bad_connection"].each { name ->
+ sql "drop catalog if exists ${name}"
+ }
+ sql """create catalog jdbc_diag_admin properties (
+ "type"="jdbc", "user"="${adminUser}", "password"="${adminPassword}",
+ "jdbc_url"="${jdbcUrl}", "driver_url"="${driverUrl}",
+ "driver_class"="com.mysql.cj.jdbc.Driver"
+ )"""
+
+ // The remote fixture belongs to this suite; keep it after execution for debugging.
+ sql """call execute_stmt("jdbc_diag_admin", "DROP TABLE IF EXISTS doris_test.jdbc_diag_table")"""
+ sql """call execute_stmt("jdbc_diag_admin", "CREATE TABLE doris_test.jdbc_diag_table (id INT PRIMARY KEY)")"""
+ sql """call execute_stmt("jdbc_diag_admin", "INSERT INTO doris_test.jdbc_diag_table VALUES (1)")"""
+ sql """call execute_stmt("jdbc_diag_admin", "DROP USER IF EXISTS 'doris_jdbc_diag_reader'@'%'")"""
+ sql """call execute_stmt("jdbc_diag_admin", "CREATE USER 'doris_jdbc_diag_reader'@'%' IDENTIFIED BY 'jdbc-diag-reader-password'")"""
+ sql """call execute_stmt("jdbc_diag_admin", "GRANT SELECT ON doris_test.jdbc_diag_table TO 'doris_jdbc_diag_reader'@'%'")"""
+ sql """create catalog jdbc_diag_reader properties (
+ "type"="jdbc", "user"="doris_jdbc_diag_reader", "password"="jdbc-diag-reader-password",
+ "jdbc_url"="${jdbcUrl}", "driver_url"="${driverUrl}",
+ "driver_class"="com.mysql.cj.jdbc.Driver"
+ )"""
+
+ order_qt_read "select * from jdbc_diag_reader.doris_test.jdbc_diag_table"
+
+ // FE JDBC statement execution: a remote syntax error.
+ test {
+ sql """call execute_stmt("jdbc_diag_admin", "SELCT id FROM doris_test.jdbc_diag_table")"""
+ exception "remote_sqlstate=42000, remote_vendor_error_code=1064"
+ }
+ // BE JDBC writer: metadata is readable, but INSERT is denied remotely.
+ test {
+ sql "insert into jdbc_diag_reader.doris_test.jdbc_diag_table values (2)"
+ exception "remote_sqlstate=42000, remote_vendor_error_code=1142"
+ }
+ // BE batch execution: a remote constraint error, often carried by nextException.
+ test {
+ sql "insert into jdbc_diag_admin.doris_test.jdbc_diag_table values (1)"
+ exception "remote_sqlstate=23000, remote_vendor_error_code=1062"
+ }
+ // CREATE connection validation: rejected credentials.
+ test {
+ sql """create catalog jdbc_diag_bad_auth properties (
+ "type"="jdbc", "user"="${adminUser}", "password"="definitely-not-the-password",
+ "jdbc_url"="${jdbcUrl}", "driver_url"="${driverUrl}",
+ "driver_class"="com.mysql.cj.jdbc.Driver"
+ )"""
+ exception "remote_sqlstate=28000, remote_vendor_error_code=1045"
+ }
+ // A loopback port reserved for the intentionally unreachable JDBC target.
+ String closedPort = context.config.otherConfigs.get("jdbcDiagnosticsClosedPort") ?: "3399"
+ test {
+ sql """create catalog jdbc_diag_bad_connection properties (
+ "type"="jdbc", "user"="${adminUser}", "password"="${adminPassword}",
+ "jdbc_url"="jdbc:mysql://127.0.0.1:${closedPort}/doris_test?connectTimeout=1000",
+ "driver_url"="${driverUrl}", "driver_class"="com.mysql.cj.jdbc.Driver",
+ "connection_pool_min_size"="0", "connection_pool_max_wait_time"="1000"
+ )"""
+ exception "remote_sqlstate=08S01, remote_vendor_error_code=0"
+ }
+ order_qt_unchanged "select * from jdbc_diag_admin.doris_test.jdbc_diag_table"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org