You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2026/10/10 07:53:22 UTC
[PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Gabriel39 opened a new pull request, #68873:
URL: https://github.com/apache/doris/pull/68873
### What problem does this PR solve?
Related PR: #68786, #68532
`branch-4.1` already contains #68532. This PR adapts the applicable correctness fixes found during #68786 review to the 4.1 executors and scan planner.
| Area | Fix |
| --- | --- |
| Iceberg binary partitions | Distinguish SQL NULL from binary bytes whose Base64 path is `null` when reusing writers, preserving separate commit metadata. |
| VARBINARY literals | Compare unsigned bytes with significant trailing zeros in both FE literal implementations. |
| MySQL / OceanBase MySQL | Project instant columns as UTC text before driver decoding and preserve `zeroDateTimeBehavior`. |
| Oracle | Initialize each borrowed driver's session time zone and use explicit-offset text binds with server-side conversion. |
| Presto | Support the official driver's scalar/array timestamp representation and VARCHAR timestamp binds with explicit casts. |
| JDBC query TVFs | Preserve trailing comments, terminal delimiters, MySQL SQL mode and Trino `WITH SESSION` when wrapping timestamp projections. |
| JNI | Reject UTC years outside 0–9999 before packing TIMESTAMPTZ, including nested values, while preserving NULL and year zero. |
| Paimon | Serialize binary partition values losslessly for native readers. |
| MaxCompute | Write nested timestamp vectors using UTC microseconds. |
Existing 4.1 type mappings are retained. The master connector framework, native external UUID migration and unconditional file-TVF mapping changes are outside this backport.
### Release note
Fix external binary partition correctness, VARBINARY literal comparisons, JDBC timestamp handling and nested timestamp writes in 4.1.
### Check List (For Author)
- Test:
- [x] Unit Test: 50 FE tests, 44 Java extension tests and 31 BE writer/partition-transform tests passed locally. BE tests rebuilt the affected source and test with existing 4.1 build dependencies.
- [x] Manual test: MySQL Connector/J 5.1.49 and 8.0.28 each passed all four integration methods, covering JVM/session time zones, client/server prepared statements, cached driver time zones and zero-date policies.
- [x] Regression test added: Iceberg binary/NULL path collision, including independent Spark partition-filter checks for Parquet and ORC. The full external regression suite was not run locally.
- [x] FE Checkstyle and clang-format 16 checks.
- Behavior changed:
- [x] Yes: correct the existing read/write and comparison behavior described above.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68873:
URL: https://github.com/apache/doris/pull/68873#discussion_r4237117442
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcTable.java:
##########
@@ -152,7 +152,18 @@ public String getInsertSql(List<String> insertCols) {
if (i != 0) {
sb.append(", ");
}
- sb.append("?");
+ Column column = getColumn(insertCols.get(i));
+ TOdbcTableType remoteType = getJdbcTableType();
+ // Legacy drivers bind zoned timestamps as VARCHAR; parse their explicit offsets on the server.
+ if (column != null && column.getType().isTimeStampTz()
+ && (remoteType == TOdbcTableType.ORACLE || remoteType == TOdbcTableType.OCEANBASE_ORACLE)) {
+ sb.append("TO_TIMESTAMP_TZ(?, 'YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM')");
+ } else if (column != null && column.getType().isTimeStampTz() && remoteType == TOdbcTableType.PRESTO) {
Review Comment:
[P1] Preserve Oracle zoned-timestamp inserts during rolling FE/BE upgrades. This new `TO_TIMESTAMP_TZ(?, 'YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM')` expects the new BE's text bind, but an old BE still binds an `OffsetDateTime` as JDBC `TIMESTAMP_WITH_TIMEZONE`. Oracle must convert that typed value to text under the session `NLS_TIMESTAMP_TZ_FORMAT` before applying this explicit mask, so a different session mask breaks new-FE/old-BE writes. The reverse old-FE/new-BE pairing sends the new text bind to a raw `?` and again relies on session conversion. Keep the FE SQL and BE binding compatible across both version pairings, and exercise a nonmatching NLS mask. See [Oracle TO_TIMESTAMP_TZ](https://docs.oracle.com/en/database/oracle/oracle-database/19/sqlrf/TO_TIMESTAMP_TZ.html) and [NLS_TIMESTAMP_TZ_FORMAT](https://docs.oracle.com/en/database/oracle/oracle-database/19/refrn/NLS_TIMESTAMP_TZ_FORMAT.html).
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,151 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
+ }
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, jdbcType == TOdbcTableType.CLICKHOUSE);
+ } else if (c == '\'' || c == '"' || c == '`') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ // ANSI_QUOTES makes double quotes identifiers, where backslashes never escape the closing quote.
+ // Both quoting modes belong to the remote session.
+ if (sql.charAt(i) == '\\' && !(mysql && noBackslashEscapes)
+ && !(mysql && ansiQuotes && quote == '"') && quote != '`'
Review Comment:
[P2] Honor ClickHouse's escaped backticks when scanning identifiers. ClickHouse permits a backslash-escaped backtick inside a backtick-quoted identifier, but this condition excludes backtick quotes from escape skipping. In a timestamp TVF query with an identifier containing a backslash-escaped internal backtick followed by a terminal `;`, this closes the identifier early, treats its real closing backtick as a new opener, and leaves the final semicolon inside the generated `FROM (...)` subquery. The prior wrapper stripped that terminal semicolon. Apply ClickHouse identifier escaping in this branch and cover the escaped-backtick case. See the [ClickHouse syntax reference](https://clickhouse.com/docs/reference/syntax).
##########
fe/be-java-extensions/jdbc-scanner/pom.xml:
##########
@@ -35,6 +35,12 @@ under the License.
</properties>
<dependencies>
+ <dependency>
+ <groupId>com.oracle.database.jdbc</groupId>
+ <artifactId>ojdbc11</artifactId>
+ <version>23.26.2.0.0</version>
Review Comment:
[P2] Keep the configured Oracle integration driver from being shadowed. Adding `ojdbc11` to the test classpath makes `JdbcZonedTimestampIntegrationTest.withDriver` resolve `oracle.jdbc.OracleDriver` from its parent classloader before the JAR named by `oracle.integration.driverJar`; its `URLClassLoader` uses `getClass().getClassLoader()` as parent and calls `loadClass`. Consequently the legacy-driver matrix silently exercises 23.26.2 instead of the requested driver, hiding the compatibility behavior this PR needs to test. Isolate the selected driver or verify that its class actually came from the configured JAR. See [URLClassLoader parent-first delegation](https://docs.oracle.com/javase/8/docs/api/java/net/URLClassLoader.html).
##########
fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/TrinoJdbcExecutor.java:
##########
@@ -42,13 +42,22 @@
public class TrinoJdbcExecutor extends BaseJdbcExecutor {
private static final DateTimeFormatter TIMESTAMP_TZ_WRITE_FORMATTER =
DateTimeFormatter.ofPattern("uuuu-MM-dd HH:mm:ss.SSSSSS");
+ private static final DateTimeFormatter ARRAY_TIMESTAMP_FORMATTER = new java.time.format.DateTimeFormatterBuilder()
+ .appendPattern("uuuu-MM-dd HH:mm:ss")
+ .appendFraction(java.time.temporal.ChronoField.NANO_OF_SECOND, 0, 9, true)
+ .appendLiteral(' ').appendZoneOrOffsetId().toFormatter();
public TrinoJdbcExecutor(byte[] thriftParams) throws Exception {
super(thriftParams);
}
@Override
protected void setTimestampTz(int parameterIndex, LocalDateTime value) throws SQLException {
+ if (config != null && config.getTableType() == org.apache.doris.thrift.TOdbcTableType.PRESTO) {
+ // PrestoDB requires VARCHAR plus a server-side cast; its driver rejects JDBC type 2014.
+ preparedStatement.setString(parameterIndex, value.format(TIMESTAMP_TZ_WRITE_FORMATTER) + " UTC");
+ return;
Review Comment:
[P1] Preserve PRESTO zoned-timestamp writes with an old FE. For `io.prestosql.jdbc.PrestoDriver` 350, the previous JDBC 2014 `setObject` bind emits a typed zoned timestamp, while this new `setString` emits VARCHAR. An old FE still supplies raw `VALUES (?)`, without this head's matching `CAST(? AS TIMESTAMP(6) WITH TIME ZONE)`, and Presto cannot implicitly coerce that VARCHAR to the zoned timestamp column. Thus a BE-first rolling upgrade breaks previously working non-null writes. Gate the text bind on a compatible FE SQL contract or retain a version-compatible typed bind. See the [PrestoSQL 350 JDBC binder](https://github.com/trinodb/trino/blob/350/presto-jdbc/src/main/java/io/prestosql/jdbc/PrestoPreparedStatement.java) and [type coercion rules](https://github.com/trinodb/trino/blob/350/presto-main/src/main/java/io/prestosql/type/TypeCoercion.java).
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,151 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
+ }
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, jdbcType == TOdbcTableType.CLICKHOUSE);
+ } else if (c == '\'' || c == '"' || c == '`') {
+ char quote = c;
Review Comment:
[P2] Skip ClickHouse heredoc literals as a whole. ClickHouse treats `$tag$...$tag$` content as raw text. For the valid timestamp TVF source `SELECT ts FROM t WHERE length($h$foo'bar$h$) > 0;`, this scanner interprets the apostrophe inside the heredoc as an opening quote and leaves the terminal semicolon inside the generated derived table; the old wrapper stripped it. Match the closing heredoc tag before interpreting contained quotes or semicolons, and cover this literal form. See the [ClickHouse heredoc syntax](https://clickhouse.com/docs/reference/syntax).
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,151 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
+ }
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, jdbcType == TOdbcTableType.CLICKHOUSE);
+ } else if (c == '\'' || c == '"' || c == '`') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ // ANSI_QUOTES makes double quotes identifiers, where backslashes never escape the closing quote.
+ // Both quoting modes belong to the remote session.
+ if (sql.charAt(i) == '\\' && !(mysql && noBackslashEscapes)
+ && !(mysql && ansiQuotes && quote == '"') && quote != '`'
+ && jdbcType != TOdbcTableType.TRINO && jdbcType != TOdbcTableType.PRESTO) {
+ i++;
+ } else if (sql.charAt(i) == quote) {
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i++;
+ } else {
+ i++;
+ break;
+ }
+ }
+ }
+ delimiters.clear();
+ } else {
+ if (c == ';') {
+ delimiters.add(i);
+ } else {
+ delimiters.clear();
+ }
+ i++;
+ }
+ }
+ // A statement delimiter may precede trailing comments, which must stay outside SQL literals.
+ // Remove only the terminal delimiter run; interior delimiters still fail as multi-statements.
+ if (delimiters.isEmpty()) {
+ return sql;
+ }
+ StringBuilder result = new StringBuilder(sql);
+ for (int i = delimiters.size() - 1; i >= 0; i--) {
+ result.deleteCharAt(delimiters.get(i));
}
- return "SELECT " + Joiner.on(", ").join(columns) + " FROM (" + source + ") doris_jdbc_source";
+ return result.toString();
}
+ private static int skipBlockComment(String sql, int start, boolean nested) {
+ // Trino, Presto and MySQL end at the first closing marker; ClickHouse permits nested comments.
+ int depth = 1;
+ int i = start + 2;
+ while (i < sql.length() && depth > 0) {
+ if (nested && sql.startsWith("/*", i)) {
+ depth++;
+ i += 2;
+ } else if (sql.startsWith("*/", i)) {
+ depth--;
+ i += 2;
+ } else {
+ i++;
+ }
+ }
+ return i;
+ }
+
+ private static int trinoSessionQueryStart(String sql) {
+ int depth = 0;
+ int prefixWords = 0;
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (c == '\'' || c == '"') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ if (sql.charAt(i) == quote) {
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i++;
+ } else {
+ i++;
+ break;
+ }
+ }
+ }
+ } else if (sql.startsWith("--", i)) {
+ int newline = sql.indexOf('\n', i + 2);
Review Comment:
[P2] End Trino `--` comments on CR in the new `WITH SESSION` scanner. Trino permits a carriage return to terminate a line comment, but this branch searches only for `\n`. For the valid root query `WITH SESSION time_zone='UTC' -- note\rSELECT ts FROM t`, it skips the `SELECT`, returns 0, and embeds `WITH SESSION` inside `FROM (...)`, where Trino does not allow it. Use either CR or LF as the comment boundary and cover a CR-only TVF source. See the [Trino 481 SQL grammar](https://raw.githubusercontent.com/trinodb/trino/481/core/trino-grammar/src/main/antlr4/io/trino/grammar/sql/SqlBase.g4).
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,151 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
+ }
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, jdbcType == TOdbcTableType.CLICKHOUSE);
+ } else if (c == '\'' || c == '"' || c == '`') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ // ANSI_QUOTES makes double quotes identifiers, where backslashes never escape the closing quote.
+ // Both quoting modes belong to the remote session.
+ if (sql.charAt(i) == '\\' && !(mysql && noBackslashEscapes)
+ && !(mysql && ansiQuotes && quote == '"') && quote != '`'
+ && jdbcType != TOdbcTableType.TRINO && jdbcType != TOdbcTableType.PRESTO) {
+ i++;
+ } else if (sql.charAt(i) == quote) {
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i++;
+ } else {
+ i++;
+ break;
+ }
+ }
+ }
+ delimiters.clear();
+ } else {
+ if (c == ';') {
+ delimiters.add(i);
+ } else {
+ delimiters.clear();
+ }
+ i++;
+ }
+ }
+ // A statement delimiter may precede trailing comments, which must stay outside SQL literals.
+ // Remove only the terminal delimiter run; interior delimiters still fail as multi-statements.
+ if (delimiters.isEmpty()) {
+ return sql;
+ }
+ StringBuilder result = new StringBuilder(sql);
+ for (int i = delimiters.size() - 1; i >= 0; i--) {
+ result.deleteCharAt(delimiters.get(i));
}
- return "SELECT " + Joiner.on(", ").join(columns) + " FROM (" + source + ") doris_jdbc_source";
+ return result.toString();
}
+ private static int skipBlockComment(String sql, int start, boolean nested) {
+ // Trino, Presto and MySQL end at the first closing marker; ClickHouse permits nested comments.
+ int depth = 1;
+ int i = start + 2;
+ while (i < sql.length() && depth > 0) {
+ if (nested && sql.startsWith("/*", i)) {
+ depth++;
+ i += 2;
+ } else if (sql.startsWith("*/", i)) {
+ depth--;
+ i += 2;
+ } else {
+ i++;
+ }
+ }
+ return i;
+ }
+
+ private static int trinoSessionQueryStart(String sql) {
+ int depth = 0;
+ int prefixWords = 0;
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (c == '\'' || c == '"') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ if (sql.charAt(i) == quote) {
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i++;
+ } else {
+ i++;
+ break;
+ }
+ }
+ }
+ } else if (sql.startsWith("--", i)) {
+ int newline = sql.indexOf('\n', i + 2);
+ i = newline < 0 ? sql.length() : newline + 1;
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, false);
+ } else if (c == '(') {
+ depth++;
Review Comment:
[P2] Recognize a parenthesized query after Trino `WITH SESSION`. Trino's grammar permits `WITH SESSION time_zone='UTC' (SELECT ts FROM t)`, but this `(` raises `depth`, so the scanner ignores the inner `SELECT` and returns 0. The timestamp TVF wrapper then puts `WITH SESSION` inside `FROM (...)`, where Trino only permits a derived `query`. This form already failed in the old wrapper, but the new `WITH SESSION` handling explicitly added here leaves this valid query-primary form unsupported. Detect the parenthesized root query boundary and add a TVF case. See the [Trino 481 SQL grammar](https://raw.githubusercontent.com/trinodb/trino/481/core/trino-grammar/src/main/antlr4/io/trino/grammar/sql/SqlBase.g4).
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,151 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
+ }
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
Review Comment:
[P2] Recognize ClickHouse `//` line comments before scanning quotes. A valid timestamp TVF source ending with `SELECT ts FROM t // 'note` followed by a newline and `;` has a quote only in the comment. This branch does not recognize `//`, so the quote branch consumes the final semicolon and the wrapper leaves `;` inside `FROM (...)`; the previous wrapper removed it. Handle ClickHouse line-comment forms (`//`, `#!`, and `# `) and cover a comment containing an unmatched quote before a terminal delimiter. See the [ClickHouse comment syntax](https://clickhouse.com/docs/reference/syntax).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6096106126
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #68873:
URL: https://github.com/apache/doris/pull/68873
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6105353235
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (12/12)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68873_02baebff2a4ab8122e848e115a2b45416ae4b63a_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68873_02baebff2a4ab8122e848e115a2b45416ae4b63a_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.96% (31982/42663) |
| Line Coverage | 59.33% (358359/603989) |
| Region Coverage | 56.00% (298638/533265) |
| Branch Coverage | 56.91% (135539/238180) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68873:
URL: https://github.com/apache/doris/pull/68873#discussion_r4239762085
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,183 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
Review Comment:
Fixed in 02baebff2a. The TVF scan now reads SQL mode during setup from its copied JDBC URL, credentials and driver settings using a private, single-connection client, which is closed on both success and failure. EXPLAIN and Thrift serialization use the captured mode without looking up the catalog or borrowing its pool. The regression test verifies serialization after catalog access becomes unavailable, connection configuration capture, and client cleanup on a failed mode query. All 76 focused FE tests pass; the snapshot mode lookup and generated queries also passed real MariaDB 10.4 execution.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,183 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
+ }
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ } else if (jdbcType == TOdbcTableType.CLICKHOUSE
+ && (sql.startsWith("//", i) || sql.startsWith("#!", i) || sql.startsWith("# ", i))) {
+ // ClickHouse comments may contain unmatched quotes; their contents end only at LF.
+ int newline = sql.indexOf('\n', i + 2);
+ i = newline < 0 ? sql.length() : newline + 1;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, jdbcType == TOdbcTableType.CLICKHOUSE);
+ } else if (jdbcType == TOdbcTableType.CLICKHOUSE && c == '$') {
+ i = skipClickHouseHeredoc(sql, i);
+ delimiters.clear();
+ } else if (c == '\'' || c == '"' || c == '`') {
Review Comment:
Fixed in 02baebff2a. With MSSQL mode enabled, the wrapper recognizes bracket-quoted identifiers, doubled closing brackets, and literal backslashes, preserving their contents while removing only terminal delimiters. Added regression cases for an embedded apostrophe, escaped closing bracket with comment/delimiter characters, and a trailing backslash. All three generated queries passed timestamp readback on MariaDB 10.4. The FE tests and Checkstyle pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68873:
URL: https://github.com/apache/doris/pull/68873#discussion_r4237052758
##########
fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/OracleJdbcExecutor.java:
##########
@@ -50,8 +52,27 @@ public OracleJdbcExecutor(byte[] thriftParams) throws Exception {
@Override
protected void setTimestampTz(int parameterIndex, LocalDateTime value) throws SQLException {
- // An unzoned TIMESTAMP bind is session-local for both Oracle TZ and LOCAL TIME ZONE columns.
- preparedStatement.setObject(parameterIndex, value.atOffset(ZoneOffset.UTC), Types.TIMESTAMP_WITH_TIMEZONE);
+ // Older Oracle drivers reject JDBC type 2014; the sink SQL parses an explicit UTC offset.
+ preparedStatement.setString(parameterIndex, value.format(TIMESTAMP_TZ_FORMATTER) + " +00:00");
+ }
+
+ @Override
+ protected void setTimestampTzNull(int parameterIndex) throws SQLException {
+ preparedStatement.setNull(parameterIndex, Types.VARCHAR);
+ }
+
+ @Override
+ protected void initializeStatement(Connection conn, JdbcDataSourceConfig config, String sql) throws SQLException {
+ // ALTER SESSION alone leaves the driver's TSLTZ zone unset; initialize every borrowed connection.
+ try {
+ Connection physical = conn.unwrap(Connection.class);
+ Class<?> oracleConnection = Class.forName("oracle.jdbc.OracleConnection", true,
+ physical.getClass().getClassLoader());
Review Comment:
Left unchanged as requested: OceanBase Oracle-mode driver compatibility is excluded from this revision. This thread remains unresolved.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,154 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && noBackslashEscapes == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ noBackslashEscapes = catalog.getJdbcClient().isNoBackslashEscapes();
+ }
+ String source = stripTerminalDelimiter(query.trim(), mysql && noBackslashEscapes);
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ int depth = 1;
+ i += 2;
+ while (i < sql.length() && depth > 0) {
+ if (sql.startsWith("/*", i)) {
+ depth++;
+ i += 2;
+ } else if (sql.startsWith("*/", i)) {
+ depth--;
+ i += 2;
+ } else {
+ i++;
+ }
+ }
+ } else if (c == '\'' || c == '"' || c == '`') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ // SQL mode belongs to the remote connection, not the Doris session.
+ if (sql.charAt(i) == '\\' && !(mysql && noBackslashEscapes)
+ && quote != '`' && jdbcType != TOdbcTableType.TRINO && jdbcType != TOdbcTableType.PRESTO) {
Review Comment:
Fixed in 88625ed939. The wrapper now reads and caches both ANSI_QUOTES and NO_BACKSLASH_ESCAPES from the same catalog session. Added coverage for an identifier ending in a backslash, both modes together, and ordinary escaped string literals. The FE tests pass, and the wrapped ANSI_QUOTES query was also executed successfully on MySQL.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java:
##########
@@ -812,9 +812,10 @@ private static String serializePartitionValue(org.apache.paimon.types.DataType t
return null;
}
return Double.toString((Double) value);
- // case binary:
- // case varbinary: should not supported, because if return string with utf8,
- // the data maybe be corrupted
+ case BINARY:
+ case VARBINARY:
+ // Native readers reconstruct omitted partition columns from lossless hex literals.
+ return value == null ? null : "0x" + com.google.common.io.BaseEncoding.base16().encode((byte[]) value);
case DATE:
Review Comment:
Left unchanged as requested: format V1 reader changes are excluded from this revision. This thread remains unresolved.
##########
fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/OracleJdbcExecutor.java:
##########
@@ -50,8 +52,27 @@ public OracleJdbcExecutor(byte[] thriftParams) throws Exception {
@Override
protected void setTimestampTz(int parameterIndex, LocalDateTime value) throws SQLException {
- // An unzoned TIMESTAMP bind is session-local for both Oracle TZ and LOCAL TIME ZONE columns.
- preparedStatement.setObject(parameterIndex, value.atOffset(ZoneOffset.UTC), Types.TIMESTAMP_WITH_TIMEZONE);
+ // Older Oracle drivers reject JDBC type 2014; the sink SQL parses an explicit UTC offset.
+ preparedStatement.setString(parameterIndex, value.format(TIMESTAMP_TZ_FORMATTER) + " +00:00");
+ }
Review Comment:
Fixed in 88625ed939. The Oracle integration test now uses the production TO_TIMESTAMP_TZ placeholder for both TIMESTAMP WITH TIME ZONE and TIMESTAMP WITH LOCAL TIME ZONE. It also sets a conflicting NLS_TIMESTAMP_TZ_FORMAT to ensure the explicit mask is exercised. The real Oracle write/readback test passed under all three JVM time zones, including null and pre-epoch values.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,154 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && noBackslashEscapes == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ noBackslashEscapes = catalog.getJdbcClient().isNoBackslashEscapes();
+ }
+ String source = stripTerminalDelimiter(query.trim(), mysql && noBackslashEscapes);
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ int depth = 1;
+ i += 2;
+ while (i < sql.length() && depth > 0) {
+ if (sql.startsWith("/*", i)) {
+ depth++;
+ i += 2;
+ } else if (sql.startsWith("*/", i)) {
+ depth--;
+ i += 2;
+ } else {
+ i++;
+ }
+ }
+ } else if (c == '\'' || c == '"' || c == '`') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ // SQL mode belongs to the remote connection, not the Doris session.
+ if (sql.charAt(i) == '\\' && !(mysql && noBackslashEscapes)
+ && quote != '`' && jdbcType != TOdbcTableType.TRINO && jdbcType != TOdbcTableType.PRESTO) {
+ i++;
+ } else if (sql.charAt(i) == quote) {
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i++;
+ } else {
+ i++;
+ break;
+ }
+ }
+ }
+ delimiters.clear();
+ } else {
+ if (c == ';') {
+ delimiters.add(i);
+ } else {
+ delimiters.clear();
+ }
+ i++;
+ }
+ }
+ // A statement delimiter may precede trailing comments, which must stay outside SQL literals.
+ // Remove only the terminal delimiter run; interior delimiters still fail as multi-statements.
+ if (delimiters.isEmpty()) {
+ return sql;
+ }
+ StringBuilder result = new StringBuilder(sql);
+ for (int i = delimiters.size() - 1; i >= 0; i--) {
+ result.deleteCharAt(delimiters.get(i));
}
- return "SELECT " + Joiner.on(", ").join(columns) + " FROM (" + source + ") doris_jdbc_source";
+ return result.toString();
}
+ private static int trinoSessionQueryStart(String sql) {
+ int depth = 0;
+ int prefixWords = 0;
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (c == '\'' || c == '"') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ if (sql.charAt(i) == quote) {
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i++;
+ } else {
+ i++;
+ break;
+ }
+ }
+ }
+ } else if (sql.startsWith("--", i)) {
+ int newline = sql.indexOf('\n', i + 2);
+ i = newline < 0 ? sql.length() : newline + 1;
+ } else if (sql.startsWith("/*", i)) {
+ int comments = 1;
+ i += 2;
+ while (i < sql.length() && comments > 0) {
+ if (sql.startsWith("/*", i)) {
+ comments++;
Review Comment:
Fixed in 88625ed939. Both wrapper scanners share the block-comment scanner and stop at the first closing marker for Trino/Presto/MySQL. ClickHouse retains its nested-comment behavior. Tests cover the WITH SESSION prefix and terminal delimiter cases, plus a ClickHouse nested-comment case. The Trino queries were also executed successfully against a real server.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6095318424
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6096106303
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6097868560
# FE Regression Coverage Report
Increment line coverage ` 23.65% (35/148)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68873_873b1547a21a8f4769d116bd1b7631adb99e20ac_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68873_873b1547a21a8f4769d116bd1b7631adb99e20ac_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6097834400
# BE Regression && UT Coverage Report
Increment line coverage `100.00% (12/12)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68873_873b1547a21a8f4769d116bd1b7631adb99e20ac_merge/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68873_873b1547a21a8f4769d116bd1b7631adb99e20ac_merge/report/index.html)
| Category | Coverage |
|-------------------|--------------------|
| Function Coverage | 74.89% (31950/42662) |
| Line Coverage | 59.31% (358204/603985) |
| Region Coverage | 55.91% (298129/533264) |
| Branch Coverage | 56.85% (135405/238180) |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68873:
URL: https://github.com/apache/doris/pull/68873#discussion_r4237217340
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,183 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
Review Comment:
[P2] Bind the session mode to the TVF's catalog snapshot before scan serialization. `getTvfQuery()` now looks up the catalog and its JDBC client after `getScanNode` has configured the scan table. A concurrent `ALTER CATALOG` can close/null that client before this call, causing an NPE or closed-pool failure; if the catalog is reinitialized first, the mode can instead come from a new JDBC URL while the scan table still uses the old one. A concurrent DROP can also remove the catalog ID. Capture the mode with the same configuration and safe client lifetime used to configure the scan table, then reuse it during `toThrift`; cover a reset between TVF setup and serialization.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,183 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
+ }
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ } else if (jdbcType == TOdbcTableType.CLICKHOUSE
+ && (sql.startsWith("//", i) || sql.startsWith("#!", i) || sql.startsWith("# ", i))) {
+ // ClickHouse comments may contain unmatched quotes; their contents end only at LF.
+ int newline = sql.indexOf('\n', i + 2);
+ i = newline < 0 ? sql.length() : newline + 1;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, jdbcType == TOdbcTableType.CLICKHOUSE);
+ } else if (jdbcType == TOdbcTableType.CLICKHOUSE && c == '$') {
+ i = skipClickHouseHeredoc(sql, i);
+ delimiters.clear();
+ } else if (c == '\'' || c == '"' || c == '`') {
Review Comment:
[P2] Scan MariaDB bracket identifiers when stripping TVF delimiters. MariaDB connections map to MYSQL here, and with `sql_mode=MSSQL` the query `SELECT ts AS [x'y] FROM t;` has a valid bracket quoted alias. For a mapped TIMESTAMP TVF, this scanner treats the apostrophe inside `[...]` as an opening string quote, misses the terminal `;`, and emits `FROM (SELECT ts AS [x'y] FROM t;\n)`, which is invalid. The previous wrapper removed the final semicolon. Handle bracket quoted identifiers (including escaped `]]`) when MSSQL mode is active and cover this TVF case. MariaDB documents [identifier quoting](https://mariadb.com/docs/server/reference/sql-structure/sql-language-structure/identifier-names) and [MSSQL mode](https://mariadb.com/docs/server/server-management/variables-and-modes/sql_mode).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6095320899
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6095319139
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6095736047
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68873:
URL: https://github.com/apache/doris/pull/68873#discussion_r4237171050
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,151 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
+ }
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, jdbcType == TOdbcTableType.CLICKHOUSE);
+ } else if (c == '\'' || c == '"' || c == '`') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ // ANSI_QUOTES makes double quotes identifiers, where backslashes never escape the closing quote.
+ // Both quoting modes belong to the remote session.
+ if (sql.charAt(i) == '\\' && !(mysql && noBackslashEscapes)
+ && !(mysql && ansiQuotes && quote == '"') && quote != '`'
+ && jdbcType != TOdbcTableType.TRINO && jdbcType != TOdbcTableType.PRESTO) {
+ i++;
+ } else if (sql.charAt(i) == quote) {
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i++;
+ } else {
+ i++;
+ break;
+ }
+ }
+ }
+ delimiters.clear();
+ } else {
+ if (c == ';') {
+ delimiters.add(i);
+ } else {
+ delimiters.clear();
+ }
+ i++;
+ }
+ }
+ // A statement delimiter may precede trailing comments, which must stay outside SQL literals.
+ // Remove only the terminal delimiter run; interior delimiters still fail as multi-statements.
+ if (delimiters.isEmpty()) {
+ return sql;
+ }
+ StringBuilder result = new StringBuilder(sql);
+ for (int i = delimiters.size() - 1; i >= 0; i--) {
+ result.deleteCharAt(delimiters.get(i));
}
- return "SELECT " + Joiner.on(", ").join(columns) + " FROM (" + source + ") doris_jdbc_source";
+ return result.toString();
}
+ private static int skipBlockComment(String sql, int start, boolean nested) {
+ // Trino, Presto and MySQL end at the first closing marker; ClickHouse permits nested comments.
+ int depth = 1;
+ int i = start + 2;
+ while (i < sql.length() && depth > 0) {
+ if (nested && sql.startsWith("/*", i)) {
+ depth++;
+ i += 2;
+ } else if (sql.startsWith("*/", i)) {
+ depth--;
+ i += 2;
+ } else {
+ i++;
+ }
+ }
+ return i;
+ }
+
+ private static int trinoSessionQueryStart(String sql) {
+ int depth = 0;
+ int prefixWords = 0;
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (c == '\'' || c == '"') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ if (sql.charAt(i) == quote) {
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i++;
+ } else {
+ i++;
+ break;
+ }
+ }
+ }
+ } else if (sql.startsWith("--", i)) {
+ int newline = sql.indexOf('\n', i + 2);
Review Comment:
Left unchanged per the requested scope. WITH SESSION queries were already wrapped incorrectly before this PR; this CR-only variant is a remaining gap in the new support, rather than a regression of a previously working query. This revision is limited to newly introduced regressions.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,151 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
+ }
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, jdbcType == TOdbcTableType.CLICKHOUSE);
+ } else if (c == '\'' || c == '"' || c == '`') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ // ANSI_QUOTES makes double quotes identifiers, where backslashes never escape the closing quote.
+ // Both quoting modes belong to the remote session.
+ if (sql.charAt(i) == '\\' && !(mysql && noBackslashEscapes)
+ && !(mysql && ansiQuotes && quote == '"') && quote != '`'
+ && jdbcType != TOdbcTableType.TRINO && jdbcType != TOdbcTableType.PRESTO) {
+ i++;
+ } else if (sql.charAt(i) == quote) {
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i++;
+ } else {
+ i++;
+ break;
+ }
+ }
+ }
+ delimiters.clear();
+ } else {
+ if (c == ';') {
+ delimiters.add(i);
+ } else {
+ delimiters.clear();
+ }
+ i++;
+ }
+ }
+ // A statement delimiter may precede trailing comments, which must stay outside SQL literals.
+ // Remove only the terminal delimiter run; interior delimiters still fail as multi-statements.
+ if (delimiters.isEmpty()) {
+ return sql;
+ }
+ StringBuilder result = new StringBuilder(sql);
+ for (int i = delimiters.size() - 1; i >= 0; i--) {
+ result.deleteCharAt(delimiters.get(i));
}
- return "SELECT " + Joiner.on(", ").join(columns) + " FROM (" + source + ") doris_jdbc_source";
+ return result.toString();
}
+ private static int skipBlockComment(String sql, int start, boolean nested) {
+ // Trino, Presto and MySQL end at the first closing marker; ClickHouse permits nested comments.
+ int depth = 1;
+ int i = start + 2;
+ while (i < sql.length() && depth > 0) {
+ if (nested && sql.startsWith("/*", i)) {
+ depth++;
+ i += 2;
+ } else if (sql.startsWith("*/", i)) {
+ depth--;
+ i += 2;
+ } else {
+ i++;
+ }
+ }
+ return i;
+ }
+
+ private static int trinoSessionQueryStart(String sql) {
+ int depth = 0;
+ int prefixWords = 0;
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (c == '\'' || c == '"') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ if (sql.charAt(i) == quote) {
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i++;
+ } else {
+ i++;
+ break;
+ }
+ }
+ }
+ } else if (sql.startsWith("--", i)) {
+ int newline = sql.indexOf('\n', i + 2);
+ i = newline < 0 ? sql.length() : newline + 1;
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, false);
+ } else if (c == '(') {
+ depth++;
Review Comment:
Left unchanged per the requested scope. The parenthesized WITH SESSION form already failed before this PR, as noted in the finding. Preexisting bugs are excluded from this revision.
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcTable.java:
##########
@@ -152,7 +152,18 @@ public String getInsertSql(List<String> insertCols) {
if (i != 0) {
sb.append(", ");
}
- sb.append("?");
+ Column column = getColumn(insertCols.get(i));
+ TOdbcTableType remoteType = getJdbcTableType();
+ // Legacy drivers bind zoned timestamps as VARCHAR; parse their explicit offsets on the server.
+ if (column != null && column.getType().isTimeStampTz()
+ && (remoteType == TOdbcTableType.ORACLE || remoteType == TOdbcTableType.OCEANBASE_ORACLE)) {
+ sb.append("TO_TIMESTAMP_TZ(?, 'YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM')");
+ } else if (column != null && column.getType().isTimeStampTz() && remoteType == TOdbcTableType.PRESTO) {
Review Comment:
Left unchanged per the requested scope: mixed-version FE/BE compatibility fixes are excluded. This thread remains unresolved.
##########
fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/TrinoJdbcExecutor.java:
##########
@@ -42,13 +42,22 @@
public class TrinoJdbcExecutor extends BaseJdbcExecutor {
private static final DateTimeFormatter TIMESTAMP_TZ_WRITE_FORMATTER =
DateTimeFormatter.ofPattern("uuuu-MM-dd HH:mm:ss.SSSSSS");
+ private static final DateTimeFormatter ARRAY_TIMESTAMP_FORMATTER = new java.time.format.DateTimeFormatterBuilder()
+ .appendPattern("uuuu-MM-dd HH:mm:ss")
+ .appendFraction(java.time.temporal.ChronoField.NANO_OF_SECOND, 0, 9, true)
+ .appendLiteral(' ').appendZoneOrOffsetId().toFormatter();
public TrinoJdbcExecutor(byte[] thriftParams) throws Exception {
super(thriftParams);
}
@Override
protected void setTimestampTz(int parameterIndex, LocalDateTime value) throws SQLException {
+ if (config != null && config.getTableType() == org.apache.doris.thrift.TOdbcTableType.PRESTO) {
+ // PrestoDB requires VARCHAR plus a server-side cast; its driver rejects JDBC type 2014.
+ preparedStatement.setString(parameterIndex, value.format(TIMESTAMP_TZ_WRITE_FORMATTER) + " UTC");
+ return;
Review Comment:
Left unchanged per the requested scope: mixed-version FE/BE compatibility fixes are excluded. This thread remains unresolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6105435897
# FE Regression Coverage Report
Increment line coverage ` 28.95% (44/152)` :tada:
[Increment coverage report](http://coverage.selectdb-in.cc/coverage/68873_02baebff2a4ab8122e848e115a2b45416ae4b63a_merge_fe/increment_report/index.html)
[Complete coverage report](http://coverage.selectdb-in.cc/coverage/68873_02baebff2a4ab8122e848e115a2b45416ae4b63a_merge_fe/report/index.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6095735869
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68873:
URL: https://github.com/apache/doris/pull/68873#discussion_r4237013943
##########
fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/OracleJdbcExecutor.java:
##########
@@ -50,8 +52,27 @@ public OracleJdbcExecutor(byte[] thriftParams) throws Exception {
@Override
protected void setTimestampTz(int parameterIndex, LocalDateTime value) throws SQLException {
- // An unzoned TIMESTAMP bind is session-local for both Oracle TZ and LOCAL TIME ZONE columns.
- preparedStatement.setObject(parameterIndex, value.atOffset(ZoneOffset.UTC), Types.TIMESTAMP_WITH_TIMEZONE);
+ // Older Oracle drivers reject JDBC type 2014; the sink SQL parses an explicit UTC offset.
+ preparedStatement.setString(parameterIndex, value.format(TIMESTAMP_TZ_FORMATTER) + " +00:00");
+ }
+
+ @Override
+ protected void setTimestampTzNull(int parameterIndex) throws SQLException {
+ preparedStatement.setNull(parameterIndex, Types.VARCHAR);
+ }
+
+ @Override
+ protected void initializeStatement(Connection conn, JdbcDataSourceConfig config, String sql) throws SQLException {
+ // ALTER SESSION alone leaves the driver's TSLTZ zone unset; initialize every borrowed connection.
+ try {
+ Connection physical = conn.unwrap(Connection.class);
+ Class<?> oracleConnection = Class.forName("oracle.jdbc.OracleConnection", true,
+ physical.getClass().getClassLoader());
Review Comment:
[P1] Preserve OceanBase Oracle-mode connections. `OCEANBASE_ORACLE` is dispatched to this executor, but its supported `com.oceanbase.jdbc.Driver` connection does not implement `oracle.jdbc.OracleConnection`. This unconditional load/unwrap therefore fails before every scan or write statement (even for tables without TIMESTAMPTZ). Apply this Oracle-driver-specific setter only to actual Oracle connections and use a supported session-zone path for OceanBase.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonUtil.java:
##########
@@ -812,9 +812,10 @@ private static String serializePartitionValue(org.apache.paimon.types.DataType t
return null;
}
return Double.toString((Double) value);
- // case binary:
- // case varbinary: should not supported, because if return string with utf8,
- // the data maybe be corrupted
+ case BINARY:
+ case VARBINARY:
+ // Native readers reconstruct omitted partition columns from lossless hex literals.
+ return value == null ? null : "0x" + com.google.common.io.BaseEncoding.base16().encode((byte[]) value);
case DATE:
Review Comment:
[P1] Decode binary partition hex in the V1 ORC reader. For a native Paimon ORC-only scan with `enable_file_scanner_v2=false`, this sends `0x00FF80` through `columns_from_path`, but V1's ORC reader uses the partition filler and VARBINARY JSON deserializer, which inserts those eight ASCII bytes instead of `00 FF 80`. V2 uses `from_string` and decodes correctly. Make the V1 partition fill decode the typed hex and add a V1 readback test.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,154 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && noBackslashEscapes == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ noBackslashEscapes = catalog.getJdbcClient().isNoBackslashEscapes();
+ }
+ String source = stripTerminalDelimiter(query.trim(), mysql && noBackslashEscapes);
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ int depth = 1;
+ i += 2;
+ while (i < sql.length() && depth > 0) {
+ if (sql.startsWith("/*", i)) {
+ depth++;
+ i += 2;
+ } else if (sql.startsWith("*/", i)) {
+ depth--;
+ i += 2;
+ } else {
+ i++;
+ }
+ }
+ } else if (c == '\'' || c == '"' || c == '`') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ // SQL mode belongs to the remote connection, not the Doris session.
+ if (sql.charAt(i) == '\\' && !(mysql && noBackslashEscapes)
+ && quote != '`' && jdbcType != TOdbcTableType.TRINO && jdbcType != TOdbcTableType.PRESTO) {
+ i++;
+ } else if (sql.charAt(i) == quote) {
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i++;
+ } else {
+ i++;
+ break;
+ }
+ }
+ }
+ delimiters.clear();
+ } else {
+ if (c == ';') {
+ delimiters.add(i);
+ } else {
+ delimiters.clear();
+ }
+ i++;
+ }
+ }
+ // A statement delimiter may precede trailing comments, which must stay outside SQL literals.
+ // Remove only the terminal delimiter run; interior delimiters still fail as multi-statements.
+ if (delimiters.isEmpty()) {
+ return sql;
+ }
+ StringBuilder result = new StringBuilder(sql);
+ for (int i = delimiters.size() - 1; i >= 0; i--) {
+ result.deleteCharAt(delimiters.get(i));
}
- return "SELECT " + Joiner.on(", ").join(columns) + " FROM (" + source + ") doris_jdbc_source";
+ return result.toString();
}
+ private static int trinoSessionQueryStart(String sql) {
+ int depth = 0;
+ int prefixWords = 0;
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (c == '\'' || c == '"') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ if (sql.charAt(i) == quote) {
+ if (i + 1 < sql.length() && sql.charAt(i + 1) == quote) {
+ i++;
+ } else {
+ i++;
+ break;
+ }
+ }
+ }
+ } else if (sql.startsWith("--", i)) {
+ int newline = sql.indexOf('\n', i + 2);
+ i = newline < 0 ? sql.length() : newline + 1;
+ } else if (sql.startsWith("/*", i)) {
+ int comments = 1;
+ i += 2;
+ while (i < sql.length() && comments > 0) {
+ if (sql.startsWith("/*", i)) {
+ comments++;
Review Comment:
[P2] Match the remote block-comment rule in both wrapper scanners. Trino ends `/* note /* marker */` at its first `*/`, so `WITH SESSION time_zone='UTC' /* note /* marker */ SELECT ts FROM t` is a valid root query. This loop counts the inner `/*` as nested, never sees the later `SELECT`, and returns 0; `getTvfQuery` then places `WITH SESSION` inside `FROM (...)`, where it is invalid. The same nesting assumption in `stripTerminalDelimiter` can hide a trailing semicolon. Stop at the first `*/` in both scanners and add both TVF cases.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,154 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && noBackslashEscapes == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ noBackslashEscapes = catalog.getJdbcClient().isNoBackslashEscapes();
+ }
+ String source = stripTerminalDelimiter(query.trim(), mysql && noBackslashEscapes);
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ int depth = 1;
+ i += 2;
+ while (i < sql.length() && depth > 0) {
+ if (sql.startsWith("/*", i)) {
+ depth++;
+ i += 2;
+ } else if (sql.startsWith("*/", i)) {
+ depth--;
+ i += 2;
+ } else {
+ i++;
+ }
+ }
+ } else if (c == '\'' || c == '"' || c == '`') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ // SQL mode belongs to the remote connection, not the Doris session.
+ if (sql.charAt(i) == '\\' && !(mysql && noBackslashEscapes)
+ && quote != '`' && jdbcType != TOdbcTableType.TRINO && jdbcType != TOdbcTableType.PRESTO) {
Review Comment:
[P2] Handle `ANSI_QUOTES` when scanning double quotes. With MySQL `sql_mode=ANSI_QUOTES`, `SELECT ts, 1 AS "x\" FROM t;` is valid: the alias ends in a literal backslash. This branch skips its closing `"` because `NO_BACKSLASH_ESCAPES` is absent, so `stripTerminalDelimiter` leaves the semicolon and `getTvfQuery` places `... FROM t;` inside the derived table. Timestamp TVF scans then fail. Read the identifier-quoting mode as well and test this case.
##########
fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/TrinoJdbcExecutor.java:
##########
@@ -42,13 +42,22 @@
public class TrinoJdbcExecutor extends BaseJdbcExecutor {
private static final DateTimeFormatter TIMESTAMP_TZ_WRITE_FORMATTER =
DateTimeFormatter.ofPattern("uuuu-MM-dd HH:mm:ss.SSSSSS");
+ private static final DateTimeFormatter ARRAY_TIMESTAMP_FORMATTER = new java.time.format.DateTimeFormatterBuilder()
+ .appendPattern("uuuu-MM-dd HH:mm:ss")
+ .appendFraction(java.time.temporal.ChronoField.NANO_OF_SECOND, 0, 9, true)
+ .appendLiteral(' ').appendZoneOrOffsetId().toFormatter();
public TrinoJdbcExecutor(byte[] thriftParams) throws Exception {
super(thriftParams);
}
@Override
protected void setTimestampTz(int parameterIndex, LocalDateTime value) throws SQLException {
+ if (config != null && config.getTableType() == org.apache.doris.thrift.TOdbcTableType.PRESTO) {
+ // PrestoDB requires VARCHAR plus a server-side cast; its driver rejects JDBC type 2014.
Review Comment:
[P2] Exercise the Presto branch in the server integration tests. Both `testPrestoTimestampWriteRoundTrip` and `testPrestoUtcProjectionPreservesNamedZoneOverlap` create partial `TrinoJdbcExecutor` mocks with `config == null`, so this `PRESTO` branch and the new scalar `getTimestamp` branch are skipped. The write helper also uses raw `?` instead of the new FE `CAST` placeholder. Set the test executor's table type to PRESTO and use the production cast so these tests validate the driver path this change adds.
##########
fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/OracleJdbcExecutor.java:
##########
@@ -50,8 +52,27 @@ public OracleJdbcExecutor(byte[] thriftParams) throws Exception {
@Override
protected void setTimestampTz(int parameterIndex, LocalDateTime value) throws SQLException {
- // An unzoned TIMESTAMP bind is session-local for both Oracle TZ and LOCAL TIME ZONE columns.
- preparedStatement.setObject(parameterIndex, value.atOffset(ZoneOffset.UTC), Types.TIMESTAMP_WITH_TIMEZONE);
+ // Older Oracle drivers reject JDBC type 2014; the sink SQL parses an explicit UTC offset.
+ preparedStatement.setString(parameterIndex, value.format(TIMESTAMP_TZ_FORMATTER) + " +00:00");
+ }
Review Comment:
[P2] Update the Oracle write integration test for the text bind. `testOracleTimestampWriteRoundTrip` still prepares raw `INSERT ... VALUES (?)` and calls this setter, while only `JdbcTable.getInsertSql` now wraps the placeholder in `TO_TIMESTAMP_TZ(..., 'YYYY-MM-DD HH24:MI:SS.FF6 TZH:TZM')`. With a different `NLS_TIMESTAMP_TZ_FORMAT`, the conditional test's implicit conversion fails and never validates the production write path. Use the explicit cast in that test for both TZ and local TZ columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68873:
URL: https://github.com/apache/doris/pull/68873#discussion_r4239783790
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +233,178 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"),
+ mysql && mysqlSqlMode.contains("MSSQL"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes, boolean mssql) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ } else if (jdbcType == TOdbcTableType.CLICKHOUSE
+ && (sql.startsWith("//", i) || sql.startsWith("#!", i) || sql.startsWith("# ", i))) {
+ // ClickHouse comments may contain unmatched quotes; their contents end only at LF.
+ int newline = sql.indexOf('\n', i + 2);
+ i = newline < 0 ? sql.length() : newline + 1;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, jdbcType == TOdbcTableType.CLICKHOUSE);
Review Comment:
[P2] Scan ClickHouse Unicode-quoted tokens before ASCII quotes. ClickHouse v25.8 accepts `“a'b”` as a quoted identifier. For a TIMESTAMPTZ TVF source `SELECT ts, 1 AS “a'b” FROM t;`, this scanner treats the apostrophe inside the identifier as an opening string quote and leaves the terminal `;`, generating `FROM (SELECT ts, 1 AS “a'b” FROM t;\n)`, which is invalid. The previous wrapper removed the final delimiter. Recognize ClickHouse's Unicode quote pairs as whole tokens and add a TVF case. This is a distinct lexical form from the reported bracket, heredoc, comment, and backtick cases. See the [ClickHouse lexer](https://raw.githubusercontent.com/ClickHouse/ClickHouse/v25.8.1.5101-lts/src/Parsers/Lexer.cpp).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6119203227
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68873:
URL: https://github.com/apache/doris/pull/68873#discussion_r4237053272
##########
fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/TrinoJdbcExecutor.java:
##########
@@ -42,13 +42,22 @@
public class TrinoJdbcExecutor extends BaseJdbcExecutor {
private static final DateTimeFormatter TIMESTAMP_TZ_WRITE_FORMATTER =
DateTimeFormatter.ofPattern("uuuu-MM-dd HH:mm:ss.SSSSSS");
+ private static final DateTimeFormatter ARRAY_TIMESTAMP_FORMATTER = new java.time.format.DateTimeFormatterBuilder()
+ .appendPattern("uuuu-MM-dd HH:mm:ss")
+ .appendFraction(java.time.temporal.ChronoField.NANO_OF_SECOND, 0, 9, true)
+ .appendLiteral(' ').appendZoneOrOffsetId().toFormatter();
public TrinoJdbcExecutor(byte[] thriftParams) throws Exception {
super(thriftParams);
}
@Override
protected void setTimestampTz(int parameterIndex, LocalDateTime value) throws SQLException {
+ if (config != null && config.getTableType() == org.apache.doris.thrift.TOdbcTableType.PRESTO) {
+ // PrestoDB requires VARCHAR plus a server-side cast; its driver rejects JDBC type 2014.
Review Comment:
Fixed in 88625ed939. Both Presto integration paths now set tableType=PRESTO, exercising the VARCHAR binder and untyped getTimestamp reader. The write helper uses the production CAST placeholder, and readback no longer calls unsupported typed getObject on the Presto branch. Both tests passed with PrestoSQL JDBC 350 against Trino 481 configured for the Presto protocol, under all three JVM time zones. A standalone PrestoDB server was not used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on code in PR #68873:
URL: https://github.com/apache/doris/pull/68873#discussion_r4237170604
##########
fe/be-java-extensions/jdbc-scanner/pom.xml:
##########
@@ -35,6 +35,12 @@ under the License.
</properties>
<dependencies>
+ <dependency>
+ <groupId>com.oracle.database.jdbc</groupId>
+ <artifactId>ojdbc11</artifactId>
+ <version>23.26.2.0.0</version>
Review Comment:
Fixed in 873b1547a2. Oracle integration drivers are now loaded with the JDBC API classloader as parent, excluding the Oracle mock dependency. The helper checks the actual source JAR before connecting. Added a regression test that fails with the old parent-first setup; also verified isolated loading and instantiation of Oracle JDBC 11.2 and 23.26 from their respective configured JARs. No driver compatibility changes are included.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,151 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
+ }
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, jdbcType == TOdbcTableType.CLICKHOUSE);
+ } else if (c == '\'' || c == '"' || c == '`') {
+ char quote = c;
+ for (i++; i < sql.length(); i++) {
+ // ANSI_QUOTES makes double quotes identifiers, where backslashes never escape the closing quote.
+ // Both quoting modes belong to the remote session.
+ if (sql.charAt(i) == '\\' && !(mysql && noBackslashEscapes)
+ && !(mysql && ansiQuotes && quote == '"') && quote != '`'
Review Comment:
Fixed in 873b1547a2. ClickHouse backtick identifiers now honor backslash escapes. Regression cases cover an escaped backtick and a semicolon inside the identifier. Generated queries were executed successfully with ClickHouse 23.8.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,151 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
+ }
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
Review Comment:
Fixed in 873b1547a2. The wrapper skips ClickHouse //, #!, and # space comments before interpreting their contents. Tests cover unmatched quotes, LF/CRLF, and terminal delimiters. Actual ClickHouse execution also covers comments after the terminal delimiter.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/source/JdbcScanNode.java:
##########
@@ -213,13 +219,151 @@ private String getTvfQuery() {
if (!projectsTimestamps) {
return query;
}
- String source = query.trim();
- if (source.endsWith(";")) {
- source = source.substring(0, source.length() - 1);
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ if (mysql && mysqlSqlMode == null) {
+ // Resolve only for wrapped MySQL queries and reuse the result for EXPLAIN and scan serialization.
+ org.apache.doris.datasource.jdbc.JdbcExternalCatalog catalog =
+ (org.apache.doris.datasource.jdbc.JdbcExternalCatalog) org.apache.doris.catalog.Env
+ .getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ mysqlSqlMode = catalog.getJdbcClient().getSessionSqlMode();
+ }
+ String source = stripTerminalDelimiter(query.trim(),
+ mysql && mysqlSqlMode.contains("NO_BACKSLASH_ESCAPES"), mysql && mysqlSqlMode.contains("ANSI_QUOTES"));
+ // WITH SESSION must stay outside the derived table; terminate trailing line comments with a newline.
+ int start = jdbcType == TOdbcTableType.TRINO ? trinoSessionQueryStart(source) : 0;
+ return source.substring(0, start) + "SELECT " + Joiner.on(", ").join(columns)
+ + " FROM (" + source.substring(start) + "\n) doris_jdbc_source";
+ }
+
+ private String stripTerminalDelimiter(String sql, boolean noBackslashEscapes, boolean ansiQuotes) {
+ boolean mysql = jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE;
+ List<Integer> delimiters = new java.util.ArrayList<>();
+ for (int i = 0; i < sql.length();) {
+ char c = sql.charAt(i);
+ if (Character.isWhitespace(c)) {
+ i++;
+ // MySQL's second dash needs a following whitespace/control character; --1 is arithmetic.
+ } else if ((sql.startsWith("--", i) && (!mysql || (i + 2 < sql.length()
+ && (Character.isWhitespace(sql.charAt(i + 2)) || Character.isISOControl(sql.charAt(i + 2))))))
+ || (c == '#'
+ && (jdbcType == TOdbcTableType.MYSQL || jdbcType == TOdbcTableType.OCEANBASE))) {
+ while (i < sql.length() && sql.charAt(i) != '\n' && sql.charAt(i) != '\r') {
+ i++;
+ }
+ } else if (sql.startsWith("/*", i)) {
+ i = skipBlockComment(sql, i, jdbcType == TOdbcTableType.CLICKHOUSE);
+ } else if (c == '\'' || c == '"' || c == '`') {
+ char quote = c;
Review Comment:
Fixed in 873b1547a2. The wrapper skips matched ClickHouse heredoc tags as raw literals, preserving embedded quotes, comment markers and semicolons. Tests cover named/empty tags, empty payloads, and preservation of interior statement delimiters. Generated queries passed actual ClickHouse execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6103900790
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [branch-4.1](fix) Correct external binary partitions and timestamp handling [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68873:
URL: https://github.com/apache/doris/pull/68873#issuecomment-6103900587
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org