You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2024/02/06 00:36:10 UTC
(doris) branch branch-2.1 updated (cde74cbfeef -> 4701dd49c36)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
from cde74cbfeef [feature](test)Add mtmv agg under join cases (#30793)
new 06ed5780e4f [opt](catalog) cache the converted properties (#30668)
new 92226c986a1 [fix](catalog) fix data_sub/data_add func pushdown in jdbcscan (#30807)
new 8e147f4c93e [BugFix](MultiCatalog) Fix oss file location is not avaiable in iceberg hadoop catalog (#30761)
new bc2e8ac8f97 [fix](kerberos) fix kerberos ugi login method (#30766)
new 4701dd49c36 (selectdb-cloud) Use info level in recordCreatePartitionFailedMsg() due to intersection happens all the time (#30448)
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../src/main/java/org/apache/doris/hudi/Utils.java | 1 +
.../doris/catalog/HiveMetaStoreClientHelper.java | 1 +
.../doris/clone/DynamicPartitionScheduler.java | 2 +-
.../org/apache/doris/common/util/LocationPath.java | 191 ++++++++++++---------
.../apache/doris/datasource/ExternalCatalog.java | 20 ++-
.../planner/external/iceberg/IcebergScanNode.java | 9 +-
.../external/jdbc/JdbcFunctionPushDownRule.java | 64 +++++--
.../apache/doris/common/util/LocationPathTest.java | 14 ++
.../datasource/jdbc/JdbcExternalCatalogTest.java | 1 +
.../jdbc/test_mysql_jdbc_catalog.out | 42 +++++
.../jdbc/test_mysql_jdbc_catalog.groovy | 89 ++++++++++
11 files changed, 334 insertions(+), 100 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 01/05: [opt](catalog) cache the converted properties (#30668)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 06ed5780e4f992289fbe40006653abddd76086db
Author: Mingyu Chen <mo...@163.com>
AuthorDate: Mon Feb 5 22:43:33 2024 +0800
[opt](catalog) cache the converted properties (#30668)
convert properties may be a heavy operation, so we cache the result.
---
.../org/apache/doris/datasource/ExternalCatalog.java | 20 +++++++++++++++++++-
.../datasource/jdbc/JdbcExternalCatalogTest.java | 1 +
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
index 6fcd495b67f..c5408dd3103 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java
@@ -104,6 +104,10 @@ public abstract class ExternalCatalog
private ExternalSchemaCache schemaCache;
private String comment;
+ // A cached and being converted properties for external catalog.
+ // generated from catalog properties.
+ private byte[] propLock = new byte[0];
+ private Map<String, String> convertedProperties = null;
public ExternalCatalog() {
}
@@ -298,6 +302,9 @@ public abstract class ExternalCatalog
public void onRefresh(boolean invalidCache) {
this.objectCreated = false;
this.initialized = false;
+ synchronized (this.propLock) {
+ this.convertedProperties = null;
+ }
this.invalidCacheInInit = invalidCache;
if (invalidCache) {
Env.getCurrentEnv().getExtMetaCacheMgr().invalidateCatalogCache(id);
@@ -421,7 +428,17 @@ public abstract class ExternalCatalog
@Override
public Map<String, String> getProperties() {
- return PropertyConverter.convertToMetaProperties(catalogProperty.getProperties());
+ // convert properties may be a heavy operation, so we cache the result.
+ if (convertedProperties != null) {
+ return convertedProperties;
+ }
+ synchronized (propLock) {
+ if (convertedProperties != null) {
+ return convertedProperties;
+ }
+ convertedProperties = PropertyConverter.convertToMetaProperties(catalogProperty.getProperties());
+ return convertedProperties;
+ }
}
@Override
@@ -552,6 +569,7 @@ public abstract class ExternalCatalog
}
}
}
+ this.propLock = new byte[0];
}
public void addDatabaseForTest(ExternalDatabase<? extends ExternalTable> db) {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalogTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalogTest.java
index 7bc268b6421..8394daf0682 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalogTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalogTest.java
@@ -54,6 +54,7 @@ public class JdbcExternalCatalogTest {
Map<String, String> newProperties = Maps.newHashMap();
newProperties.put(JdbcResource.CONNECTION_POOL_MIN_SIZE, "2");
jdbcExternalCatalog.getCatalogProperty().modifyCatalogProps(newProperties);
+ jdbcExternalCatalog.notifyPropertiesUpdated(newProperties);
JdbcExternalCatalog replayJdbcCatalog2 = (JdbcExternalCatalog) CatalogFactory.createFromLog(
jdbcExternalCatalog.constructEditLog());
Map<String, String> properties2 = replayJdbcCatalog2.getProperties();
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 05/05: (selectdb-cloud) Use info level in recordCreatePartitionFailedMsg() due to intersection happens all the time (#30448)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 4701dd49c3608d3e914fb1631e18991f87541854
Author: Gavin Chou <ga...@gmail.com>
AuthorDate: Tue Feb 6 08:34:10 2024 +0800
(selectdb-cloud) Use info level in recordCreatePartitionFailedMsg() due to intersection happens all the time (#30448)
---
.../src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
index 565f4c066f8..81f8cc7cd29 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
@@ -588,7 +588,7 @@ public class DynamicPartitionScheduler extends MasterDaemon {
}
private void recordCreatePartitionFailedMsg(String dbName, String tableName, String msg, long tableId) {
- LOG.warn("dynamic add partition failed: {}, db: {}, table: {}", msg, dbName, tableName);
+ LOG.info("dynamic add partition failed: {}, db: {}, table: {}", msg, dbName, tableName);
createOrUpdateRuntimeInfo(tableId, DYNAMIC_PARTITION_STATE, State.ERROR.toString());
createOrUpdateRuntimeInfo(tableId, CREATE_PARTITION_MSG, msg);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 03/05: [BugFix](MultiCatalog) Fix oss file location is not avaiable in iceberg hadoop catalog (#30761)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 8e147f4c93eff20be5668b7498efc958fb29eed8
Author: GoGoWen <82...@users.noreply.github.com>
AuthorDate: Tue Feb 6 08:31:59 2024 +0800
[BugFix](MultiCatalog) Fix oss file location is not avaiable in iceberg hadoop catalog (#30761)
1, create iceberg hadoop catalog like below:
CREATE CATALOG iceberg_catalog PROPERTIES (
"warehouse" = "s3a://xxx/xxx",
"type" = "iceberg",
"s3.secret_key" = "*XXX",
"s3.region" = "region",
"s3.endpoint" = "http://xxx.jd.local",
"s3.bucket" = "xxx-test",
"s3.access_key" = "xxxxx",
"iceberg.catalog.type" = "hadoop",
"fs.s3a.impl" = "org.apache.hadoop.fs.s3a.S3AFileSystem",
"create_time" = "2024-02-02 11:15:28.570"
);
2, run select * from iceberg_catalog.table limit 1;
will get errCode = 2, detailMessage = Unknown file location nullnulls3a:/xxxx
expect:
OK
also need to bp to branch-2.0
---
.../org/apache/doris/common/util/LocationPath.java | 191 ++++++++++++---------
.../planner/external/iceberg/IcebergScanNode.java | 9 +-
.../apache/doris/common/util/LocationPathTest.java | 14 ++
3 files changed, 127 insertions(+), 87 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java b/fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java
index 0ddba406cdc..fd7da29e519 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java
@@ -39,6 +39,8 @@ import java.net.URISyntaxException;
import java.net.URLDecoder;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
+import java.nio.file.InvalidPathException;
+import java.nio.file.Paths;
import java.util.HashMap;
import java.util.Map;
@@ -49,7 +51,7 @@ public class LocationPath {
private final LocationType locationType;
private final String location;
- enum LocationType {
+ public enum LocationType {
HDFS,
LOCAL, // Local File
BOS, // Baidu
@@ -66,7 +68,8 @@ public class LocationPath {
S3A,
S3N,
VIEWFS,
- UNKNOWN
+ UNKNOWN,
+ NOSCHEME // no scheme info
}
private LocationPath(String location) {
@@ -75,107 +78,123 @@ public class LocationPath {
public LocationPath(String location, Map<String, String> props) {
String scheme = parseScheme(location).toLowerCase();
- switch (scheme) {
- case FeConstants.FS_PREFIX_HDFS:
- locationType = LocationType.HDFS;
- // Need add hdfs host to location
- String host = props.get(HdfsResource.DSF_NAMESERVICES);
- this.location = normalizedHdfsPath(location, host);
- break;
- case FeConstants.FS_PREFIX_S3:
- locationType = LocationType.S3;
- this.location = location;
- break;
- case FeConstants.FS_PREFIX_S3A:
- locationType = LocationType.S3A;
- this.location = convertToS3(location);
- break;
- case FeConstants.FS_PREFIX_S3N:
- // include the check for multi locations and in a table, such as both s3 and hdfs are in a table.
- locationType = LocationType.S3N;
- this.location = convertToS3(location);
- break;
- case FeConstants.FS_PREFIX_BOS:
- locationType = LocationType.BOS;
- // use s3 client to access
- this.location = convertToS3(location);
- break;
- case FeConstants.FS_PREFIX_GCS:
- locationType = LocationType.GCS;
- // use s3 client to access
- this.location = convertToS3(location);
- break;
- case FeConstants.FS_PREFIX_OSS:
- if (isHdfsOnOssEndpoint(location)) {
- locationType = LocationType.OSS_HDFS;
+ if (scheme.isEmpty()) {
+ locationType = LocationType.NOSCHEME;
+ this.location = location;
+ } else {
+ switch (scheme) {
+ case FeConstants.FS_PREFIX_HDFS:
+ locationType = LocationType.HDFS;
+ // Need add hdfs host to location
+ String host = props.get(HdfsResource.DSF_NAMESERVICES);
+ this.location = normalizedHdfsPath(location, host);
+ break;
+ case FeConstants.FS_PREFIX_S3:
+ locationType = LocationType.S3;
this.location = location;
- } else {
+ break;
+ case FeConstants.FS_PREFIX_S3A:
+ locationType = LocationType.S3A;
+ this.location = convertToS3(location);
+ break;
+ case FeConstants.FS_PREFIX_S3N:
+ // include the check for multi locations and in a table, such as both s3 and hdfs are in a table.
+ locationType = LocationType.S3N;
+ this.location = convertToS3(location);
+ break;
+ case FeConstants.FS_PREFIX_BOS:
+ locationType = LocationType.BOS;
+ // use s3 client to access
+ this.location = convertToS3(location);
+ break;
+ case FeConstants.FS_PREFIX_GCS:
+ locationType = LocationType.GCS;
+ // use s3 client to access
+ this.location = convertToS3(location);
+ break;
+ case FeConstants.FS_PREFIX_OSS:
+ if (isHdfsOnOssEndpoint(location)) {
+ locationType = LocationType.OSS_HDFS;
+ this.location = location;
+ } else {
+ if (useS3EndPoint(props)) {
+ this.location = convertToS3(location);
+ } else {
+ this.location = location;
+ }
+ locationType = LocationType.OSS;
+ }
+ break;
+ case FeConstants.FS_PREFIX_COS:
if (useS3EndPoint(props)) {
this.location = convertToS3(location);
} else {
this.location = location;
}
- locationType = LocationType.OSS;
- }
- break;
- case FeConstants.FS_PREFIX_COS:
- if (useS3EndPoint(props)) {
- this.location = convertToS3(location);
- } else {
+ locationType = LocationType.COS;
+ break;
+ case FeConstants.FS_PREFIX_OBS:
+ if (useS3EndPoint(props)) {
+ this.location = convertToS3(location);
+ } else {
+ this.location = location;
+ }
+ locationType = LocationType.OBS;
+ break;
+ case FeConstants.FS_PREFIX_OFS:
+ locationType = LocationType.OFS;
this.location = location;
- }
- locationType = LocationType.COS;
- break;
- case FeConstants.FS_PREFIX_OBS:
- if (useS3EndPoint(props)) {
- this.location = convertToS3(location);
- } else {
+ break;
+ case FeConstants.FS_PREFIX_JFS:
+ locationType = LocationType.JFS;
this.location = location;
- }
- locationType = LocationType.OBS;
- break;
- case FeConstants.FS_PREFIX_OFS:
- locationType = LocationType.OFS;
- this.location = location;
- break;
- case FeConstants.FS_PREFIX_JFS:
- locationType = LocationType.JFS;
- this.location = location;
- break;
- case FeConstants.FS_PREFIX_GFS:
- locationType = LocationType.GFS;
- this.location = location;
- break;
- case FeConstants.FS_PREFIX_COSN:
- // if treat cosn(tencent hadoop-cos) as a s3 file system, may bring incompatible issues
- locationType = LocationType.COSN;
- this.location = location;
- break;
- case FeConstants.FS_PREFIX_VIEWFS:
- locationType = LocationType.VIEWFS;
- this.location = location;
- break;
- case FeConstants.FS_PREFIX_FILE:
- locationType = LocationType.LOCAL;
- this.location = location;
- break;
- default:
- locationType = LocationType.UNKNOWN;
- this.location = location;
+ break;
+ case FeConstants.FS_PREFIX_GFS:
+ locationType = LocationType.GFS;
+ this.location = location;
+ break;
+ case FeConstants.FS_PREFIX_COSN:
+ // if treat cosn(tencent hadoop-cos) as a s3 file system, may bring incompatible issues
+ locationType = LocationType.COSN;
+ this.location = location;
+ break;
+ case FeConstants.FS_PREFIX_VIEWFS:
+ locationType = LocationType.VIEWFS;
+ this.location = location;
+ break;
+ case FeConstants.FS_PREFIX_FILE:
+ locationType = LocationType.LOCAL;
+ this.location = location;
+ break;
+ default:
+ locationType = LocationType.UNKNOWN;
+ this.location = location;
+ }
}
}
private static String parseScheme(String location) {
+ String scheme = "";
String[] schemeSplit = location.split(SCHEME_DELIM);
if (schemeSplit.length > 1) {
- return schemeSplit[0];
+ scheme = schemeSplit[0];
} else {
schemeSplit = location.split(NONSTANDARD_SCHEME_DELIM);
if (schemeSplit.length > 1) {
- return schemeSplit[0];
+ scheme = schemeSplit[0];
}
- throw new IllegalArgumentException("Fail to parse scheme, invalid location: " + location);
}
+
+ // if not get scheme, need consider /path/to/local to no scheme
+ if (scheme.isEmpty()) {
+ try {
+ Paths.get(location);
+ } catch (InvalidPathException exception) {
+ throw new IllegalArgumentException("Fail to parse scheme, invalid location: " + location);
+ }
+ }
+
+ return scheme;
}
private boolean useS3EndPoint(Map<String, String> props) {
@@ -196,6 +215,7 @@ public class LocationPath {
/**
* The converted path is used for FE to get metadata
+ *
* @param location origin location
* @return metadata location path. just convert when storage is compatible with s3 client.
*/
@@ -219,7 +239,7 @@ public class LocationPath {
// Need to encode these characters before creating URI.
// But doesn't encode '/' and ':' so that we can get the correct uri host.
location = URLEncoder.encode(location, StandardCharsets.UTF_8.name())
- .replace("%2F", "/").replace("%3A", ":");
+ .replace("%2F", "/").replace("%3A", ":");
URI normalizedUri = new URI(location);
// compatible with 'hdfs:///' or 'hdfs:/'
if (StringUtils.isEmpty(normalizedUri.getHost())) {
@@ -336,6 +356,7 @@ public class LocationPath {
/**
* The converted path is used for BE
+ *
* @return BE scan range path
*/
public Path toScanRangeLocation() {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/external/iceberg/IcebergScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/planner/external/iceberg/IcebergScanNode.java
index 7d055d5a8b1..2a0de32acbc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/external/iceberg/IcebergScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/external/iceberg/IcebergScanNode.java
@@ -352,11 +352,16 @@ public class IcebergScanNode extends FileQueryScanNode {
private String normalizeLocation(String location) {
Map<String, String> props = source.getCatalog().getProperties();
+ LocationPath locationPath = new LocationPath(location, props);
String icebergCatalogType = props.get(IcebergExternalCatalog.ICEBERG_CATALOG_TYPE);
if ("hadoop".equalsIgnoreCase(icebergCatalogType)) {
- if (!location.startsWith(HdfsResource.HDFS_PREFIX)) {
+ // if no scheme info, fill will HADOOP_FS_NAME
+ // if no HADOOP_FS_NAME, then should be local file system
+ if (locationPath.getLocationType() == LocationPath.LocationType.NOSCHEME) {
String fsName = props.get(HdfsResource.HADOOP_FS_NAME);
- location = fsName + location;
+ if (fsName != null) {
+ location = fsName + location;
+ }
}
}
return location;
diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/util/LocationPathTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/util/LocationPathTest.java
index 71ee9100ffc..571826aa9c8 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/common/util/LocationPathTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/common/util/LocationPathTest.java
@@ -171,8 +171,22 @@ public class LocationPathTest {
LocationPath locationPath = new LocationPath("unknown://test.com", rangeProps);
// FE
Assertions.assertTrue(locationPath.get().startsWith("unknown://"));
+ Assertions.assertTrue(locationPath.getLocationType() == LocationPath.LocationType.UNKNOWN);
// BE
String beLocation = locationPath.toScanRangeLocation().toString();
Assertions.assertTrue(beLocation.startsWith("unknown://"));
}
+
+ @Test
+ public void testNoSchemeLocation() {
+ // when use unknown location, pass to BE
+ Map<String, String> rangeProps = new HashMap<>();
+ LocationPath locationPath = new LocationPath("/path/to/local", rangeProps);
+ // FE
+ Assertions.assertTrue(locationPath.get().equalsIgnoreCase("/path/to/local"));
+ Assertions.assertTrue(locationPath.getLocationType() == LocationPath.LocationType.NOSCHEME);
+ // BE
+ String beLocation = locationPath.toScanRangeLocation().toString();
+ Assertions.assertTrue(beLocation.equalsIgnoreCase("/path/to/local"));
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 02/05: [fix](catalog) fix data_sub/data_add func pushdown in jdbcscan (#30807)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 92226c986a1a24833c69fd8265f1273a06e29cd4
Author: zy-kkk <zh...@gmail.com>
AuthorDate: Mon Feb 5 22:51:26 2024 +0800
[fix](catalog) fix data_sub/data_add func pushdown in jdbcscan (#30807)
---
.../external/jdbc/JdbcFunctionPushDownRule.java | 64 +++++++++++++---
.../jdbc/test_mysql_jdbc_catalog.out | 42 ++++++++++
.../jdbc/test_mysql_jdbc_catalog.groovy | 89 ++++++++++++++++++++++
3 files changed, 184 insertions(+), 11 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcFunctionPushDownRule.java b/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcFunctionPushDownRule.java
index d328952593f..74e04256f43 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcFunctionPushDownRule.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/external/jdbc/JdbcFunctionPushDownRule.java
@@ -20,6 +20,7 @@ package org.apache.doris.planner.external.jdbc;
import org.apache.doris.analysis.Expr;
import org.apache.doris.analysis.FunctionCallExpr;
import org.apache.doris.analysis.FunctionName;
+import org.apache.doris.analysis.TimestampArithmeticExpr;
import org.apache.doris.catalog.TableIf.TableType;
import org.apache.doris.thrift.TOdbcTableType;
@@ -75,10 +76,6 @@ public class JdbcFunctionPushDownRule {
REPLACE_MYSQL_FUNCTIONS.put("to_date", "date");
}
- private static boolean isReplaceMysqlFunctions(String functionName) {
- return REPLACE_MYSQL_FUNCTIONS.containsKey(functionName.toLowerCase());
- }
-
private static final Map<String, String> REPLACE_CLICKHOUSE_FUNCTIONS = Maps.newHashMap();
static {
@@ -86,16 +83,20 @@ public class JdbcFunctionPushDownRule {
REPLACE_CLICKHOUSE_FUNCTIONS.put("unix_timestamp", "toUnixTimestamp");
}
- private static boolean isReplaceClickHouseFunctions(String functionName) {
- return REPLACE_CLICKHOUSE_FUNCTIONS.containsKey(functionName.toLowerCase());
- }
-
private static final Map<String, String> REPLACE_ORACLE_FUNCTIONS = Maps.newHashMap();
static {
REPLACE_ORACLE_FUNCTIONS.put("ifnull", "nvl");
}
+ private static boolean isReplaceMysqlFunctions(String functionName) {
+ return REPLACE_MYSQL_FUNCTIONS.containsKey(functionName.toLowerCase());
+ }
+
+ private static boolean isReplaceClickHouseFunctions(String functionName) {
+ return REPLACE_CLICKHOUSE_FUNCTIONS.containsKey(functionName.toLowerCase());
+ }
+
private static boolean isReplaceOracleFunctions(String functionName) {
return REPLACE_ORACLE_FUNCTIONS.containsKey(functionName.toLowerCase());
}
@@ -141,6 +142,8 @@ public class JdbcFunctionPushDownRule {
}
replaceFunctionNameIfNecessary(func, replaceFunction, functionCallExpr, tableType);
+
+ expr = replaceGenericFunctionExpr(functionCallExpr, func);
}
List<Expr> children = expr.getChildren();
@@ -153,7 +156,7 @@ public class JdbcFunctionPushDownRule {
return expr;
}
- private static String replaceFunctionNameIfNecessary(String func, Predicate<String> replaceFunction,
+ private static void replaceFunctionNameIfNecessary(String func, Predicate<String> replaceFunction,
FunctionCallExpr functionCallExpr, TOdbcTableType tableType) {
if (replaceFunction.test(func)) {
String newFunc;
@@ -168,9 +171,48 @@ public class JdbcFunctionPushDownRule {
}
if (newFunc != null) {
functionCallExpr.setFnName(FunctionName.createBuiltinName(newFunc));
- func = functionCallExpr.getFnName().getFunction();
}
}
- return func;
+ }
+
+ // Function used to convert nereids planner's function to old planner's function
+ private static Expr replaceGenericFunctionExpr(FunctionCallExpr functionCallExpr, String func) {
+ Map<String, String> supportedTimeUnits = Maps.newHashMap();
+ supportedTimeUnits.put("years", "YEAR");
+ supportedTimeUnits.put("months", "MONTH");
+ supportedTimeUnits.put("weeks", "WEEK");
+ supportedTimeUnits.put("days", "DAY");
+ supportedTimeUnits.put("hours", "HOUR");
+ supportedTimeUnits.put("minutes", "MINUTE");
+ supportedTimeUnits.put("seconds", "SECOND");
+
+ String baseFuncName = null;
+ String timeUnit = null;
+
+ for (Map.Entry<String, String> entry : supportedTimeUnits.entrySet()) {
+ if (func.endsWith(entry.getKey() + "_add")) {
+ baseFuncName = "date_add";
+ timeUnit = entry.getValue();
+ break;
+ } else if (func.endsWith(entry.getKey() + "_sub")) {
+ baseFuncName = "date_sub";
+ timeUnit = entry.getValue();
+ break;
+ }
+ }
+
+ if (baseFuncName != null && timeUnit != null) {
+ if (functionCallExpr.getChildren().size() == 2) {
+ Expr child1 = functionCallExpr.getChild(0);
+ Expr child2 = functionCallExpr.getChild(1);
+ return new TimestampArithmeticExpr(
+ baseFuncName,
+ child1,
+ child2,
+ timeUnit
+ );
+ }
+ }
+ return functionCallExpr;
}
}
diff --git a/regression-test/data/external_table_p0/jdbc/test_mysql_jdbc_catalog.out b/regression-test/data/external_table_p0/jdbc/test_mysql_jdbc_catalog.out
index 555665f08a1..dea68a6b71f 100644
--- a/regression-test/data/external_table_p0/jdbc/test_mysql_jdbc_catalog.out
+++ b/regression-test/data/external_table_p0/jdbc/test_mysql_jdbc_catalog.out
@@ -368,3 +368,45 @@ year SMALLINT Yes false \N NONE
-- !money_format --
1
+-- !date_add_year --
+2 2022-01-01
+
+-- !date_add_month --
+2 2022-01-01
+
+-- !date_add_week --
+2 2022-01-01
+
+-- !date_add_day --
+2 2022-01-01
+
+-- !date_add_hour --
+2 2022-01-01
+
+-- !date_add_min --
+2 2022-01-01
+
+-- !date_add_sec --
+2 2022-01-01
+
+-- !date_sub_year --
+2 2022-01-01
+
+-- !date_sub_month --
+2 2022-01-01
+
+-- !date_sub_week --
+2 2022-01-01
+
+-- !date_sub_day --
+2 2022-01-01
+
+-- !date_sub_hour --
+2 2022-01-01
+
+-- !date_sub_min --
+2 2022-01-01
+
+-- !date_sub_sec --
+2 2022-01-01
+
diff --git a/regression-test/suites/external_table_p0/jdbc/test_mysql_jdbc_catalog.groovy b/regression-test/suites/external_table_p0/jdbc/test_mysql_jdbc_catalog.groovy
index 3457ae2f6f8..71391ed21ea 100644
--- a/regression-test/suites/external_table_p0/jdbc/test_mysql_jdbc_catalog.groovy
+++ b/regression-test/suites/external_table_p0/jdbc/test_mysql_jdbc_catalog.groovy
@@ -421,6 +421,95 @@ suite("test_mysql_jdbc_catalog", "p0,external,mysql,external_docker,external_doc
contains "QUERY: SELECT `k6`, `k8` FROM `doris_test`.`test1` WHERE ((`k8` = 1))"
}
sql """ set enable_ext_func_pred_pushdown = "true"; """
+ // test date_add
+ sql """ set disable_nereids_rules='NORMALIZE_REWRITE_RULES'; """
+ order_qt_date_add_year """ select * from test_zd where date_add(d_z,interval 1 year) = '2023-01-01' order by 1; """
+ explain {
+ sql("select * from test_zd where date_add(d_z,interval 1 year) = '2023-01-01' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_add(`d_z`, INTERVAL 1 year) = '2023-01-01')"
+ }
+ order_qt_date_add_month """ select * from test_zd where date_add(d_z,interval 1 month) = '2022-02-01' order by 1; """
+ explain {
+ sql("select * from test_zd where date_add(d_z,interval 1 month) = '2022-02-01' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_add(`d_z`, INTERVAL 1 month) = '2022-02-01')"
+ }
+ order_qt_date_add_week """ select * from test_zd where date_add(d_z,interval 1 week) = '2022-01-08' order by 1; """
+ explain {
+ sql("select * from test_zd where date_add(d_z,interval 1 week) = '2022-01-08' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_add(`d_z`, INTERVAL 1 week) = '2022-01-08')"
+ }
+ order_qt_date_add_day """ select * from test_zd where date_add(d_z,interval 1 day) = '2022-01-02' order by 1; """
+ explain {
+ sql("select * from test_zd where date_add(d_z,interval 1 day) = '2022-01-02' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_add(`d_z`, INTERVAL 1 day) = '2022-01-02')"
+ }
+ order_qt_date_add_hour """ select * from test_zd where date_add(d_z,interval 1 hour) = '2022-01-01 01:00:00' order by 1; """
+ explain {
+ sql("select * from test_zd where date_add(d_z,interval 1 hour) = '2022-01-01 01:00:00' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_add(`d_z`, INTERVAL 1 hour) = '2022-01-01 01:00:00')"
+ }
+ order_qt_date_add_min """ select * from test_zd where date_add(d_z,interval 1 minute) = '2022-01-01 00:01:00' order by 1; """
+ explain {
+ sql("select * from test_zd where date_add(d_z,interval 1 minute) = '2022-01-01 00:01:00' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_add(`d_z`, INTERVAL 1 minute) = '2022-01-01 00:01:00')"
+ }
+ order_qt_date_add_sec """ select * from test_zd where date_add(d_z,interval 1 second) = '2022-01-01 00:00:01' order by 1; """
+ explain {
+ sql("select * from test_zd where date_add(d_z,interval 1 second) = '2022-01-01 00:00:01' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_add(`d_z`, INTERVAL 1 second) = '2022-01-01 00:00:01')"
+ }
+ // date_sub
+ order_qt_date_sub_year """ select * from test_zd where date_sub(d_z,interval 1 year) = '2021-01-01' order by 1; """
+ explain {
+ sql("select * from test_zd where date_sub(d_z,interval 1 year) = '2021-01-01' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_sub(`d_z`, INTERVAL 1 year) = '2021-01-01')"
+ }
+ order_qt_date_sub_month """ select * from test_zd where date_sub(d_z,interval 1 month) = '2021-12-01' order by 1; """
+ explain {
+ sql("select * from test_zd where date_sub(d_z,interval 1 month) = '2021-12-01' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_sub(`d_z`, INTERVAL 1 month) = '2021-12-01')"
+ }
+ order_qt_date_sub_week """ select * from test_zd where date_sub(d_z,interval 1 week) = '2021-12-25' order by 1; """
+ explain {
+ sql("select * from test_zd where date_sub(d_z,interval 1 week) = '2021-12-25' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_sub(`d_z`, INTERVAL 1 week) = '2021-12-25')"
+ }
+ order_qt_date_sub_day """ select * from test_zd where date_sub(d_z,interval 1 day) = '2021-12-31' order by 1; """
+ explain {
+ sql("select * from test_zd where date_sub(d_z,interval 1 day) = '2021-12-31' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_sub(`d_z`, INTERVAL 1 day) = '2021-12-31')"
+ }
+ order_qt_date_sub_hour """ select * from test_zd where date_sub(d_z,interval 1 hour) = '2021-12-31 23:00:00' order by 1; """
+ explain {
+ sql("select * from test_zd where date_sub(d_z,interval 1 hour) = '2021-12-31 23:00:00' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_sub(`d_z`, INTERVAL 1 hour) = '2021-12-31 23:00:00')"
+ }
+ order_qt_date_sub_min """ select * from test_zd where date_sub(d_z,interval 1 minute) = '2021-12-31 23:59:00' order by 1; """
+ explain {
+ sql("select * from test_zd where date_sub(d_z,interval 1 minute) = '2021-12-31 23:59:00' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_sub(`d_z`, INTERVAL 1 minute) = '2021-12-31 23:59:00')"
+ }
+ order_qt_date_sub_sec """ select * from test_zd where date_sub(d_z,interval 1 second) = '2021-12-31 23:59:59' order by 1; """
+ explain {
+ sql("select * from test_zd where date_sub(d_z,interval 1 second) = '2021-12-31 23:59:59' order by 1;")
+
+ contains " QUERY: SELECT `id`, `d_z` FROM `doris_test`.`test_zd` WHERE (date_sub(`d_z`, INTERVAL 1 second) = '2021-12-31 23:59:59')"
+ }
+ sql """ set disable_nereids_rules=''; """
+
} finally {
res_dbs_log = sql "show databases;"
for(int i = 0;i < res_dbs_log.size();i++) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 04/05: [fix](kerberos) fix kerberos ugi login method (#30766)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit bc2e8ac8f97c864b2c9308060a53971b681cf4b1
Author: slothever <18...@users.noreply.github.com>
AuthorDate: Tue Feb 6 08:33:25 2024 +0800
[fix](kerberos) fix kerberos ugi login method (#30766)
---
.../hudi-scanner/src/main/java/org/apache/doris/hudi/Utils.java | 1 +
.../main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java | 1 +
2 files changed, 2 insertions(+)
diff --git a/fe/be-java-extensions/hudi-scanner/src/main/java/org/apache/doris/hudi/Utils.java b/fe/be-java-extensions/hudi-scanner/src/main/java/org/apache/doris/hudi/Utils.java
index 9dcfacebb8c..4e8d670dac2 100644
--- a/fe/be-java-extensions/hudi-scanner/src/main/java/org/apache/doris/hudi/Utils.java
+++ b/fe/be-java-extensions/hudi-scanner/src/main/java/org/apache/doris/hudi/Utils.java
@@ -118,6 +118,7 @@ public class Utils {
HoodieTableMetaClient metaClient;
if (ugi != null) {
try {
+ ugi.checkTGTAndReloginFromKeytab();
metaClient = ugi.doAs(
(PrivilegedExceptionAction<HoodieTableMetaClient>) () -> HoodieTableMetaClient.builder()
.setConf(conf).setBasePath(basePath).build());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
index 7d0419fcbf9..6273b78b66b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
@@ -807,6 +807,7 @@ public class HiveMetaStoreClientHelper {
UserGroupInformation ugi = getUserGroupInformation(conf);
try {
if (ugi != null) {
+ ugi.checkTGTAndReloginFromKeytab();
return ugi.doAs(action);
} else {
return action.run();
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org