You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2024/10/30 04:21:33 UTC
(doris) branch selectdb-doris-2.1.6-ndsd created (now 705fa134ac6)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch selectdb-doris-2.1.6-ndsd
in repository https://gitbox.apache.org/repos/asf/doris.git
at 705fa134ac6 2.1.6-rc04-ndsd-v20241030
This branch includes the following new commits:
new 3052e3ae264 [selectdb] add build release scripts (#2)
new e73ac992a04 [2.1][fix](external) Fix case-insensitive table name mapping retrieval rules (#42259)
new 705fa134ac6 2.1.6-rc04-ndsd-v20241030
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 02/03: [2.1][fix](external) Fix case-insensitive table name mapping retrieval rules (#42259)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch selectdb-doris-2.1.6-ndsd
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e73ac992a048ac75530de1f98b49e06b5c26d6ed
Author: zy-kkk <zh...@gmail.com>
AuthorDate: Tue Oct 22 23:28:01 2024 +0800
[2.1][fix](external) Fix case-insensitive table name mapping retrieval rules (#42259)
pick (#38227)
```
Doris > CREATE CATALOG `oracle` PROPERTIES (
-> "user" = "doris_test",
-> "type" = "jdbc",
-> "password" = "xxx",
-> "jdbc_url" = "jdbc:oracle:thin:@xxx:1521:XE",
-> "driver_url" = "ojdbc8-19.3.0.0.jar",
-> "driver_class" = "oracle.jdbc.driver.OracleDriver"
-> );
Query OK, 0 rows affected (2.16 sec)
```
```
Doris > show variables like '%lower_case%';
+------------------------+-------+---------------+---------+
| Variable_name | Value | Default_Value | Changed |
+------------------------+-------+---------------+---------+
| lower_case_table_names | 1 | 0 | 1 |
+------------------------+-------+---------------+---------+
1 row in set (0.00 sec)
Doris > show tables from oracle.DORIS_TEST;
+----------------------+
| Tables_in_DORIS_TEST |
+----------------------+
| aa/d |
| aaad |
| lower_test |
| student |
| student2 |
| student3 |
| test_all_types |
| test_char |
| test_clob |
| test_date |
| test_insert |
| test_int |
| test_num |
| test_number |
| test_number2 |
| test_number3 |
| test_number4 |
| test_raw |
| test_timestamp |
+----------------------+
19 rows in set (0.01 sec)
```
```
Doris > select * from oracle.DORIS_TEST.test_int limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1 | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1 | 99 | 9999 | 999999999 | 999999999999999999 | 999 | 99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (1.03 sec)
Doris > select * from oracle.DORIS_TEST.TEST_INT limit 1;
ERROR 1105 (HY000): errCode = 2, detailMessage = Table [TEST_INT] does not exist in database [DORIS_TEST].
```
```
Doris > select * from oracle.DORIS_TEST.test_int limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1 | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1 | 99 | 9999 | 999999999 | 999999999999999999 | 999 | 99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (0.20 sec)
Doris > select * from oracle.DORIS_TEST.TEST_INT limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1 | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1 | 99 | 9999 | 999999999 | 999999999999999999 | 999 | 99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (0.20 sec)
```
```
Doris > show variables like '%lower_case%';
+------------------------+-------+---------------+---------+
| Variable_name | Value | Default_Value | Changed |
+------------------------+-------+---------------+---------+
| lower_case_table_names | 2 | 0 | 1 |
+------------------------+-------+---------------+---------+
1 row in set (0.01 sec)
Doris > show tables from oracle.DORIS_TEST;
+----------------------+
| Tables_in_DORIS_TEST |
+----------------------+
| AA/D |
| AAAD |
| LOWER_TEST |
| STUDENT |
| TEST_ALL_TYPES |
| TEST_CHAR |
| TEST_CLOB |
| TEST_DATE |
| TEST_INSERT |
| TEST_INT |
| TEST_NUM |
| TEST_NUMBER |
| TEST_NUMBER2 |
| TEST_NUMBER3 |
| TEST_NUMBER4 |
| TEST_RAW |
| TEST_TIMESTAMP |
| student2 |
| student3 |
+----------------------+
19 rows in set (1.05 sec)
```
```
Doris > select * from oracle.DORIS_TEST.test_int limit 1;
ERROR 1105 (HY000): errCode = 2, detailMessage = Table [test_int] does not exist in database [DORIS_TEST].
Doris > select * from oracle.DORIS_TEST.TEST_INT limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1 | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1 | 99 | 9999 | 999999999 | 999999999999999999 | 999 | 99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (1.07 sec)
```
```
Doris > select * from oracle.DORIS_TEST.test_int limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1 | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1 | 99 | 9999 | 999999999 | 999999999999999999 | 999 | 99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (0.21 sec)
Doris > select * from oracle.DORIS_TEST.TEST_INT limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1 | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1 | 99 | 9999 | 999999999 | 999999999999999999 | 999 | 99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (0.20 sec)
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
---
.../java/org/apache/doris/catalog/Database.java | 2 +-
.../apache/doris/datasource/ExternalDatabase.java | 37 +++++-
.../ExternalTableNameComparedLowercaseTest.java | 119 +++++++++++++++++++
.../ExternalTableNameStoredLowercaseTest.java | 127 +++++++++++++++++++++
4 files changed, 283 insertions(+), 2 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
index 0eed9534f0a..5eb330e1cb2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Database.java
@@ -90,7 +90,7 @@ public class Database extends MetaObject implements Writable, DatabaseIf<Table>
private final Map<Long, Table> idToTable;
@SerializedName(value = "nameToTable")
private Map<String, Table> nameToTable;
- // table name lower cast -> table name
+ // table name lower case -> table name
private final Map<String, String> lowerCaseToTableName;
// user define function
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java
index d653a5a178e..eda98efb9b6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java
@@ -59,6 +59,7 @@ import java.util.Optional;
import java.util.OptionalLong;
import java.util.Set;
import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
/**
* Base class of external database.
@@ -83,6 +84,8 @@ public abstract class ExternalDatabase<T extends ExternalTable>
protected Map<String, Long> tableNameToId = Maps.newConcurrentMap();
@SerializedName(value = "idToTbl")
protected Map<Long, T> idToTbl = Maps.newConcurrentMap();
+ // table name lower case -> table name
+ private Map<String, String> lowerCaseToTableName = Maps.newConcurrentMap();
@SerializedName(value = "lastUpdateTime")
protected long lastUpdateTime;
protected final InitDatabaseLog.Type dbLogType;
@@ -239,7 +242,14 @@ public abstract class ExternalDatabase<T extends ExternalTable>
} else if (name.equals(MysqlDb.DATABASE_NAME)) {
tableNames = ExternalMysqlDatabase.listTableNames();
} else {
- tableNames = extCatalog.listTableNames(null, name);
+ tableNames = extCatalog.listTableNames(null, name).stream().map(tableName -> {
+ lowerCaseToTableName.put(tableName.toLowerCase(), tableName);
+ if (Env.isStoredTableNamesLowerCase()) {
+ return tableName.toLowerCase();
+ } else {
+ return tableName;
+ }
+ }).collect(Collectors.toList());
}
return tableNames;
}
@@ -325,6 +335,12 @@ public abstract class ExternalDatabase<T extends ExternalTable>
@Override
public boolean isTableExist(String tableName) {
+ if (Env.isTableNamesCaseInsensitive()) {
+ tableName = lowerCaseToTableName.get(tableName.toLowerCase());
+ if (tableName == null) {
+ return false;
+ }
+ }
return extCatalog.tableExist(ConnectContext.get().getSessionContext(), name, tableName);
}
@@ -375,6 +391,15 @@ public abstract class ExternalDatabase<T extends ExternalTable>
@Override
public T getTableNullable(String tableName) {
makeSureInitialized();
+ if (Env.isStoredTableNamesLowerCase()) {
+ tableName = tableName.toLowerCase();
+ }
+ if (Env.isTableNamesCaseInsensitive()) {
+ tableName = lowerCaseToTableName.get(tableName.toLowerCase());
+ if (tableName == null) {
+ return null;
+ }
+ }
if (extCatalog.getUseMetaCache().get()) {
// must use full qualified name to generate id.
// otherwise, if 2 databases have the same table name, the id will be the same.
@@ -419,6 +444,7 @@ public abstract class ExternalDatabase<T extends ExternalTable>
@Override
public void gsonPostProcess() throws IOException {
tableNameToId = Maps.newConcurrentMap();
+ lowerCaseToTableName = Maps.newConcurrentMap();
Map<Long, T> tmpIdToTbl = Maps.newConcurrentMap();
for (Object obj : idToTbl.values()) {
if (obj instanceof LinkedTreeMap) {
@@ -443,6 +469,8 @@ public abstract class ExternalDatabase<T extends ExternalTable>
Preconditions.checkState(obj instanceof ExternalTable);
tmpIdToTbl.put(((ExternalTable) obj).getId(), (T) obj);
tableNameToId.put(((ExternalTable) obj).getName(), ((ExternalTable) obj).getId());
+ lowerCaseToTableName.put(((ExternalTable) obj).getName().toLowerCase(),
+ ((ExternalTable) obj).getName());
}
}
idToTbl = tmpIdToTbl;
@@ -452,6 +480,9 @@ public abstract class ExternalDatabase<T extends ExternalTable>
@Override
public void unregisterTable(String tableName) {
makeSureInitialized();
+ if (Env.isStoredTableNamesLowerCase()) {
+ tableName = tableName.toLowerCase();
+ }
if (LOG.isDebugEnabled()) {
LOG.debug("create table [{}]", tableName);
}
@@ -459,6 +490,7 @@ public abstract class ExternalDatabase<T extends ExternalTable>
if (extCatalog.getUseMetaCache().get()) {
if (isInitialized()) {
metaCache.invalidate(tableName, Util.genIdByName(getQualifiedName(tableName)));
+ lowerCaseToTableName.remove(tableName.toLowerCase());
}
} else {
Long tableId = tableNameToId.remove(tableName);
@@ -467,6 +499,7 @@ public abstract class ExternalDatabase<T extends ExternalTable>
return;
}
idToTbl.remove(tableId);
+ lowerCaseToTableName.remove(tableName.toLowerCase());
}
setLastUpdateTime(System.currentTimeMillis());
Env.getCurrentEnv().getExtMetaCacheMgr().invalidateTableCache(
@@ -490,11 +523,13 @@ public abstract class ExternalDatabase<T extends ExternalTable>
if (extCatalog.getUseMetaCache().get()) {
if (isInitialized()) {
metaCache.updateCache(tableName, (T) tableIf, Util.genIdByName(getQualifiedName(tableName)));
+ lowerCaseToTableName.put(tableName.toLowerCase(), tableName);
}
} else {
if (!tableNameToId.containsKey(tableName)) {
tableNameToId.put(tableName, tableId);
idToTbl.put(tableId, buildTableForInit(tableName, tableId, extCatalog));
+ lowerCaseToTableName.put(tableName.toLowerCase(), tableName);
}
}
setLastUpdateTime(System.currentTimeMillis());
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lowercase/ExternalTableNameComparedLowercaseTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lowercase/ExternalTableNameComparedLowercaseTest.java
new file mode 100644
index 00000000000..c48e18cb271
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/lowercase/ExternalTableNameComparedLowercaseTest.java
@@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.lowercase;
+
+import org.apache.doris.analysis.CreateCatalogStmt;
+import org.apache.doris.analysis.DropCatalogStmt;
+import org.apache.doris.analysis.SwitchStmt;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.FeConstants;
+import org.apache.doris.datasource.test.TestExternalCatalog;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.GlobalVariable;
+import org.apache.doris.utframe.TestWithFeService;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class ExternalTableNameComparedLowercaseTest extends TestWithFeService {
+ private static Env env;
+ private ConnectContext rootCtx;
+
+ @Override
+ protected void runBeforeAll() throws Exception {
+ rootCtx = createDefaultCtx();
+ env = Env.getCurrentEnv();
+ // 1. create test catalog
+ CreateCatalogStmt testCatalog = (CreateCatalogStmt) parseAndAnalyzeStmt("create catalog test1 properties(\n"
+ + " \"type\" = \"test\",\n"
+ + " \"catalog_provider.class\" "
+ + "= \"org.apache.doris.datasource.lowercase.ExternalTableNameComparedLowercaseTest$ExternalTableNameComparedLowercaseProvider\"\n"
+ + ");",
+ rootCtx);
+ env.getCatalogMgr().createCatalog(testCatalog);
+ }
+
+ @Override
+ protected void beforeCluster() {
+ Config.lower_case_table_names = 2;
+ FeConstants.runningUnitTest = true;
+ }
+
+ @Override
+ protected void runAfterAll() throws Exception {
+ super.runAfterAll();
+ rootCtx.setThreadLocalInfo();
+ DropCatalogStmt stmt = (DropCatalogStmt) parseAndAnalyzeStmt("drop catalog test1");
+ env.getCatalogMgr().dropCatalog(stmt);
+ }
+
+
+ @Test
+ public void testGlobalVariable() {
+ Assertions.assertEquals(2, GlobalVariable.lowerCaseTableNames);
+ }
+
+ @Test
+ public void testTableNameLowerCase() {
+ Set<String> tableNames = env.getCatalogMgr().getCatalog("test1").getDbNullable("db1").getTableNamesWithLock();
+ Assertions.assertEquals(2, tableNames.size());
+ Assertions.assertTrue(tableNames.contains("TABLE1"));
+ Assertions.assertTrue(tableNames.contains("TABLE2"));
+ }
+
+ private void switchTest() throws Exception {
+ SwitchStmt switchTest = (SwitchStmt) parseAndAnalyzeStmt("switch test1;");
+ Env.getCurrentEnv().changeCatalog(connectContext, switchTest.getCatalogName());
+ }
+
+ public static class ExternalTableNameComparedLowercaseProvider implements TestExternalCatalog.TestCatalogProvider {
+ public static final Map<String, Map<String, List<Column>>> MOCKED_META;
+
+ static {
+ MOCKED_META = Maps.newHashMap();
+ Map<String, List<Column>> tblSchemaMap1 = Maps.newHashMap();
+ // db1
+ tblSchemaMap1.put("TABLE1", Lists.newArrayList(
+ new Column("siteid", PrimitiveType.INT),
+ new Column("citycode", PrimitiveType.SMALLINT),
+ new Column("username", PrimitiveType.VARCHAR),
+ new Column("pv", PrimitiveType.BIGINT)));
+ tblSchemaMap1.put("TABLE2", Lists.newArrayList(
+ new Column("k1", PrimitiveType.INT),
+ new Column("k2", PrimitiveType.VARCHAR),
+ new Column("k3", PrimitiveType.VARCHAR),
+ new Column("k4", PrimitiveType.INT),
+ new Column("k5", PrimitiveType.LARGEINT)));
+ MOCKED_META.put("db1", tblSchemaMap1);
+ }
+
+ @Override
+ public Map<String, Map<String, List<Column>>> getMetadata() {
+ return MOCKED_META;
+ }
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lowercase/ExternalTableNameStoredLowercaseTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lowercase/ExternalTableNameStoredLowercaseTest.java
new file mode 100644
index 00000000000..d55e209fa63
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/lowercase/ExternalTableNameStoredLowercaseTest.java
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.lowercase;
+
+import org.apache.doris.analysis.CreateCatalogStmt;
+import org.apache.doris.analysis.DropCatalogStmt;
+import org.apache.doris.analysis.SwitchStmt;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.PrimitiveType;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.FeConstants;
+import org.apache.doris.datasource.test.TestExternalCatalog;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.GlobalVariable;
+import org.apache.doris.utframe.TestWithFeService;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+
+public class ExternalTableNameStoredLowercaseTest extends TestWithFeService {
+ private static Env env;
+ private ConnectContext rootCtx;
+
+ @Override
+ protected void runBeforeAll() throws Exception {
+ rootCtx = createDefaultCtx();
+ env = Env.getCurrentEnv();
+ // 1. create test catalog
+ CreateCatalogStmt testCatalog = (CreateCatalogStmt) parseAndAnalyzeStmt("create catalog test1 properties(\n"
+ + " \"type\" = \"test\",\n"
+ + " \"catalog_provider.class\" "
+ + "= \"org.apache.doris.datasource.lowercase.ExternalTableNameStoredLowercaseTest$ExternalTableNameStoredLowercaseProvider\"\n"
+ + ");",
+ rootCtx);
+ env.getCatalogMgr().createCatalog(testCatalog);
+ }
+
+ @Override
+ protected void beforeCluster() {
+ Config.lower_case_table_names = 1;
+ FeConstants.runningUnitTest = true;
+ }
+
+ @Override
+ protected void runAfterAll() throws Exception {
+ super.runAfterAll();
+ rootCtx.setThreadLocalInfo();
+ DropCatalogStmt stmt = (DropCatalogStmt) parseAndAnalyzeStmt("drop catalog test1");
+ env.getCatalogMgr().dropCatalog(stmt);
+ }
+
+
+ @Test
+ public void testGlobalVariable() {
+ Assertions.assertEquals(1, GlobalVariable.lowerCaseTableNames);
+ }
+
+ @Test
+ public void testTableNameLowerCase() {
+ Set<String> tableNames = env.getCatalogMgr().getCatalog("test1").getDbNullable("db1").getTableNamesWithLock();
+ Assertions.assertEquals(3, tableNames.size());
+ Assertions.assertTrue(tableNames.contains("table1"));
+ Assertions.assertTrue(tableNames.contains("table2"));
+ Assertions.assertTrue(tableNames.contains("table3"));
+ Assertions.assertFalse(tableNames.contains("TABLE1"));
+ }
+
+ private void switchTest() throws Exception {
+ SwitchStmt switchTest = (SwitchStmt) parseAndAnalyzeStmt("switch test1;");
+ Env.getCurrentEnv().changeCatalog(connectContext, switchTest.getCatalogName());
+ }
+
+ public static class ExternalTableNameStoredLowercaseProvider implements TestExternalCatalog.TestCatalogProvider {
+ public static final Map<String, Map<String, List<Column>>> MOCKED_META;
+
+ static {
+ MOCKED_META = Maps.newHashMap();
+ Map<String, List<Column>> tblSchemaMap1 = Maps.newHashMap();
+ // db1
+ tblSchemaMap1.put("table1", Lists.newArrayList(
+ new Column("siteid", PrimitiveType.INT),
+ new Column("citycode", PrimitiveType.SMALLINT),
+ new Column("username", PrimitiveType.VARCHAR),
+ new Column("pv", PrimitiveType.BIGINT)));
+ tblSchemaMap1.put("table2", Lists.newArrayList(
+ new Column("k1", PrimitiveType.INT),
+ new Column("k2", PrimitiveType.VARCHAR),
+ new Column("k3", PrimitiveType.VARCHAR),
+ new Column("k4", PrimitiveType.INT),
+ new Column("k5", PrimitiveType.LARGEINT)));
+ tblSchemaMap1.put("TABLE3", Lists.newArrayList(
+ new Column("k1", PrimitiveType.INT),
+ new Column("k2", PrimitiveType.VARCHAR),
+ new Column("k3", PrimitiveType.VARCHAR),
+ new Column("k4", PrimitiveType.INT),
+ new Column("k5", PrimitiveType.LARGEINT)));
+ MOCKED_META.put("db1", tblSchemaMap1);
+ }
+
+ @Override
+ public Map<String, Map<String, List<Column>>> getMetadata() {
+ return MOCKED_META;
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 01/03: [selectdb] add build release scripts (#2)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch selectdb-doris-2.1.6-ndsd
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 3052e3ae264b7ace7f18674db53a06a668006f82
Author: xxiao2018 <78...@users.noreply.github.com>
AuthorDate: Fri May 5 11:57:05 2023 +0800
[selectdb] add build release scripts (#2)
ci: align build-for-release-selectdb.sh script (#10)
[release](build script) fix build release bug when build multiple version on same machine #32104
change jdk path to open jdk (#108)
[script] remote audit loader from build-for-release-selectdb.sh
modify build-for-release-selectdb.sh
cp apache_hdfs_broker and audit_loader to extensions dir
---
bin/start_be.sh | 9 +
bin/start_fe.sh | 11 ++
build-for-release-selectdb.sh | 181 +++++++++++++++++++++
conf/be.conf | 3 +
conf/fe.conf | 3 +
.../java/org/apache/doris/qe/GlobalVariable.java | 4 +-
fs_brokers/apache_hdfs_broker/bin/start_broker.sh | 9 +
gensrc/script/gen_build_version.sh | 4 +-
8 files changed, 220 insertions(+), 4 deletions(-)
diff --git a/bin/start_be.sh b/bin/start_be.sh
index d9800e1b047..abc3a2a86ae 100755
--- a/bin/start_be.sh
+++ b/bin/start_be.sh
@@ -63,6 +63,15 @@ DORIS_HOME="$(
pwd
)"
export DORIS_HOME
+if [[ -z "${JAVA_HOME}" ]]; then
+ java_home="$(
+ cd "${curdir}/../.."
+ pwd
+ )"
+ export JAVA_HOME=$java_home/java8
+ export PATH=$JAVA_HOME/bin:$PATH
+ echo "${JAVA_HOME}"
+fi
if [[ "$(uname -s)" != 'Darwin' ]]; then
MAX_MAP_COUNT="$(cat /proc/sys/vm/max_map_count)"
diff --git a/bin/start_fe.sh b/bin/start_fe.sh
index 0e5033c094c..aed3d25c965 100755
--- a/bin/start_fe.sh
+++ b/bin/start_fe.sh
@@ -89,6 +89,17 @@ DORIS_HOME="$(
)"
export DORIS_HOME
+if [[ -z "${JAVA_HOME}" ]]; then
+ java_home="$(
+ cd "${curdir}/../.."
+ pwd
+ )"
+ export JAVA_HOME=$java_home/java8
+ export PATH=$JAVA_HOME/bin:$PATH
+ echo "${JAVA_HOME}"
+fi
+
+
# export env variables from fe.conf
#
# JAVA_OPTS
diff --git a/build-for-release-selectdb.sh b/build-for-release-selectdb.sh
new file mode 100755
index 00000000000..2d13b05655b
--- /dev/null
+++ b/build-for-release-selectdb.sh
@@ -0,0 +1,181 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+##############################################################
+# This script is used to build for SelectDB Enterprise Core
+##############################################################
+
+set -eo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" &>/dev/null && pwd)"
+
+export DORIS_HOME="${ROOT}"
+
+# Check args
+usage() {
+ echo "
+Usage: $0 --version version <options>
+ Optional options:
+ [no option] build with avx2
+ --noavx2 build without avx2
+ --tar pack the output
+
+ Eg.
+ $0 --version 1.2.0 build with avx2
+ $0 --noavx2 --version 1.2.0 build without avx2
+ $0 --version 1.2.0 --tar build with avx2 and pack the output
+ "
+ exit 1
+}
+
+if ! OPTS="$(getopt \
+ -n "$0" \
+ -o '' \
+ -l 'noavx2' \
+ -l 'tar' \
+ -l 'version:' \
+ -l 'help' \
+ -- "$@")"; then
+ usage
+fi
+
+eval set -- "${OPTS}"
+
+_USE_AVX2=1
+TAR=0
+VERSION=
+if [[ "$#" == 1 ]]; then
+ _USE_AVX2=1
+else
+ while true; do
+ case "$1" in
+ --noavx2)
+ _USE_AVX2=0
+ shift
+ ;;
+ --tar)
+ TAR=1
+ shift
+ ;;
+ --version)
+ VERSION="$2"
+ shift 2
+ ;;
+ --help)
+ HELP=1
+ shift
+ ;;
+ --)
+ shift
+ break
+ ;;
+ *)
+ echo "Internal error"
+ exit 1
+ ;;
+ esac
+ done
+fi
+
+if [[ "${HELP}" -eq 1 ]]; then
+ usage
+ exit
+fi
+
+if [[ -z ${VERSION} ]]; then
+ echo "Must specify version"
+ usage
+ exit 1
+fi
+
+echo "Get params:
+ VERSION -- ${VERSION}
+ USE_AVX2 -- ${_USE_AVX2}
+ TAR -- ${TAR}
+"
+
+ARCH="$(uname -m)"
+if [[ "${ARCH}" == "aarch64" ]]; then
+ ARCH="arm64"
+elif [[ "${ARCH}" == "x86_64" ]]; then
+ ARCH="x64"
+else
+ echo "Unknown arch: ${ARCH}"
+ exit 1
+fi
+
+ORI_OUTPUT="${ROOT}/output"
+
+FE="fe"
+BE="be"
+BROKER="apache_hdfs_broker"
+
+PACKAGE_NAME="selectdb-doris-${VERSION}-bin-${ARCH}"
+if [[ "${_USE_AVX2}" == "0" && "${ARCH}" == "x64" ]]; then
+ PACKAGE_NAME="${PACKAGE_NAME}-noavx2"
+fi
+OUTPUT="${ORI_OUTPUT}/${PACKAGE_NAME}"
+
+rm -rf "${OUTPUT}" && mkdir -p "${OUTPUT}" "${OUTPUT}/extensions"
+echo "Package Path: ${OUTPUT}"
+
+# download and setup java
+JAVA8_DOWNLOAD_LINK=
+JAVA8_DIR_NAME=
+if [[ "${ARCH}" == "x64" ]]; then
+ JAVA8_DOWNLOAD_LINK="https://selectdb-doris-1308700295.cos.ap-beijing.myqcloud.com/release/jdbc_driver/openjdk-8u352-b08-linux-x64.tar.gz"
+ JAVA8_DIR_NAME="openjdk-8u352-b08-linux-x64"
+elif [[ "${ARCH}" == "arm64" ]]; then
+ JAVA8_DOWNLOAD_LINK="https://selectdb-doris-1308700295.cos.ap-beijing.myqcloud.com/release/jdbc_driver/bisheng-jdk-8u352-linux-aarch64.tar.gz"
+ JAVA8_DIR_NAME="bisheng-jdk1.8.0_352"
+else
+ echo "Unknown arch: ${ARCH}"
+ exit 1
+fi
+
+OUTPUT_JAVA8="${OUTPUT}/java8"
+curl -# "${JAVA8_DOWNLOAD_LINK}" | tar xz -C "${OUTPUT}/" && mv "${OUTPUT}/${JAVA8_DIR_NAME}/" "${OUTPUT_JAVA8}"
+export JAVA_HOME="${OUTPUT_JAVA8}"
+export PATH="${JAVA_HOME}/bin:${PATH}"
+
+# build core
+#sh build.sh --clean &&
+USE_AVX2="${_USE_AVX2}" sh build.sh && USE_AVX2="${_USE_AVX2}" sh build.sh --be --meta-tool
+
+echo "Begin to install"
+cp -r "${ORI_OUTPUT}/fe" "${OUTPUT}/fe"
+cp -r "${ORI_OUTPUT}/be" "${OUTPUT}/be"
+cp -r "${ORI_OUTPUT}/apache_hdfs_broker" "${OUTPUT}/extensions/apache_hdfs_broker"
+
+JDBC_DRIVERS_DIR="${OUTPUT}/jdbc_drivers/"
+mkdir -p "${JDBC_DRIVERS_DIR}"
+wget https://selectdb-doris-1308700295.cos.ap-beijing.myqcloud.com/release/jdbc_driver/clickhouse-jdbc-0.3.2-patch11-all.jar -P "${JDBC_DRIVERS_DIR}/"
+wget https://selectdb-doris-1308700295.cos.ap-beijing.myqcloud.com/release/jdbc_driver/mssql-jdbc-11.2.0.jre8.jar -P "${JDBC_DRIVERS_DIR}/"
+wget https://selectdb-doris-1308700295.cos.ap-beijing.myqcloud.com/release/jdbc_driver/mysql-connector-java-8.0.25.jar -P "${JDBC_DRIVERS_DIR}/"
+wget https://selectdb-doris-1308700295.cos.ap-beijing.myqcloud.com/release/jdbc_driver/ojdbc6.jar -P "${JDBC_DRIVERS_DIR}/"
+wget https://selectdb-doris-1308700295.cos.ap-beijing.myqcloud.com/release/jdbc_driver/postgresql-42.5.0.jar -P "${JDBC_DRIVERS_DIR}/"
+
+if [[ "${TAR}" -eq 1 ]]; then
+ echo "Begin to compress"
+ cd "${ORI_OUTPUT}"
+ tar -cf - "${PACKAGE_NAME}" | xz -T0 -z - >"${PACKAGE_NAME}".tar.xz
+ cd -
+fi
+
+echo "Output dir: ${OUTPUT}"
+exit 0
diff --git a/conf/be.conf b/conf/be.conf
index a1c40c6a124..cc96c257846 100644
--- a/conf/be.conf
+++ b/conf/be.conf
@@ -51,6 +51,9 @@ ssl_certificate_path = "$DORIS_HOME/conf/cert.pem"
ssl_private_key_path = "$DORIS_HOME/conf/key.pem"
+# jdbc driver path configuration
+jdbc_drivers_dir=${DORIS_HOME}/../jdbc_drivers/
+
# Choose one if there are more than one ip except loopback address.
# Note that there should at most one ip match this list.
# If no ip match this rule, will choose one randomly.
diff --git a/conf/fe.conf b/conf/fe.conf
index 845dce87cdb..57c7a2ea140 100644
--- a/conf/fe.conf
+++ b/conf/fe.conf
@@ -58,6 +58,9 @@ query_port = 9030
edit_log_port = 9010
arrow_flight_sql_port = -1
+# jdbc driver path configuration
+jdbc_drivers_dir=${DORIS_HOME}/../jdbc_drivers/
+
# Choose one if there are more than one ip except loopback address.
# Note that there should at most one ip match this list.
# If no ip match this rule, will choose one randomly.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java b/fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java
index abe5d452a8f..54363804217 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/GlobalVariable.java
@@ -67,8 +67,8 @@ public final class GlobalVariable {
@VariableMgr.VarAttr(name = VERSION_COMMENT, flag = VariableMgr.READ_ONLY)
- public static String versionComment = "Doris version "
- + Version.DORIS_BUILD_VERSION + "-" + Version.DORIS_BUILD_SHORT_HASH;
+ public static String versionComment =
+ Version.DORIS_BUILD_VERSION + "-" + Version.DORIS_BUILD_SHORT_HASH;
@VariableMgr.VarAttr(name = VERSION, flag = VariableMgr.READ_ONLY)
public static String version = MysqlHandshakePacket.SERVER_VERSION;
diff --git a/fs_brokers/apache_hdfs_broker/bin/start_broker.sh b/fs_brokers/apache_hdfs_broker/bin/start_broker.sh
index b6511332a61..8061b013fcf 100755
--- a/fs_brokers/apache_hdfs_broker/bin/start_broker.sh
+++ b/fs_brokers/apache_hdfs_broker/bin/start_broker.sh
@@ -56,6 +56,15 @@ BROKER_HOME="$(
pwd
)"
export BROKER_HOME
+if [[ -z "${JAVA_HOME}" ]]; then
+ java_home="$(
+ cd "${curdir}/../.."
+ pwd
+ )"
+ export JAVA_HOME=$java_home/java8
+ export PATH=$JAVA_HOME/bin:$PATH
+ echo "${JAVA_HOME}"
+fi
PID_DIR="$(
cd "${curdir}"
diff --git a/gensrc/script/gen_build_version.sh b/gensrc/script/gen_build_version.sh
index b0657b54a7c..d111ed12e13 100755
--- a/gensrc/script/gen_build_version.sh
+++ b/gensrc/script/gen_build_version.sh
@@ -27,7 +27,7 @@
set -eo pipefail
-build_version_prefix="doris"
+build_version_prefix="selectdb-doris"
build_version_major=2
build_version_minor=1
build_version_patch=6
@@ -65,7 +65,7 @@ if [[ -d '.git' ]]; then
short_revision="$(git log -1 --pretty=format:"%h")"
url="git://${hostname}"
else
- revision="Unknown"
+ revision="tag"
short_revision="${revision}"
url="file://${hostname}"
fi
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 03/03: 2.1.6-rc04-ndsd-v20241030
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch selectdb-doris-2.1.6-ndsd
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 705fa134ac603e044750fa07258a6543f936b80e
Author: morningman <mo...@163.com>
AuthorDate: Wed Oct 30 12:21:02 2024 +0800
2.1.6-rc04-ndsd-v20241030
---
gensrc/script/gen_build_version.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/gensrc/script/gen_build_version.sh b/gensrc/script/gen_build_version.sh
index d111ed12e13..88dc1333dc3 100755
--- a/gensrc/script/gen_build_version.sh
+++ b/gensrc/script/gen_build_version.sh
@@ -31,7 +31,7 @@ build_version_prefix="selectdb-doris"
build_version_major=2
build_version_minor=1
build_version_patch=6
-build_version_rc_version="rc04"
+build_version_rc_version="rc04-ndsd-v20241030"
build_version="${build_version_prefix}-${build_version_major}.${build_version_minor}.${build_version_patch}-${build_version_rc_version}"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org