You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2023/07/03 15:33:23 UTC
[doris] branch branch-1.2-lts updated (4db9725940 -> 4ff1ffc8c7)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
from 4db9725940 [branch-1.2][improvement](limit) check exec node limit against invalid value (#21463)
new c6dbc5729f [Fix](multi-catalog) Add hadoop system classpath to CLASSPATH to resolve can not enable hadoop short circuit reading in some environments. (#21430)
new 3d67dea444 [Improve](regresstest) update collect distinct regress test for array hash (#21417)
new bca7edbc58 [fix](truncate) it will directly return and avoid throwing IllegalStateException caused by bufferSize equals zero when table has no partition (#21378)
new 9ba074a597 [bugfix](build script) fix noavx2 package name branch condition #21356
new a6918535c1 [decimalv2](compatibility) add config to allow invalid decimalv2 literal (#21327)
new 4ff1ffc8c7 [Fix](multi-catalog) optimize hashcode for PartitionKey. (#21307)
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/common/config.h | 3 +
be/src/runtime/decimalv2_value.cpp | 4 +-
bin/start_be.sh | 5 +-
build-for-release.sh | 2 +-
.../org/apache/doris/analysis/DateLiteral.java | 6 +-
.../org/apache/doris/catalog/PartitionKey.java | 6 +-
.../apache/doris/datasource/InternalCatalog.java | 5 +
.../org/apache/doris/catalog/PartitionKeyTest.java | 17 +++
.../query_p0/join/test_collect_list_distinct.out | 16 +++
...te_table.groovy => test_truncate_table1.groovy} | 55 ++++++----
.../join/test_collect_list_distinct.groovy | 116 +++++++++++++++++++++
11 files changed, 209 insertions(+), 26 deletions(-)
create mode 100644 regression-test/data/query_p0/join/test_collect_list_distinct.out
copy regression-test/suites/ddl_p0/{test_truncate_table.groovy => test_truncate_table1.groovy} (55%)
create mode 100644 regression-test/suites/query_p0/join/test_collect_list_distinct.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 05/06: [decimalv2](compatibility) add config to allow invalid decimalv2 literal (#21327)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit a6918535c1090746212748c8512167928a0152b7
Author: Gabriel <ga...@gmail.com>
AuthorDate: Mon Jul 3 10:55:27 2023 +0800
[decimalv2](compatibility) add config to allow invalid decimalv2 literal (#21327)
---
be/src/common/config.h | 3 +++
be/src/runtime/decimalv2_value.cpp | 4 +++-
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/be/src/common/config.h b/be/src/common/config.h
index 6787fba585..3aab83398f 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -921,6 +921,9 @@ CONF_mBool(enable_stack_trace, "true");
// enable shrink memory, default is false
CONF_Bool(enable_shrink_memory, "false");
+// Allow invalid decimalv2 literal for compatible with old version. Recommend set it false strongly.
+CONF_mBool(allow_invalid_decimalv2_literal, "false");
+
#ifdef BE_TEST
// test s3
CONF_String(test_s3_resource, "resource");
diff --git a/be/src/runtime/decimalv2_value.cpp b/be/src/runtime/decimalv2_value.cpp
index a21bc5a1d1..cffa7f59c8 100644
--- a/be/src/runtime/decimalv2_value.cpp
+++ b/be/src/runtime/decimalv2_value.cpp
@@ -353,7 +353,9 @@ int DecimalV2Value::parse_from_str(const char* decimal_str, int32_t length) {
_value = StringParser::string_to_decimal<__int128>(decimal_str, length, PRECISION, SCALE,
&result);
- if (result != StringParser::PARSE_SUCCESS) {
+ if (!config::allow_invalid_decimalv2_literal && result != StringParser::PARSE_SUCCESS) {
+ error = E_DEC_BAD_NUM;
+ } else if (config::allow_invalid_decimalv2_literal && result == StringParser::PARSE_FAILURE) {
error = E_DEC_BAD_NUM;
}
return error;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 06/06: [Fix](multi-catalog) optimize hashcode for PartitionKey. (#21307)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 4ff1ffc8c757ec98e9b593360b93db700d12f420
Author: Xiangyu Wang <du...@gmail.com>
AuthorDate: Fri Jun 30 13:48:08 2023 +0800
[Fix](multi-catalog) optimize hashcode for PartitionKey. (#21307)
---
.../java/org/apache/doris/analysis/DateLiteral.java | 6 ++++--
.../java/org/apache/doris/catalog/PartitionKey.java | 6 +++++-
.../java/org/apache/doris/catalog/PartitionKeyTest.java | 17 +++++++++++++++++
3 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
index dabadcd707..598181b256 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DateLiteral.java
@@ -56,7 +56,6 @@ import java.time.temporal.TemporalAccessor;
import java.util.Collections;
import java.util.List;
import java.util.Map;
-import java.util.Objects;
import java.util.TimeZone;
import java.util.regex.Pattern;
@@ -1068,7 +1067,10 @@ public class DateLiteral extends LiteralExpr {
@Override
public int hashCode() {
- return 31 * super.hashCode() + Objects.hashCode(unixTimestamp(TimeZone.getDefault()));
+ // do not invoke the super.hashCode(), just use the return value of getLongValue()
+ // a DateV2 DateLiteral obj may be equal to a Date DateLiteral
+ // if the return value of getLongValue() is the same
+ return Long.hashCode(getLongValue());
}
// parse the date string value in 'value' by 'format' pattern.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionKey.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionKey.java
index 6f26d4ff3a..d6152fb545 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionKey.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/PartitionKey.java
@@ -398,10 +398,14 @@ public class PartitionKey implements Comparable<PartitionKey>, Writable {
@Override
public int hashCode() {
+ int hashCode = 1;
+ for (LiteralExpr key : keys) {
+ hashCode = 31 * hashCode + (key == null ? 0 : key.hashCode());
+ }
int ret = types.size() * 1000;
for (PrimitiveType type : types) {
ret += type.ordinal();
}
- return ret;
+ return hashCode + ret;
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/PartitionKeyTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/PartitionKeyTest.java
index 38806ad1b1..56f58e252c 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/PartitionKeyTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/PartitionKeyTest.java
@@ -79,6 +79,7 @@ public class PartitionKeyTest {
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("32767")),
Arrays.asList(tinyInt, smallInt));
pk2 = PartitionKey.createInfinityPartitionKey(Arrays.asList(tinyInt, smallInt), true);
+ Assert.assertTrue(pk1.hashCode() != pk2.hashCode());
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case2
@@ -86,6 +87,7 @@ public class PartitionKeyTest {
Arrays.asList(tinyInt, smallInt));
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("-32768")),
Arrays.asList(tinyInt, smallInt));
+ Assert.assertTrue(pk1.hashCode() == pk2.hashCode());
Assert.assertTrue(pk1.equals(pk2) && pk1.compareTo(pk2) == 0);
// case3
@@ -93,6 +95,7 @@ public class PartitionKeyTest {
Arrays.asList(int32, bigInt));
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("128"), new PartitionValue("-32768")),
Arrays.asList(int32, bigInt));
+ Assert.assertTrue(pk1.hashCode() != pk2.hashCode());
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case4
@@ -100,6 +103,7 @@ public class PartitionKeyTest {
Arrays.asList(largeInt, bigInt));
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("12346")),
Arrays.asList(largeInt, bigInt));
+ Assert.assertTrue(pk1.hashCode() != pk2.hashCode());
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case5
@@ -107,24 +111,28 @@ public class PartitionKeyTest {
Arrays.asList(date, datetime));
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("2014-12-12"), new PartitionValue("2014-12-12 10:00:01")),
Arrays.asList(date, datetime));
+ Assert.assertTrue(pk1.hashCode() != pk2.hashCode());
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case6
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("-128")),
Arrays.asList(tinyInt, smallInt));
pk2 = PartitionKey.createInfinityPartitionKey(Arrays.asList(tinyInt, smallInt), false);
+ Assert.assertTrue(pk1.hashCode() == pk2.hashCode());
Assert.assertTrue(pk1.equals(pk2) && pk1.compareTo(pk2) == 0);
// case7
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127")),
Arrays.asList(tinyInt, smallInt));
pk2 = PartitionKey.createInfinityPartitionKey(Arrays.asList(tinyInt, smallInt), true);
+ Assert.assertTrue(pk1.hashCode() != pk2.hashCode());
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case7
pk1 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("127"), new PartitionValue("32767")),
Arrays.asList(tinyInt, smallInt));
pk2 = PartitionKey.createInfinityPartitionKey(Arrays.asList(tinyInt, smallInt), true);
+ Assert.assertTrue(pk1.hashCode() != pk2.hashCode());
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case8
@@ -134,6 +142,7 @@ public class PartitionKeyTest {
new PartitionValue("9999-12-31"), new PartitionValue("9999-12-31 23:59:59")),
allColumns);
pk2 = PartitionKey.createInfinityPartitionKey(allColumns, true);
+ Assert.assertTrue(pk1.hashCode() != pk2.hashCode());
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case9
@@ -143,6 +152,7 @@ public class PartitionKeyTest {
new PartitionValue("0000-01-01"), new PartitionValue("0000-01-01 00:00:00")),
allColumns);
pk2 = PartitionKey.createInfinityPartitionKey(allColumns, false);
+ Assert.assertTrue(pk1.hashCode() == pk2.hashCode());
Assert.assertTrue(pk1.equals(pk2) && pk1.compareTo(pk2) == 0);
// case10
@@ -151,6 +161,7 @@ public class PartitionKeyTest {
new PartitionValue("0"), new PartitionValue("1970-01-01"), new PartitionValue("1970-01-01 00:00:00")),
allColumns);
pk2 = PartitionKey.createInfinityPartitionKey(allColumns, false);
+ Assert.assertTrue(pk1.hashCode() != pk2.hashCode());
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == 1);
// case11
@@ -158,6 +169,7 @@ public class PartitionKeyTest {
Arrays.asList(charString, varchar));
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("beijing"), new PartitionValue("shanghai")),
Arrays.asList(charString, varchar));
+ Assert.assertTrue(pk1.hashCode() == pk2.hashCode());
Assert.assertTrue(pk1.equals(pk2) && pk1.compareTo(pk2) == 0);
// case12
@@ -165,6 +177,7 @@ public class PartitionKeyTest {
Arrays.asList(charString, varchar));
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("shijiazhuang"), new PartitionValue("tianjin")),
Arrays.asList(charString, varchar));
+ Assert.assertTrue(pk1.hashCode() != pk2.hashCode());
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case13
@@ -172,6 +185,7 @@ public class PartitionKeyTest {
Arrays.asList(charString, varchar));
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("beijing"), new PartitionValue("tianjin")),
Arrays.asList(charString, varchar));
+ Assert.assertTrue(pk1.hashCode() != pk2.hashCode());
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == -1);
// case14
@@ -179,6 +193,7 @@ public class PartitionKeyTest {
Arrays.asList(bool));
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("false")),
Arrays.asList(bool));
+ Assert.assertTrue(pk1.hashCode() != pk2.hashCode());
Assert.assertTrue(!pk1.equals(pk2) && pk1.compareTo(pk2) == 1);
// case15
@@ -186,6 +201,7 @@ public class PartitionKeyTest {
Arrays.asList(bool));
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("true")),
Arrays.asList(bool));
+ Assert.assertTrue(pk1.hashCode() == pk2.hashCode());
Assert.assertTrue(pk1.equals(pk2) && pk1.compareTo(pk2) == 0);
// case16
@@ -193,6 +209,7 @@ public class PartitionKeyTest {
Arrays.asList(bool));
pk2 = PartitionKey.createPartitionKey(Arrays.asList(new PartitionValue("false")),
Arrays.asList(bool));
+ Assert.assertTrue(pk1.hashCode() == pk2.hashCode());
Assert.assertTrue(pk1.equals(pk2) && pk1.compareTo(pk2) == 0);
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 03/06: [fix](truncate) it will directly return and avoid throwing IllegalStateException caused by bufferSize equals zero when table has no partition (#21378)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit bca7edbc5851d3bdcc16b2fb9723a8017ef593f1
Author: htyoung <ht...@hotmail.com>
AuthorDate: Sat Jul 1 08:39:38 2023 +0800
[fix](truncate) it will directly return and avoid throwing IllegalStateException caused by bufferSize equals zero when table has no partition (#21378)
if table currently has no partition, the truncate SQL will be a empty command, it should directly return and avoid throwing IllegalStateException caused by bufferSize equals zero
Issue Number: close #21316
Co-authored-by: tongyang.han <to...@jiduauto.com>
---
.../apache/doris/datasource/InternalCatalog.java | 5 ++
.../suites/ddl_p0/test_truncate_table1.groovy | 78 ++++++++++++++++++++++
2 files changed, 83 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
index beaf2c4794..8a93277bf5 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java
@@ -2565,6 +2565,11 @@ public class InternalCatalog implements CatalogIf<Database> {
partitionsDistributionInfo.put(partition.getId(), partition.getDistributionInfo());
}
}
+ // if table currently has no partitions, this sql like empty command and do nothing, should return directly
+ // at the same time, it will avoid throwing IllegalStateException when `bufferSize` equals zero
+ if (origPartitions.isEmpty()) {
+ return;
+ }
copiedTbl = olapTable.selectiveCopy(origPartitions.keySet(), IndexExtState.VISIBLE, false);
} finally {
olapTable.readUnlock();
diff --git a/regression-test/suites/ddl_p0/test_truncate_table1.groovy b/regression-test/suites/ddl_p0/test_truncate_table1.groovy
new file mode 100644
index 0000000000..7f25dca5ae
--- /dev/null
+++ b/regression-test/suites/ddl_p0/test_truncate_table1.groovy
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+suite("test_truncate_table1") {
+ // test truncate partition table which has no partition
+ def testTable = "test_truncate_no_partition_table"
+ sql "DROP TABLE IF EXISTS ${testTable}"
+ sql """
+ CREATE TABLE ${testTable}
+ (
+ k1 DATE,
+ k2 DECIMAL(10, 2) DEFAULT "10.5",
+ k3 CHAR(10) COMMENT "string column",
+ k4 INT NOT NULL DEFAULT "1" COMMENT "int column"
+ )
+ PARTITION BY RANGE(k1)
+ ()
+ DISTRIBUTED BY HASH(k2) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1"
+ );
+ """
+ List<List<Object>> result = sql "show partitions from ${testTable}"
+ logger.info("${result}")
+ assertEquals(result.size(), 0)
+
+ sql """truncate table ${testTable};"""
+ result = sql "show partitions from ${testTable}"
+ logger.info("${result}")
+ assertEquals(result.size(), 0)
+
+ sql "DROP TABLE IF EXISTS ${testTable}"
+
+
+
+ // test truncate non partition table
+ def testNonPartitionTable = "test_truncate_non_partition_table"
+ sql "DROP TABLE IF EXISTS ${testNonPartitionTable}"
+ sql """
+ CREATE TABLE ${testNonPartitionTable}
+ (
+ k1 DATE,
+ k2 DECIMAL(10, 2) DEFAULT "10.5",
+ k3 CHAR(10) COMMENT "string column",
+ k4 INT NOT NULL DEFAULT "1" COMMENT "int column"
+ )
+ DISTRIBUTED BY HASH(k2) BUCKETS 1
+ PROPERTIES (
+ "replication_num" = "1"
+ );
+ """
+ List<List<Object>> result1 = sql "show partitions from ${testNonPartitionTable}"
+ logger.info("${result1}")
+ assertEquals(result1.size(), 1)
+
+ sql """truncate table ${testNonPartitionTable};"""
+ result1 = sql "show partitions from ${testNonPartitionTable}"
+ logger.info("${result1}")
+ assertEquals(result1.size(), 1)
+
+ sql "DROP TABLE IF EXISTS ${testNonPartitionTable}"
+}
+
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 02/06: [Improve](regresstest) update collect distinct regress test for array hash (#21417)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 3d67dea444a076896b96df912abb946c6dbbb5ef
Author: amory <wa...@selectdb.com>
AuthorDate: Mon Jul 3 12:16:11 2023 +0800
[Improve](regresstest) update collect distinct regress test for array hash (#21417)
this regress sql can make sense of array hashing function is working fine
---
.../query_p0/join/test_collect_list_distinct.out | 16 +++
.../join/test_collect_list_distinct.groovy | 116 +++++++++++++++++++++
2 files changed, 132 insertions(+)
diff --git a/regression-test/data/query_p0/join/test_collect_list_distinct.out b/regression-test/data/query_p0/join/test_collect_list_distinct.out
new file mode 100644
index 0000000000..23cfae2049
--- /dev/null
+++ b/regression-test/data/query_p0/join/test_collect_list_distinct.out
@@ -0,0 +1,16 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !select_tt --
+1 2 12
+2 1 12
+
+-- !select_bd --
+1 90 \N car
+7 30 1 amory clever
+
+-- !select_bdp --
+1 7 12 car amory is cute
+
+-- !select --
+12 ["amory clever"] car
+12 ["amory clever"] car
+
diff --git a/regression-test/suites/query_p0/join/test_collect_list_distinct.groovy b/regression-test/suites/query_p0/join/test_collect_list_distinct.groovy
new file mode 100644
index 0000000000..059e15180b
--- /dev/null
+++ b/regression-test/suites/query_p0/join/test_collect_list_distinct.groovy
@@ -0,0 +1,116 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_collect_list_distinct") {
+ def table1 = "bu_delivery"
+ def table2 = "bu_delivery_product"
+ def table3 = "bu_trans_transfer"
+
+ sql "DROP TABLE IF EXISTS ${table1}"
+ sql "DROP TABLE IF EXISTS ${table2}"
+ sql "DROP TABLE IF EXISTS ${table3}"
+
+ sql """
+ CREATE TABLE `${table1}` (
+ `delivery_id` bigint(20) NOT NULL,
+ `val_status` varchar(6) NULL,
+ `delivery_type` int(11) NULL,
+ `catalog_name` varchar(96) NULL
+ ) ENGINE=OLAP
+ UNIQUE KEY(`delivery_id`)
+ COMMENT 'OLAP'
+ DISTRIBUTED BY HASH(`delivery_id`) BUCKETS 9
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "storage_format" = "V2",
+ "light_schema_change" = "true",
+ "disable_auto_compaction" = "false",
+ "enable_single_replica_compaction" = "false"
+ );
+ """
+ sql """
+ CREATE TABLE `${table2}` (
+ `delivery_product_id` bigint(20) NOT NULL,
+ `delivery_id` bigint(20) NULL,
+ `nc_num` text NULL,
+ `material_qty` text NULL,
+ `catalog_name` text NULL
+ ) ENGINE=OLAP
+ UNIQUE KEY(`delivery_product_id`)
+ COMMENT 'OLAP'
+ DISTRIBUTED BY HASH(`delivery_product_id`) BUCKETS 9
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "storage_format" = "V2",
+ "light_schema_change" = "true",
+ "disable_auto_compaction" = "false",
+ "enable_single_replica_compaction" = "false"
+ );
+ """
+ sql """
+ CREATE TABLE `${table3}` (
+ `transfer_id` bigint(20) NOT NULL,
+ `delivery_product_id` bigint(20) NULL,
+ `nc_num` text NULL
+ ) ENGINE=OLAP
+ UNIQUE KEY(`transfer_id`)
+ COMMENT '涓浆杩愯緭鍗曚俊鎭�'
+ DISTRIBUTED BY HASH(`transfer_id`) BUCKETS 9
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "storage_format" = "V2",
+ "light_schema_change" = "true",
+ "disable_auto_compaction" = "false",
+ "enable_single_replica_compaction" = "false"
+ );
+ """
+
+ sql """insert into ${table3} values (2, 1, "12");"""
+ sql """insert into ${table2} values (1, 7, "12", "car", "amory is cute"); """
+ sql """insert into ${table1} values (7, 30, 1, "amory clever"); """
+ sql """insert into ${table1} values (1, 90, NULL, "car");"""
+ sql """insert into ${table3} values (1, 2, "12");"""
+
+ qt_select_tt """ select * from ${table3} order by transfer_id;"""
+ qt_select_bd """ select * from ${table1} order by delivery_id;"""
+ qt_select_bdp """ select * from ${table2} order by delivery_product_id;"""
+
+ // this select stmt can trigger column array call update_hashes_with_value
+ qt_select """ select * from (
+ select bdp.nc_num,
+ collect_list(distinct(bd.catalog_name)) as catalog_name,
+ material_qty
+ from ${table2} bdp
+ left join ${table3} btt on bdp.delivery_product_id = btt.delivery_product_id
+ left join ${table1} bd on bdp.delivery_id = bd.delivery_id
+ where bd.val_status in ('10', '20', '30', '90') and bd.delivery_type in (0, 1, 2)
+ group by nc_num, material_qty
+ union all
+ select bdp.nc_num,
+ collect_list(distinct(bd.catalog_name)) as catalog_name,
+ material_qty
+ from ${table3} btt
+ left join ${table2} bdp on bdp.delivery_product_id = btt.delivery_product_id
+ left join ${table1} bd on bdp.delivery_id = bd.delivery_id
+ where bd.val_status in ('10', '20', '30', '90') and bd.delivery_type in (0, 1, 2)
+ group by nc_num, material_qty ) aa;
+ """
+
+ sql "DROP TABLE IF EXISTS ${table1};"
+ sql "DROP TABLE IF EXISTS ${table2};"
+ sql "DROP DATABASE IF EXISTS ${table3};"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 01/06: [Fix](multi-catalog) Add hadoop system classpath to CLASSPATH to resolve can not enable hadoop short circuit reading in some environments. (#21430)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c6dbc5729f4ae629570623a37efb78a1cc5e5614
Author: Qi Chen <ka...@gmail.com>
AuthorDate: Mon Jul 3 14:51:34 2023 +0800
[Fix](multi-catalog) Add hadoop system classpath to CLASSPATH to resolve can not enable hadoop short circuit reading in some environments. (#21430)
Add hadoop system classpath to CLASSPATH to resolve can not enable hadoop short circuit reading in some environments.
---
bin/start_be.sh | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/bin/start_be.sh b/bin/start_be.sh
index e0821b2e00..e4bd71a940 100755
--- a/bin/start_be.sh
+++ b/bin/start_be.sh
@@ -104,7 +104,10 @@ fi
# the CLASSPATH and LIBHDFS_OPTS is used for hadoop libhdfs
# and conf/ dir so that hadoop libhdfs can read .xml config file in conf/
-export CLASSPATH="${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"
+if command -v hadoop >/dev/null 2>&1; then
+ HADOOP_SYSTEM_CLASSPATH="$(hadoop classpath --glob)"
+fi
+export CLASSPATH="${HADOOP_SYSTEM_CLASSPATH}:${DORIS_HOME}/conf/:${DORIS_CLASSPATH}"
# DORIS_CLASSPATH is for self-managed jni
export DORIS_CLASSPATH="-Djava.class.path=${DORIS_CLASSPATH}"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 04/06: [bugfix](build script) fix noavx2 package name branch condition #21356
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 9ba074a597fe6b1490be126089a0d548c5bad69b
Author: Kang <kx...@gmail.com>
AuthorDate: Fri Jun 30 09:03:05 2023 +0800
[bugfix](build script) fix noavx2 package name branch condition #21356
ARCH changed from x86_64 to x64, but the check condition for noavx2 remains x86_64. Just remove check for ARCH.
---
build-for-release.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/build-for-release.sh b/build-for-release.sh
index 976d8217c1..4b48e5f810 100755
--- a/build-for-release.sh
+++ b/build-for-release.sh
@@ -118,7 +118,7 @@ BE="be"
EXT="extensions"
PACKAGE="apache-doris-${VERSION}-bin-${ARCH}"
-if [[ "${_USE_AVX2}" == "0" && "${ARCH}" == "x86_64" ]]; then
+if [[ "${_USE_AVX2}" == "0" ]]; then
PACKAGE="${PACKAGE}-noavx2"
fi
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org