You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2023/08/11 15:48:44 UTC
[doris] branch branch-1.2-lts updated (3a1c5e7ed2 -> b7f163e4d6)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
from 3a1c5e7ed2 [fix](planner)shouldn't change the child type to assignmentCompatible Type if it's INVALID_TYPE (#22844)
new 31c54db5b6 [fix](forward) fix MissingFormatArgumentException when failed to forward stmt to Master (#22142)
new 5baf889336 [bugfix](array) fix element revert on error in DataTypeArray::from_string (#16434)
new eeaa0d2702 [Fix](Planner) fix create view tosql not include partition (#22482)
new c007617d30 [typo](kerberos)support read jdk auth creds and add some krb tips in FAQ (#22535)
new 54af007cb7 [fix](planner) fix bug of push conjuncts through second phase agg (#22417)
new f6092806f3 [Fix](Planner) fix window function in aggregation (#22603)
new b7f163e4d6 [fix](agg) fix coredump caused by push down count aggregation (#22699)
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/vec/data_types/data_type_array.cpp | 6 +-
conf/be.conf | 5 +-
conf/fe.conf | 5 +-
docs/en/docs/lakehouse/multi-catalog/faq.md | 80 ++++++++++++++++++++++
docs/zh-CN/docs/lakehouse/multi-catalog/faq.md | 75 ++++++++++++++++++++
.../java/org/apache/doris/analysis/SelectStmt.java | 10 +++
.../java/org/apache/doris/analysis/TableRef.java | 14 ++--
.../apache/doris/planner/SingleNodePlanner.java | 3 +-
.../java/org/apache/doris/qe/MasterOpExecutor.java | 6 +-
.../load/insert/test_array_insert_into_select.out | 13 ++++
.../test_aggregate_push_down_agg.out} | 2 +-
.../suites/correctness_p0/test_distinct_agg.groovy | 14 ++++
.../suites/ddl_p0/test_create_view.groovy | 42 ++++++++++++
.../insert/test_array_insert_into_select.groovy | 65 ++++++++++++++++++
.../test_aggregate_push_down_agg.groovy} | 50 +++++++-------
.../window_functions/test_window_fn.groovy | 23 +++++++
16 files changed, 366 insertions(+), 47 deletions(-)
create mode 100644 regression-test/data/load/insert/test_array_insert_into_select.out
copy regression-test/data/{datatype_p0/date/test_default_current_timestamp.out => query_p0/sql_functions/aggregate_functions/test_aggregate_push_down_agg.out} (98%)
create mode 100644 regression-test/suites/load/insert/test_array_insert_into_select.groovy
copy regression-test/suites/query_p0/{aggregate/agg_on_view.groovy => sql_functions/aggregate_functions/test_aggregate_push_down_agg.groovy} (51%)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 05/07: [fix](planner) fix bug of push conjuncts through second phase agg (#22417)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 54af007cb7bf1bb7d91672423be4cbc47523d9bb
Author: starocean999 <40...@users.noreply.github.com>
AuthorDate: Fri Aug 4 15:21:18 2023 +0800
[fix](planner) fix bug of push conjuncts through second phase agg (#22417)
If there is a second phase agg, the output of the 1st phase agg is its intermediate tuple not the output tuple.
This pr fix it
---
.../suites/correctness_p0/test_distinct_agg.groovy | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/regression-test/suites/correctness_p0/test_distinct_agg.groovy b/regression-test/suites/correctness_p0/test_distinct_agg.groovy
index 8c80c9f184..ea2cd1f4a4 100644
--- a/regression-test/suites/correctness_p0/test_distinct_agg.groovy
+++ b/regression-test/suites/correctness_p0/test_distinct_agg.groovy
@@ -55,4 +55,18 @@ suite("test_distinct_agg") {
'''
result([['1', '2023-01-10', 1L]])
}
+
+ sql '''SELECT `b`.`dt` AS `dt`
+ FROM
+ (SELECT `dt`AS `dt`,
+ count(DISTINCT `role_id`) AS `pay_role`,
+ avg(`cost`) AS `avg_cost`
+ FROM
+ (SELECT `k6` AS `dt`,
+ `k1` AS `role_id`,
+ sum(CAST(`k2` AS INT)) AS `cost`
+ FROM `t`
+ GROUP BY `dt`, `role_id`) a
+ GROUP BY `dt`) b
+ WHERE `dt` = '2023-06-18';'''
}
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 04/07: [typo](kerberos)support read jdk auth creds and add some krb tips in FAQ (#22535)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c007617d30e5dd0de08b9851bf7826f7b47f9ca4
Author: slothever <18...@users.noreply.github.com>
AuthorDate: Fri Aug 4 14:51:31 2023 +0800
[typo](kerberos)support read jdk auth creds and add some krb tips in FAQ (#22535)
support read jdk auth creds and add some krb tips in FAQ
1. about the 'javax.security.auth.useSubjectCredsOnly': https://stackoverflow.com/questions/43660265/java-automatically-uses-kerberos-ticketcache-when-it-shouldnt
2. add tips for `No common protection layer between client and server` and yum jdk version.
---
conf/be.conf | 5 +-
conf/fe.conf | 5 +-
docs/en/docs/lakehouse/multi-catalog/faq.md | 80 ++++++++++++++++++++++++++
docs/zh-CN/docs/lakehouse/multi-catalog/faq.md | 75 ++++++++++++++++++++++++
4 files changed, 159 insertions(+), 6 deletions(-)
diff --git a/conf/be.conf b/conf/be.conf
index 4604d2b9ad..e25c636bce 100644
--- a/conf/be.conf
+++ b/conf/be.conf
@@ -17,11 +17,10 @@
PPROF_TMPDIR="$DORIS_HOME/log/"
-CUR_DATE=`date +%Y%m%d-%H%M%S`
-JAVA_OPTS="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xloggc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -DJDBC_MIN_POOL=1 -DJDBC_MAX_POOL=100 -DJDBC_MAX_IDEL_TIME=300000 -DJDBC_MAX_WAIT_TIME=5000"
+JAVA_OPTS="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xloggc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -DJDBC_MIN_POOL=1 -DJDBC_MAX_POOL=100 -DJDBC_MAX_IDLE_TIME=300000 -DJDBC_MAX_WAIT_TIME=5000"
# For jdk 9+, this JAVA_OPTS will be used as default JVM options
-JAVA_OPTS_FOR_JDK_9="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xlog:gc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -DJDBC_MIN_POOL=1 -DJDBC_MAX_POOL=100 -DJDBC_MAX_IDEL_TIME=300000 -DJDBC_MAX_WAIT_TIME=5000"
+JAVA_OPTS_FOR_JDK_9="-Xmx1024m -DlogPath=$DORIS_HOME/log/jni.log -Xlog:gc:$DORIS_HOME/log/be.gc.log.$CUR_DATE -Djavax.security.auth.useSubjectCredsOnly=false -Dsun.java.command=DorisBE -XX:-CriticalJNINatives -DJDBC_MIN_POOL=1 -DJDBC_MAX_POOL=100 -DJDBC_MAX_IDLE_TIME=300000 -DJDBC_MAX_WAIT_TIME=5000"
# since 1.2, the JAVA_HOME need to be set to run BE process.
# JAVA_HOME=/path/to/jdk/
diff --git a/conf/fe.conf b/conf/fe.conf
index 4a52b6ee06..896c9b98a5 100644
--- a/conf/fe.conf
+++ b/conf/fe.conf
@@ -24,11 +24,10 @@
# the output dir of stderr and stdout
LOG_DIR = ${DORIS_HOME}/log
-DATE = `date +%Y%m%d-%H%M%S`
-JAVA_OPTS="-Xmx8192m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"
+JAVA_OPTS="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$CUR_DATE"
# For jdk 9+, this JAVA_OPTS will be used as default JVM options
-JAVA_OPTS_FOR_JDK_9="-Xmx8192m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$DATE:time"
+JAVA_OPTS_FOR_JDK_9="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx8192m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$CUR_DATE:time"
##
## the lowercase properties are read by main program.
diff --git a/docs/en/docs/lakehouse/multi-catalog/faq.md b/docs/en/docs/lakehouse/multi-catalog/faq.md
index 9ec3f3ffbc..e73a0a5bf4 100644
--- a/docs/en/docs/lakehouse/multi-catalog/faq.md
+++ b/docs/en/docs/lakehouse/multi-catalog/faq.md
@@ -109,3 +109,83 @@ under the License.
```
'fs.defaultFS' = 'hdfs://<your_nameservice_or_actually_HDFS_IP_and_port>'
```
+12. The values of the partition fields in the hudi table can be found on hive, but they cannot be found on doris.
+
+ Doris and hive currently query hudi differently. Doris needs to add partition fields to the avsc file of the hudi table structure. If not added, it will cause Doris to query partition_ Val is empty (even if home. datasource. live_sync. partition_fields=partition_val is set)
+
+ ```
+ {
+ "type": "record",
+ "name": "record",
+ "fields": [{
+ "name": "partition_val",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Preset partition field, empty string when not partitioned",
+ "default": null
+ },
+ {
+ "name": "name",
+ "type": "string",
+ "doc": "名称"
+ },
+ {
+ "name": "create_time",
+ "type": "string",
+ "doc": "创建时间"
+ }
+ ]
+ }
+ ```
+
+13. The table in orc format of Hive 1.x may encounter system column names such as `_col0`, `_col1`, `_col2`... in the underlying orc file schema, which need to be specified in the catalog configuration. Add `hive.version` to 1.x.x so that it will use the column names in the hive table for mapping.
+
+ ```sql
+ CREATE CATALOG hive PROPERTIES (
+ 'hive.version' = '1.x.x'
+ );
+ ```
+
+14. When using JDBC Catalog to synchronize MySQL data to Doris, the date data synchronization error occurs. It is necessary to check whether the MySQL version corresponds to the MySQL driver package. For example, the driver com.mysql.cj.jdbc.Driver is required for MySQL8 and above.
+
+15. If an error is reported while configuring Kerberos in the catalog: `SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]`.
+
+ Need to put `core-site.xml` to the `"${DORIS_HOME}/be/conf"` directory.
+
+ If an error is reported while accessing HDFS: `No common protection layer between client and server`, check the `hadoop.rpc.protection` on the client and server to make them consistent.
+
+ ```
+ <?xml version="1.0" encoding="UTF-8"?>
+ <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+
+ <configuration>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ </configuration>
+ ```
+
+16. The solutions when configuring Kerberos in the catalog and encounter an error: `Unable to obtain password from user`.
+ - The principal used must exist in the klist, use `klist -kt your.keytab` to check.
+ - Ensure the catalog configuration correct, such as missing the `yarn.resourcemanager.principal`.
+ - If the preceding checks are correct, the JDK version installed by yum or other package-management utility in the current system maybe have an unsupported encryption algorithm. It is recommended to install JDK by yourself and set `JAVA_HOME` environment variable.
+
+17. If an error is reported while querying the catalog with Kerberos: `GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos Ticket)`.
+ - Restarting FE and BE can solve the problem in most cases.
+ - Before the restart all the nodes, can put `-Djavax.security.auth.useSubjectCredsOnly=false` to the `JAVA_OPTS` in `"${DORIS_HOME}/be/conf/be.conf"`, which can obtain credentials through the underlying mechanism, rather than through the application.
+ - Get more solutions to common JAAS errors from the [JAAS Troubleshooting](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/Troubleshooting.html).
+
+18. If an error related to the Hive Metastore is reported while querying the catalog: `Invalid method name`.
+
+ Configure the `hive.version`.
+
+ ```sql
+ CREATE CATALOG hive PROPERTIES (
+ 'hive.version' = '2.x.x'
+ );
+ ```
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/faq.md b/docs/zh-CN/docs/lakehouse/multi-catalog/faq.md
index 2a0b6eeb69..e45f37f056 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/faq.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/faq.md
@@ -110,3 +110,78 @@ under the License.
```
'fs.defaultFS' = 'hdfs://<your_nameservice_or_actually_HDFS_IP_and_port>'
```
+
+12. 在hive上可以查到hudi表分区字段的值,但是在doris查不到。
+
+ doris和hive目前查询hudi的方式不一样,doris需要在hudi表结构的avsc文件里添加上分区字段,如果没加,就会导致doris查询partition_val为空(即使设置了hoodie.datasource.hive_sync.partition_fields=partition_val也不可以)
+ ```
+ {
+ "type": "record",
+ "name": "record",
+ "fields": [{
+ "name": "partition_val",
+ "type": [
+ "null",
+ "string"
+ ],
+ "doc": "Preset partition field, empty string when not partitioned",
+ "default": null
+ },
+ {
+ "name": "name",
+ "type": "string",
+ "doc": "名称"
+ },
+ {
+ "name": "create_time",
+ "type": "string",
+ "doc": "创建时间"
+ }
+ ]
+ }
+ ```
+
+13. Hive 1.x 的 orc 格式的表可能会遇到底层 orc 文件 schema 中列名为 `_col0`,`_col1`,`_col2`... 这类系统列名,此时需要在 catalog 配置中添加 `hive.version` 为 1.x.x,这样就会使用 hive 表中的列名进行映射。
+
+ ```sql
+ CREATE CATALOG hive PROPERTIES (
+ 'hive.version' = '1.x.x'
+ );
+ ```
+
+14. 使用JDBC Catalog将MySQL数据同步到Doris中,日期数据同步错误。需要校验下MySQL的版本是否与MySQL的驱动包是否对应,比如MySQL8以上需要使用驱动com.mysql.cj.jdbc.Driver。
+
+15. 在Catalog中配置Kerberos时,如果报错`SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]`,那么需要将`core-site.xml`文件放到`"${DORIS_HOME}/be/conf"`目录下。
+
+ 如果访问HDFS报错`No common protection layer between client and server`,检查客户端和服务端的`hadoop.rpc.protection`属性,使他们保持一致。
+
+ ```
+ <?xml version="1.0" encoding="UTF-8"?>
+ <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+
+ <configuration>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ </configuration>
+ ```
+
+16. 在Catalog中配置Kerberos时,报错`Unable to obtain password from user`的解决方法:
+ - 用到的principal必须在klist中存在,使用`klist -kt your.keytab`检查。
+ - 检查catalog配置是否正确,比如漏配`yarn.resourcemanager.principal`。
+ - 若上述检查没问题,则当前系统yum或者其他包管理软件安装的JDK版本存在不支持的加密算法,建议自行安装JDK并设置`JAVA_HOME`环境变量。
+
+17. 查询配置了Kerberos的外表,遇到该报错:`GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos Ticket)`,一般重启FE和BE能够解决该问题。
+ - 重启所有节点前可在`"${DORIS_HOME}/be/conf/be.conf"`中的JAVA_OPTS参数里配置`-Djavax.security.auth.useSubjectCredsOnly=false`,通过底层机制去获取JAAS credentials信息,而不是应用程序。
+ - 在[JAAS Troubleshooting](https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/Troubleshooting.html)中可获取更多常见JAAS报错的解决方法。
+
+18. 使用Catalog查询表数据时发现与Hive Metastore相关的报错:`Invalid method name`,需要设置`hive.version`参数。
+
+ ```sql
+ CREATE CATALOG hive PROPERTIES (
+ 'hive.version' = '1.x.x'
+ );
+ ```
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 02/07: [bugfix](array) fix element revert on error in DataTypeArray::from_string (#16434)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5baf88933610420fea2a0de9f7f8d2279e30ff53
Author: Kang <kx...@gmail.com>
AuthorDate: Mon Feb 6 18:27:36 2023 +0800
[bugfix](array) fix element revert on error in DataTypeArray::from_string (#16434)
* fix array from_string element revert on error
* add testcase
---
be/src/vec/data_types/data_type_array.cpp | 6 +-
.../load/insert/test_array_insert_into_select.out | 13 +++++
.../insert/test_array_insert_into_select.groovy | 65 ++++++++++++++++++++++
3 files changed, 82 insertions(+), 2 deletions(-)
diff --git a/be/src/vec/data_types/data_type_array.cpp b/be/src/vec/data_types/data_type_array.cpp
index 6827cb3779..97af5ac720 100644
--- a/be/src/vec/data_types/data_type_array.cpp
+++ b/be/src/vec/data_types/data_type_array.cpp
@@ -267,6 +267,8 @@ Status DataTypeArray::from_string(ReadBuffer& rb, IColumn* column) const {
StringRef element(rb.position(), rb.count());
bool has_quota = false;
if (!next_element_from_string(rb, element, has_quota)) {
+ // we should do array element column revert if error
+ nested_column.pop_back(element_num);
return Status::InvalidArgument("Cannot read array element from text '{}'",
element.to_string());
}
@@ -294,8 +296,8 @@ Status DataTypeArray::from_string(ReadBuffer& rb, IColumn* column) const {
ReadBuffer read_buffer(const_cast<char*>(element.data), element.size);
auto st = nested->from_string(read_buffer, &nested_column);
if (!st.ok()) {
- // we should do revert if error
- array_column->pop_back(element_num);
+ // we should do array element column revert if error
+ nested_column.pop_back(element_num);
return st;
}
++element_num;
diff --git a/regression-test/data/load/insert/test_array_insert_into_select.out b/regression-test/data/load/insert/test_array_insert_into_select.out
new file mode 100644
index 0000000000..2dcf990314
--- /dev/null
+++ b/regression-test/data/load/insert/test_array_insert_into_select.out
@@ -0,0 +1,13 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !select_string --
+0 [{"a":"1","b":"2","c":"3",}, {"a":"1","b":"2"}]
+1 ['{"a":"1","b":"2","c":"3",}', '{"a":"1","b":"2"}']
+2 [{"a":"1","b":"2"}]
+3 []
+
+-- !select_array --
+0 \N
+1 ['{"a":"1","b":"2","c":"3",}', '{"a":"1","b":"2"}']
+2 \N
+3 []
+
diff --git a/regression-test/suites/load/insert/test_array_insert_into_select.groovy b/regression-test/suites/load/insert/test_array_insert_into_select.groovy
new file mode 100644
index 0000000000..f171ae3220
--- /dev/null
+++ b/regression-test/suites/load/insert/test_array_insert_into_select.groovy
@@ -0,0 +1,65 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_array_insert_into_select", "load") {
+ // create table with string
+ sql "DROP TABLE IF EXISTS tstring"
+ sql """
+ CREATE TABLE `tstring` (
+ `id` INT,
+ `data` STRING
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ COMMENT 'OLAP'
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1");
+ """
+
+ // insert invalid row 0
+ sql """INSERT INTO tstring VALUES (0, '[{"a":"1","b":"2","c":"3",}, {"a":"1","b":"2"}]')"""
+ // insert valid row 1
+ sql """INSERT INTO tstring VALUES (1, '[\\'{"a":"1","b":"2","c":"3",}\\', \\'{"a":"1","b":"2"}\\']')"""
+ // insert invalid row 2
+ sql """INSERT INTO tstring VALUES (2, '[{"a":"1","b":"2"}]')"""
+ // insert valid row 3
+ sql """INSERT INTO tstring VALUES (3, '[]')"""
+
+ // check data in tstring
+ qt_select_string "SELECT * FROM tstring ORDER BY id"
+
+
+ // create table with array
+ sql "DROP TABLE IF EXISTS tarray"
+ sql """
+ CREATE TABLE `tarray` (
+ `id` INT,
+ `data` ARRAY<STRING>
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`id`)
+ COMMENT 'OLAP'
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1");
+ """
+
+ // insert rows to tarray
+ sql "INSERT INTO tarray SELECT * FROM tstring"
+
+ // check data in tarray
+ qt_select_array "SELECT * FROM tarray ORDER BY id"
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 01/07: [fix](forward) fix MissingFormatArgumentException when failed to forward stmt to Master (#22142)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 31c54db5b6ca93c9f0b05a4da04b4bbddf61a95a
Author: Mingyu Chen <mo...@163.com>
AuthorDate: Wed Jul 26 09:00:04 2023 +0800
[fix](forward) fix MissingFormatArgumentException when failed to forward stmt to Master (#22142)
---
fe/fe-core/src/main/java/org/apache/doris/qe/MasterOpExecutor.java | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/MasterOpExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/MasterOpExecutor.java
index 5ba69a51c2..d41edf7084 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/MasterOpExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/MasterOpExecutor.java
@@ -114,9 +114,9 @@ public class MasterOpExecutor {
// may throw NullPointerException. add err msg
throw new Exception("Failed to get master client.", e);
}
- final StringBuilder forwardMsg = new StringBuilder(String.format("forward to Master %s", thriftAddress));
+ final StringBuilder forwardMsg = new StringBuilder("forward to master FE %s" + thriftAddress.toString());
if (!params.isSyncJournalOnly()) {
- forwardMsg.append(", statement: %s").append(ctx.getStmtId());
+ forwardMsg.append(", statement id: ").append(ctx.getStmtId());
}
LOG.info(forwardMsg.toString());
@@ -128,7 +128,7 @@ public class MasterOpExecutor {
} catch (TTransportException e) {
// wrap the raw exception.
forwardMsg.append(" : failed");
- Exception exception = new ForwardToMasterException(String.format(forwardMsg.toString()), e);
+ Exception exception = new ForwardToMasterException(forwardMsg.toString(), e);
boolean ok = ClientPool.frontendPool.reopen(client, thriftTimeoutMs);
if (!ok) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 07/07: [fix](agg) fix coredump caused by push down count aggregation (#22699)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b7f163e4d6d87032194dd2d5e804539eaf348c63
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Wed Aug 9 10:21:20 2023 +0800
[fix](agg) fix coredump caused by push down count aggregation (#22699)
fix coredump caused by push down count aggregation
---
.../apache/doris/planner/SingleNodePlanner.java | 3 +-
.../test_aggregate_push_down_agg.out | 4 ++
.../test_aggregate_push_down_agg.groovy | 44 ++++++++++++++++++++++
3 files changed, 50 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java b/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
index 5d59df3a8a..5cc1b85b54 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
@@ -420,7 +420,8 @@ public class SingleNodePlanner {
break;
}
- if (CollectionUtils.isNotEmpty(root.getConjuncts())) {
+ if (CollectionUtils.isNotEmpty(root.getConjuncts())
+ || CollectionUtils.isNotEmpty(root.getProjectList())) {
break;
}
diff --git a/regression-test/data/query_p0/sql_functions/aggregate_functions/test_aggregate_push_down_agg.out b/regression-test/data/query_p0/sql_functions/aggregate_functions/test_aggregate_push_down_agg.out
new file mode 100644
index 0000000000..f88a155567
--- /dev/null
+++ b/regression-test/data/query_p0/sql_functions/aggregate_functions/test_aggregate_push_down_agg.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !sql --
+3
+
diff --git a/regression-test/suites/query_p0/sql_functions/aggregate_functions/test_aggregate_push_down_agg.groovy b/regression-test/suites/query_p0/sql_functions/aggregate_functions/test_aggregate_push_down_agg.groovy
new file mode 100644
index 0000000000..443377f8e2
--- /dev/null
+++ b/regression-test/suites/query_p0/sql_functions/aggregate_functions/test_aggregate_push_down_agg.groovy
@@ -0,0 +1,44 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_aggregate_push_down_agg") {
+ sql """ drop table if exists test_aggregate_push_down_agg; """
+ sql """
+ create table test_aggregate_push_down_agg (
+ k1 int,
+ tracking_number varchar(16)
+ ) duplicate key(k1)
+ distributed by hash(k1)
+ properties("replication_num"="1");
+ """
+ sql """ insert into test_aggregate_push_down_agg values(1, "aaa"), (1, "bbb"), (2, "ccc") """
+ sql 'set enable_nereids_planner=false'
+ qt_sql """
+ SELECT
+ count(*)
+ FROM
+ (
+ select
+ case
+ when tracking_number is null then ''
+ else tracking_number
+ end as tracking_number
+ from
+ test_aggregate_push_down_agg
+ ) V_T;
+ """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 03/07: [Fix](Planner) fix create view tosql not include partition (#22482)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit eeaa0d270218b90a468ce282e515bcfaae6cf349
Author: LiBinfeng <46...@users.noreply.github.com>
AuthorDate: Wed Aug 2 20:04:59 2023 +0800
[Fix](Planner) fix create view tosql not include partition (#22482)
Problem:
When create view with join in table partitions, an error would rise like "Unknown column"
Example:
CREATE VIEW my_view AS SELECT t1.* FROM t1 PARTITION(p1) JOIN t2 PARTITION(p2) ON t1.k1 = t2.k1;
select * from my_view ==> errCode = 2, detailMessage = Unknown column 'k1' in 't2'
Reason:
When create view, we do tosql first in order to persistent view sql. And when doing tosql of table reference, partition key
word was removed to keep neat of sql string. But here when we remove partition keyword it would regarded as an alias.
So "PARTITION" keyword can not be removed.
Solved:
Add “PARTITION” keyword back to tosql string.
---
.../java/org/apache/doris/analysis/TableRef.java | 14 ++++----
.../suites/ddl_p0/test_create_view.groovy | 42 ++++++++++++++++++++++
2 files changed, 49 insertions(+), 7 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/TableRef.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/TableRef.java
index 7e9deadd84..377a3bc9ed 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/TableRef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/TableRef.java
@@ -247,13 +247,6 @@ public class TableRef implements ParseNode, Writable {
output.append("[").append(Joiner.on(", ").join(joinHints)).append("] ");
}
output.append(tableRefToSql()).append(" ");
- if (partitionNames != null) {
- StringJoiner sj = new StringJoiner(",", "", " ");
- for (String partName : partitionNames.getPartitionNames()) {
- sj.add(partName);
- }
- output.append(sj.toString());
- }
if (usingColNames != null) {
output.append("USING (").append(Joiner.on(", ").join(usingColNames)).append(")");
} else if (onClause != null) {
@@ -754,6 +747,13 @@ public class TableRef implements ParseNode, Writable {
tblName += " " + viewRef.toSql();
}
}
+ if (partitionNames != null) {
+ StringJoiner sj = new StringJoiner(",", "", " ");
+ for (String partName : partitionNames.getPartitionNames()) {
+ sj.add(partName);
+ }
+ return tblName + " PARTITION(" + sj.toString() + ")";
+ }
return tblName;
}
diff --git a/regression-test/suites/ddl_p0/test_create_view.groovy b/regression-test/suites/ddl_p0/test_create_view.groovy
index 4c401017ee..c209d42bd3 100644
--- a/regression-test/suites/ddl_p0/test_create_view.groovy
+++ b/regression-test/suites/ddl_p0/test_create_view.groovy
@@ -69,4 +69,46 @@ suite("test_create_view") {
sql """select * from test_count_distinct"""
sql """DROP VIEW IF EXISTS test_count_distinct"""
sql """DROP TABLE IF EXISTS count_distinct"""
+
+ sql """DROP TABLE IF EXISTS t1"""
+ sql """
+ CREATE TABLE `t1` (
+ k1 int,
+ k2 date,
+ v1 int
+ ) ENGINE=OLAP
+ UNIQUE KEY(`k1`,`k2`)
+ COMMENT '测试'
+ PARTITION BY RANGE(k2) (
+ PARTITION p1 VALUES [('2023-07-01'), ('2023-07-10')),
+ PARTITION p2 VALUES [('2023-07-11'), ('2023-07-20'))
+ )
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );"""
+ sql """DROP TABLE IF EXISTS t2"""
+ sql """
+ CREATE TABLE `t2` (
+ k1 int,
+ k2 date,
+ v1 int
+ ) ENGINE=OLAP
+ UNIQUE KEY(`k1`,`k2`)
+ COMMENT '测试'
+ PARTITION BY RANGE(k2) (
+ PARTITION p1 VALUES [('2023-07-01'), ('2023-07-05')),
+ PARTITION p2 VALUES [('2023-07-05'), ('2023-07-15'))
+ )
+ DISTRIBUTED BY HASH(`k1`) BUCKETS 3
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ ); """
+ sql """
+ CREATE VIEW IF NOT EXISTS my_view AS
+ SELECT t1.* FROM t1 PARTITION(p1) JOIN t2 PARTITION(p2) ON t1.k1 = t2.k1; """
+ sql """SELECT * FROM my_view"""
+ sql """DROP VIEW IF EXISTS my_view"""
+ sql """DROP TABLE IF EXISTS t1"""
+ sql """DROP TABLE IF EXISTS t2"""
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 06/07: [Fix](Planner) fix window function in aggregation (#22603)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-1.2-lts
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f6092806f39cad4f0b1ef005e353748e759ed067
Author: LiBinfeng <46...@users.noreply.github.com>
AuthorDate: Fri Aug 4 19:15:51 2023 +0800
[Fix](Planner) fix window function in aggregation (#22603)
Problem:
When window function in aggregation function, executor would report an error like: Required field 'node_type' was not present!
Example:
SELECT SUM(MAX(c1) OVER (PARTITION BY c2, c3)) FROM test_window_in_agg;
Reason:
When analyze aggregate, analytic expr (window function carrior when analyze) transfered to slot and loss message. So when
serialize to thrift package, TExpr can not determine node_type of analytic expr.
Solved:
We do not support aggregate(window function) yet. So we report an error when analyze.
---
.../java/org/apache/doris/analysis/SelectStmt.java | 10 ++++++++++
.../window_functions/test_window_fn.groovy | 23 ++++++++++++++++++++++
2 files changed, 33 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
index d634b5fb62..22b9c0a82d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
@@ -1133,6 +1133,16 @@ public class SelectStmt extends QueryStmt {
}
}
+ // can't contain analytic exprs
+ ArrayList<Expr> aggExprsForChecking = Lists.newArrayList();
+ TreeNode.collect(resultExprs, Expr.isAggregatePredicate(), aggExprsForChecking);
+ ArrayList<Expr> analyticExprs = Lists.newArrayList();
+ TreeNode.collect(aggExprsForChecking, AnalyticExpr.class, analyticExprs);
+ if (!analyticExprs.isEmpty()) {
+ throw new AnalysisException(
+ "AGGREGATE clause must not contain analytic expressions");
+ }
+
// Collect the aggregate expressions from the SELECT, HAVING and ORDER BY clauses
// of this statement.
ArrayList<FunctionCallExpr> aggExprs = Lists.newArrayList();
diff --git a/regression-test/suites/query_p0/sql_functions/window_functions/test_window_fn.groovy b/regression-test/suites/query_p0/sql_functions/window_functions/test_window_fn.groovy
index c1842d730f..0b7240425c 100644
--- a/regression-test/suites/query_p0/sql_functions/window_functions/test_window_fn.groovy
+++ b/regression-test/suites/query_p0/sql_functions/window_functions/test_window_fn.groovy
@@ -365,6 +365,29 @@ suite("test_window_fn") {
"""
sql "DROP TABLE IF EXISTS example_window_tb;"
+
+ sql """
+ CREATE TABLE IF NOT EXISTS test_window_in_agg
+ (
+ `c1` int ,
+ `c2` int ,
+ `c3` int
+ )
+ ENGINE=OLAP
+ DUPLICATE KEY(`c1`)
+ COMMENT ""
+ DISTRIBUTED BY HASH(`c1`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "in_memory" = "false",
+ "storage_format" = "V2"
+ );
+ """
+ test {
+ sql """SELECT SUM(MAX(c1) OVER (PARTITION BY c2, c3)) FROM test_window_in_agg;"""
+ exception "errCode = 2, detailMessage = AGGREGATE clause must not contain analytic expressions"
+ }
+ sql "DROP TABLE IF EXISTS test_window_in_agg;"
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org