You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by yi...@apache.org on 2024/02/21 06:03:15 UTC
(doris) branch branch-2.1 updated (4735c5b50fa -> c27692fb3bd)
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
from 4735c5b50fa 2.1.0-rc08
new 0ab81435656 [fix](catalog recycle bin) table partition meta is error if recover partition in some case (#31125)
new 99cdbc7e12a [fix](memory) Fix jemalloc lib name for thirdparty arrow (#28843)
new 87b5ed187e5 Fix hive p2 case (#31149)
new 7c1bb4cb22d [fix](docs) add joinHint.md to sidebars (#31155)
new e2514c950c6 Fix load stats cache FE ut case. (#31162)
new 27ae96c5d03 [fix](case) adjust case if running on muti BEs (#31160)
new 872e1f6687c [fix](backup) fix concurrent upload and release snapshot crash (#31144)
new 1d6dc9a5f09 [fix](catalog recycle bin) Forbid recover partition if table schame is changed #31146
new 69c7750db2d [Improment](executor)Create workload thread pool without cgroup #31170
new ccac4208e1f [fix](regression test) fix test_dynamic_partition_with_alter #31154
new 1e3968fe7ee [fix](group_commit) Need to wait wal to be deleted when creating MaterializedView (#30956)
new c734e79d146 [docs](docs) fix doc MySQL Load format (#30182)
new cd7230885f2 [fix](nereids)push more than one runtime filters into cte (#30901)
new 8b6d6d01657 [pipelineX](refactor) refactor streaming agg structure (#31151)
new 278b232e76f [Bug](json reader) object should stop processing when encounter error (#31159)
new 728b12cfdf5 [regression-test](point query) make test stable (#31140)
new f121ee907f6 [fix](invert index) fix the inaccurate rows_inverted_index_filtered in the profile (#31158)
new a4f9eec810c [chore](third-party) Link protobuf with zlib statically (#31173)
new c27692fb3bd [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog (#31153)
The 19 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/build-thirdparty.yml | 4 +-
be/cmake/thirdparty.cmake | 2 +-
be/src/agent/workload_group_listener.cpp | 10 +-
be/src/exprs/json_functions.cpp | 2 +-
be/src/io/fs/s3_file_system.cpp | 5 +-
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 2 +
be/src/olap/wal/wal_manager.cpp | 2 +-
be/src/pipeline/exec/aggregation_sink_operator.cpp | 172 +--
be/src/pipeline/exec/aggregation_sink_operator.h | 57 +-
.../pipeline/exec/aggregation_source_operator.cpp | 6 +-
be/src/pipeline/exec/aggregation_source_operator.h | 7 +-
be/src/pipeline/exec/data_queue.cpp | 14 +-
be/src/pipeline/exec/data_queue.h | 4 +-
.../distinct_streaming_aggregation_operator.cpp | 406 ++++++
.../exec/distinct_streaming_aggregation_operator.h | 128 ++
...istinct_streaming_aggregation_sink_operator.cpp | 151 ---
.../distinct_streaming_aggregation_sink_operator.h | 56 -
...tinct_streaming_aggregation_source_operator.cpp | 49 -
...istinct_streaming_aggregation_source_operator.h | 14 -
be/src/pipeline/exec/empty_source_operator.h | 2 +-
be/src/pipeline/exec/join_probe_operator.cpp | 6 +-
be/src/pipeline/exec/join_probe_operator.h | 2 +-
be/src/pipeline/exec/multi_cast_data_stream_sink.h | 2 +-
.../exec/multi_cast_data_stream_source.cpp | 2 +-
.../pipeline/exec/multi_cast_data_stream_source.h | 2 +-
.../exec/nested_loop_join_probe_operator.h | 2 +-
be/src/pipeline/exec/operator.cpp | 2 +-
be/src/pipeline/exec/operator.h | 8 +-
.../exec/streaming_aggregation_operator.cpp | 1302 ++++++++++++++++++++
.../pipeline/exec/streaming_aggregation_operator.h | 231 ++++
.../exec/streaming_aggregation_sink_operator.cpp | 298 -----
.../exec/streaming_aggregation_sink_operator.h | 61 -
.../exec/streaming_aggregation_source_operator.cpp | 39 -
.../exec/streaming_aggregation_source_operator.h | 13 -
be/src/pipeline/pipeline_x/dependency.h | 4 +-
.../local_exchange_source_operator.h | 2 +-
be/src/pipeline/pipeline_x/operator.cpp | 17 +-
be/src/pipeline/pipeline_x/operator.h | 2 +-
.../pipeline_x/pipeline_x_fragment_context.cpp | 39 +-
be/src/runtime/task_group/task_group_manager.cpp | 50 +-
be/src/runtime/task_group/task_group_manager.h | 2 +-
be/src/vec/exec/format/json/new_json_reader.cpp | 6 +-
docs/en/docs/admin-manual/workload-group.md | 10 +-
.../import/import-scenes/local-file-load.md | 12 +-
.../import/import-way/mysql-load-manual.md | 28 +-
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 24 +-
docs/en/docs/query-acceleration/hint/joinHint.md | 2 +
.../Show-Statements/SHOW-CATALOG-RECYCLE-BIN.md | 2 +
docs/sidebars.json | 7 +
docs/zh-CN/docs/admin-manual/workload-group.md | 12 +-
.../import/import-scenes/local-file-load.md | 12 +-
.../import/import-way/mysql-load-manual.md | 14 +-
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md | 24 +-
.../zh-CN/docs/query-acceleration/hint/joinHint.md | 3 +
.../Show-Statements/SHOW-CATALOG-RECYCLE-BIN.md | 2 +
.../main/java/org/apache/doris/common/Config.java | 8 +
.../doris/alter/MaterializedViewHandler.java | 10 +
.../org/apache/doris/alter/SchemaChangeJobV2.java | 33 +-
.../apache/doris/catalog/CatalogRecycleBin.java | 20 +-
.../org/apache/doris/catalog/JdbcResource.java | 19 +-
.../doris/datasource/jdbc/JdbcExternalCatalog.java | 24 +-
.../org/apache/doris/load/GroupCommitManager.java | 28 +
.../glue/translator/PhysicalPlanTranslator.java | 6 +-
.../glue/translator/RuntimeFilterTranslator.java | 7 -
.../processor/post/RuntimeFilterContext.java | 41 +-
.../processor/post/RuntimeFilterGenerator.java | 57 +-
.../org/apache/doris/catalog/JdbcResourceTest.java | 15 +
.../nereids/postprocess/RuntimeFilterTest.java | 2 +-
.../org/apache/doris/statistics/CacheTest.java | 28 +-
.../recover_with_schema_change.out | 39 +
.../data/insert_p0/insert_group_commit_into.out | 18 +
.../data/load_p0/stream_load/test_json_load.out | 4 +
.../stream_load/test_malformed_json_with_path.json | 3 +
.../noStatsRfPrune/query95.out | 2 +-
.../no_stats_shape/query95.out | 2 +-
.../doris/regression/action/TestAction.groovy | 1 +
.../suites/catalog_recycle_bin_p0/recover.groovy | 60 +
.../recover_with_schema_change.groovy | 127 ++
.../decimalv3/test_decimal256_outfile_csv.groovy | 5 +
.../hive/test_hive_analyze_db.groovy | 226 ++--
.../insert_p0/insert_group_commit_into.groovy | 126 ++
.../load_p0/stream_load/test_json_load.groovy | 33 +-
.../test_dynamic_partition_with_alter.groovy | 19 +-
.../suites/point_query_p0/test_point_query.groovy | 18 +-
.../test_point_query_cluster_key.groovy | 19 +-
thirdparty/build-thirdparty.sh | 4 +-
86 files changed, 3068 insertions(+), 1273 deletions(-)
create mode 100644 be/src/pipeline/exec/distinct_streaming_aggregation_operator.cpp
create mode 100644 be/src/pipeline/exec/distinct_streaming_aggregation_operator.h
create mode 100644 be/src/pipeline/exec/streaming_aggregation_operator.cpp
create mode 100644 be/src/pipeline/exec/streaming_aggregation_operator.h
create mode 100644 regression-test/data/catalog_recycle_bin_p0/recover_with_schema_change.out
create mode 100644 regression-test/data/load_p0/stream_load/test_malformed_json_with_path.json
create mode 100644 regression-test/suites/catalog_recycle_bin_p0/recover.groovy
create mode 100644 regression-test/suites/catalog_recycle_bin_p0/recover_with_schema_change.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 12/19: [docs](docs) fix doc MySQL Load format (#30182)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c734e79d146a056f74ccf52df71320a7c186d141
Author: Stalary <st...@163.com>
AuthorDate: Wed Feb 21 09:09:21 2024 +0800
[docs](docs) fix doc MySQL Load format (#30182)
* chore: fix doc MySql Load format
* chore: modify mysql-load-manual.md
* chore: modify to MySQL
---
.../import/import-scenes/local-file-load.md | 12 +++++-----
.../import/import-way/mysql-load-manual.md | 28 +++++++++++-----------
.../import/import-scenes/local-file-load.md | 12 +++++-----
.../import/import-way/mysql-load-manual.md | 14 +++++------
4 files changed, 33 insertions(+), 33 deletions(-)
diff --git a/docs/en/docs/data-operate/import/import-scenes/local-file-load.md b/docs/en/docs/data-operate/import/import-scenes/local-file-load.md
index 187bc5e39bc..d022ceded04 100644
--- a/docs/en/docs/data-operate/import/import-scenes/local-file-load.md
+++ b/docs/en/docs/data-operate/import/import-scenes/local-file-load.md
@@ -29,7 +29,7 @@ The following mainly introduces how to import local data in client.
Now Doris support two way to load data from client local file:
1. [Stream Load](../import-way/stream-load-manual.md)
-2. [MySql Load](../import-way/mysql-load-manual.md)
+2. [MySQL Load](../import-way/mysql-load-manual.md)
## Stream Load
@@ -224,9 +224,9 @@ public class DorisStreamLoader {
> </dependency>
> ```
-## MySql LOAD
+## MySQL Load
<version since="dev">
- Example of mysql load
+ Example of MySQL Load
</version>
### Import Data
@@ -246,7 +246,7 @@ public class DorisStreamLoader {
````
2. Import data
- Excute fellowing sql statmeent in the mysql client to load client local file:
+ Excute fellowing sql statmeent in the MySQL client to load client local file:
```sql
LOAD DATA
@@ -259,7 +259,7 @@ public class DorisStreamLoader {
3. Wait for the import result
- The MySql Load command is a synchronous command, and a successful return indicates that the import is successful. If the imported data is large, a longer waiting time may be required. Examples are as follows:
+ The MySQL Load command is a synchronous command, and a successful return indicates that the import is successful. If the imported data is large, a longer waiting time may be required. Examples are as follows:
```text
Query OK, 1 row affected (0.17 sec)
@@ -271,5 +271,5 @@ public class DorisStreamLoader {
### Import Suggestion
- - MySql Load can only import local files(which can be client local file or fe local file) and only support csv format.
+ - MySQL Load can only import local files(which can be client local file or fe local file) and only support csv format.
- It is recommended to limit the amount of data for an import request to 1 - 2 GB. If you have a large number of local files, you can submit them concurrently in batches.
diff --git a/docs/en/docs/data-operate/import/import-way/mysql-load-manual.md b/docs/en/docs/data-operate/import/import-way/mysql-load-manual.md
index 06592878c89..826b03654e9 100644
--- a/docs/en/docs/data-operate/import/import-way/mysql-load-manual.md
+++ b/docs/en/docs/data-operate/import/import-way/mysql-load-manual.md
@@ -1,6 +1,6 @@
---
{
- "title": "MySql Load",
+ "title": "MySQL Load",
"language": "en"
}
---
@@ -24,24 +24,24 @@ specific language governing permissions and limitations
under the License.
-->
-# Mysql Load
+# MySQL Load
<version since="dev">
-This is an stand syntax of MySql [LOAD DATA](https://dev.mysql.com/doc/refman/8.0/en/load-data.html) for user to load local file.
+This is an stand syntax of MySQL [LOAD DATA](https://dev.mysql.com/doc/refman/8.0/en/load-data.html) for user to load local file.
-MySql load synchronously executes the import and returns the import result. The return information will show whether the import is successful for user.
+MySQL load synchronously executes the import and returns the import result. The return information will show whether the import is successful for user.
-MySql load is mainly suitable for importing local files on the client side, or importing data from a data stream through a program.
+MySQL load is mainly suitable for importing local files on the client side, or importing data from a data stream through a program.
</version>
## Basic Principles
-The MySql Load are similar with Stream Load. Both import local files into the Doris cluster, so the MySQL Load will reuses StreamLoad:
+The MySQL Load are similar with Stream Load. Both import local files into the Doris cluster, so the MySQL Load will reuses StreamLoad:
1. FE receives the MySQL Load request executed by the client and then analyse the SQL
- 2. FE build the MySql Load request as a StreamLoad request.
+ 2. FE build the MySQL Load request as a StreamLoad request.
3. FE selects a BE node to send a StreamLoad request
@@ -52,7 +52,7 @@ The MySql Load are similar with Stream Load. Both import local files into the Do
## Support data format
-MySql Load currently only supports data formats: CSV (text).
+MySQL Load currently only supports data formats: CSV (text).
## Basic operations
@@ -61,7 +61,7 @@ MySql Load currently only supports data formats: CSV (text).
mysql --local-infile -h 127.0.0.1 -P 9030 -u root -D testdb
```
-Notice that if you wants to use mysql load, you must connect doris server with `--local-infile` in client command.
+Notice that if you wants to use MySQL load, you must connect doris server with `--local-infile` in client command.
If you're use jdbc to connect doris, you must add property named `allowLoadLocalInfile=true` in jdbc url.
@@ -70,7 +70,7 @@ If you're use jdbc to connect doris, you must add property named `allowLoadLocal
CREATE TABLE testdb.t1 (pk INT, v1 INT SUM) AGGREGATE KEY (pk) DISTRIBUTED BY hash (pk) PROPERTIES ('replication_num' = '1');
```
### import file from client node
- Suppose there is a CSV file named 'client_local.csv 'on the current path of the client side, which will be imported into the test table'testdb.t1' using the MySQL LOAD syntax.
+ Suppose there is a CSV file named 'client_local.csv 'on the current path of the client side, which will be imported into the test table'testdb.t1' using the MySQL Load syntax.
```sql
LOAD DATA LOCAL
@@ -114,7 +114,7 @@ PROPERTIES ("strict_mode" = "true")
3. Server side load was disabled by default. Enable it by setting `mysql_load_server_secure_path` with a secure path. All the load file should be under this path.
### Return result
-Since MySQL load is a synchronous import method, the imported results are returned to the user through SQL syntax.
+Since MySQL Load is a synchronous import method, the imported results are returned to the user through SQL syntax.
If the import fails, a specific error message will be displayed. If the import is successful, the number of imported rows will be displayed.
```Text
@@ -123,7 +123,7 @@ Records: 1 Deleted: 0 Skipped: 0 Warnings: 0
```
### Error result
-If mysql load process goes wrong, it will show the error in the client as below:
+If MySQL Load process goes wrong, it will show the error in the client as below:
```text
ERROR 1105 (HY000): errCode = 2, detailMessage = [DATA_QUALITY_ERROR]too many filtered rows with load id b612907c-ccf4-4ac2-82fe-107ece655f0f
```
@@ -139,12 +139,12 @@ The loadId was the label in this case.
### Configuration
1. `mysql_load_thread_pool`: the thread pool size for singe FE node, set 4 thread by default. The block queue size is 5 times of `mysql_load_thread_pool`. So FE can accept 4 + 4\*5 = 24 requests in one time. Increase this configuration if the parallelism are larger than 24.
2. `mysql_load_server_secure_path`: the secure path for load data from server. Empty path by default means that it's not allowed for server load. Recommend to create a `local_import_data` directory under `DORIS_HOME` to load data if you want enable it.
-3. `mysql_load_in_memory_record` The failed mysql load record size. The record was keep in memory and only have 20 records by default. If you want to track more records, you can rise the config but be careful about the fe memory. This record will expired after one day and there is a async thread to clean it in every day.
+3. `mysql_load_in_memory_record` The failed MySQL Load record size. The record was keep in memory and only have 20 records by default. If you want to track more records, you can rise the config but be careful about the fe memory. This record will expired after one day and there is a async thread to clean it in every day.
## Notice
-1. If you see this `LOAD DATA LOCAL INFILE file request rejected due to restrictions on access` message, you should connet mysql with `mysql --local-infile=1` command to enable client to load local file.
+1. If you see this `LOAD DATA LOCAL INFILE file request rejected due to restrictions on access` message, you should connet MySQL with `mysql --local-infile=1` command to enable client to load local file.
2. The configuration for StreamLoad will also affect MySQL Load. Such as the configurate in be named `streaming_load_max_mb`, it's 10GB by default and it will control the max size for one load.
## More Help
diff --git a/docs/zh-CN/docs/data-operate/import/import-scenes/local-file-load.md b/docs/zh-CN/docs/data-operate/import/import-scenes/local-file-load.md
index be023fe3540..ed87cbf8975 100644
--- a/docs/zh-CN/docs/data-operate/import/import-scenes/local-file-load.md
+++ b/docs/zh-CN/docs/data-operate/import/import-scenes/local-file-load.md
@@ -30,7 +30,7 @@ under the License.
目前Doris支持两种从本地导入数据的模式:
1. [Stream Load](../import-way/stream-load-manual.md)
-2. [MySql Load](../import-way/mysql-load-manual.md)
+2. [MySQL Load](../import-way/mysql-load-manual.md)
## Stream Load
Stream Load 用于将本地文件导入到 Doris 中。
@@ -226,9 +226,9 @@ public class DorisStreamLoader {
></dependency>
> ```
-## MySql LOAD
+## MySQL Load
<version since="dev">
- MySql LOAD样例
+ MySQL Load样例
</version>
### 导入数据
@@ -248,7 +248,7 @@ public class DorisStreamLoader {
```
2. 导入数据
- 在MySql客户端下执行以下 SQL 命令导入本地文件：
+ 在MySQL客户端下执行以下 SQL 命令导入本地文件：
```sql
LOAD DATA
@@ -261,7 +261,7 @@ public class DorisStreamLoader {
3. 等待导入结果
- MySql Load 命令是同步命令，返回成功即表示导入成功。如果导入数据较大，可能需要较长的等待时间。示例如下:
+ MySQL Load 命令是同步命令，返回成功即表示导入成功。如果导入数据较大，可能需要较长的等待时间。示例如下:
```text
Query OK, 1 row affected (0.17 sec)
@@ -272,5 +272,5 @@ public class DorisStreamLoader {
- 其他字段的详细介绍，请参阅 [MySQL Load](../../../sql-manual/sql-reference/Data-Manipulation-Statements/Load/MYSQL-LOAD.md) 命令文档。
### 导入建议
-- MySql Load 只能导入本地文件(可以是客户端本地或者连接的FE节点本地), 而且支持CSV格式。
+- MySQL Load 只能导入本地文件(可以是客户端本地或者连接的FE节点本地), 而且支持CSV格式。
- 建议一个导入请求的数据量控制在 1 - 2 GB 以内。如果有大量本地文件，可以分批并发提交。
diff --git a/docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md b/docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md
index 7b6a66c90cb..fcc170023a6 100644
--- a/docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md
+++ b/docs/zh-CN/docs/data-operate/import/import-way/mysql-load-manual.md
@@ -1,6 +1,6 @@
---
{
- "title": "MySql Load",
+ "title": "MySQL Load",
"language": "zh-CN"
}
---
@@ -24,20 +24,20 @@ specific language governing permissions and limitations
under the License.
-->
-# MySql load
+# MySQL Load
<version since="dev">
该语句兼容MySQL标准的[LOAD DATA](https://dev.mysql.com/doc/refman/8.0/en/load-data.html)语法，方便用户导入本地数据，并降低学习成本。
-MySql load 同步执行导入并返回导入结果。用户可直接通过SQL返回信息判断本次导入是否成功。
+MySQL Load 同步执行导入并返回导入结果。用户可直接通过SQL返回信息判断本次导入是否成功。
-MySql load 主要适用于导入客户端本地文件，或通过程序导入数据流中的数据。
+MySQL Load 主要适用于导入客户端本地文件，或通过程序导入数据流中的数据。
</version>
## 基本原理
-MySql Load和Stream Load功能相似, 都是导入本地文件到Doris集群中, 因此MySQL Load实现复用了StreamLoad的基础导入能力:
+MySQL Load和Stream Load功能相似, 都是导入本地文件到Doris集群中, 因此MySQL Load实现复用了StreamLoad的基础导入能力:
1. FE接收到客户端执行的MySQL Load请求, 完成SQL解析工作
@@ -70,7 +70,7 @@ CREATE TABLE testdb.t1 (pk INT, v1 INT SUM) AGGREGATE KEY (pk) DISTRIBUTED BY ha
```
### 导入客户端文件
-假设在客户端本地当前路径上有一个CSV文件, 名为`client_local.csv`, 使用MySQL LOAD语法将表导入到测试表`testdb.t1`中.
+假设在客户端本地当前路径上有一个CSV文件, 名为`client_local.csv`, 使用MySQL Load语法将表导入到测试表`testdb.t1`中.
```sql
LOAD DATA LOCAL
@@ -115,7 +115,7 @@ PROPERTIES ("strict_mode"="true")
### 返回结果
-由于 MySQL load 是一种同步的导入方式，所以导入的结果会通过SQL语法返回给用户。
+由于 MySQL Load 是一种同步的导入方式，所以导入的结果会通过SQL语法返回给用户。
如果导入执行失败, 会展示具体的报错信息. 如果导入成功, 则会显示导入的行数.
```text
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 09/19: [Improment](executor)Create workload thread pool without cgroup #31170
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 69c7750db2d1627ce0e8253c30423998ac8cb50e
Author: wangbo <wa...@apache.org>
AuthorDate: Tue Feb 20 22:21:55 2024 +0800
[Improment](executor)Create workload thread pool without cgroup #31170
---
be/src/agent/workload_group_listener.cpp | 10 +----
be/src/runtime/task_group/task_group_manager.cpp | 50 +++++++++++++-----------
be/src/runtime/task_group/task_group_manager.h | 2 +-
docs/en/docs/admin-manual/workload-group.md | 10 ++---
docs/zh-CN/docs/admin-manual/workload-group.md | 12 +++---
5 files changed, 41 insertions(+), 43 deletions(-)
diff --git a/be/src/agent/workload_group_listener.cpp b/be/src/agent/workload_group_listener.cpp
index 237d6c77274..5f9da64bd2a 100644
--- a/be/src/agent/workload_group_listener.cpp
+++ b/be/src/agent/workload_group_listener.cpp
@@ -26,7 +26,6 @@ namespace doris {
void WorkloadGroupListener::handle_topic_info(const std::vector<TopicInfo>& topic_info_list) {
std::set<uint64_t> current_wg_ids;
- bool is_set_cgroup_path = config::doris_cgroup_cpu_path != "";
for (const TopicInfo& topic_info : topic_info_list) {
if (!topic_info.__isset.workload_group_info) {
continue;
@@ -51,12 +50,7 @@ void WorkloadGroupListener::handle_topic_info(const std::vector<TopicInfo>& topi
task_group_info.enable_cpu_hard_limit);
// 4 create and update task scheduler
- Status ret2 = _exec_env->task_group_manager()->upsert_cg_task_scheduler(&task_group_info,
- _exec_env);
- if (is_set_cgroup_path && !ret2.ok()) {
- LOG(INFO) << "upsert task sche failed, tg_id=" << task_group_info.id
- << ", reason=" << ret2.to_string();
- }
+ _exec_env->task_group_manager()->upsert_cg_task_scheduler(&task_group_info, _exec_env);
LOG(INFO) << "update task group finish, tg info=" << tg->debug_string()
<< ", enable_cpu_hard_limit="
@@ -65,7 +59,7 @@ void WorkloadGroupListener::handle_topic_info(const std::vector<TopicInfo>& topi
<< ", cgroup cpu_hard_limit=" << task_group_info.cgroup_cpu_hard_limit
<< ", enable_cgroup_cpu_soft_limit="
<< (config::enable_cgroup_cpu_soft_limit ? "true" : "false")
- << ", is set cgroup path=" << (is_set_cgroup_path ? "true" : "flase");
+ << ", cgroup home path=" << config::doris_cgroup_cpu_path;
}
_exec_env->task_group_manager()->delete_task_group_by_ids(current_wg_ids);
diff --git a/be/src/runtime/task_group/task_group_manager.cpp b/be/src/runtime/task_group/task_group_manager.cpp
index 18e446295cb..718d69021e7 100644
--- a/be/src/runtime/task_group/task_group_manager.cpp
+++ b/be/src/runtime/task_group/task_group_manager.cpp
@@ -89,8 +89,8 @@ void TaskGroupManager::get_query_scheduler(uint64_t tg_id,
}
}
-Status TaskGroupManager::upsert_cg_task_scheduler(taskgroup::TaskGroupInfo* tg_info,
- ExecEnv* exec_env) {
+void TaskGroupManager::upsert_cg_task_scheduler(taskgroup::TaskGroupInfo* tg_info,
+ ExecEnv* exec_env) {
uint64_t tg_id = tg_info->id;
std::string tg_name = tg_info->name;
int cpu_hard_limit = tg_info->cpu_hard_limit;
@@ -101,15 +101,17 @@ Status TaskGroupManager::upsert_cg_task_scheduler(taskgroup::TaskGroupInfo* tg_i
std::lock_guard<std::shared_mutex> write_lock(_task_scheduler_lock);
// step 1: init cgroup cpu controller
CgroupCpuCtl* cg_cu_ctl_ptr = nullptr;
- if (_cgroup_ctl_map.find(tg_id) == _cgroup_ctl_map.end()) {
+ if (config::doris_cgroup_cpu_path != "" &&
+ _cgroup_ctl_map.find(tg_id) == _cgroup_ctl_map.end()) {
std::unique_ptr<CgroupCpuCtl> cgroup_cpu_ctl = std::make_unique<CgroupV1CpuCtl>(tg_id);
Status ret = cgroup_cpu_ctl->init();
if (ret.ok()) {
cg_cu_ctl_ptr = cgroup_cpu_ctl.get();
_cgroup_ctl_map.emplace(tg_id, std::move(cgroup_cpu_ctl));
+ LOG(INFO) << "[upsert wg thread pool] cgroup init success";
} else {
- return Status::InternalError<false>("cgroup init failed, gid={}, reason={}", tg_id,
- ret.to_string());
+ LOG(INFO) << "[upsert wg thread pool] cgroup init failed, gid= " << tg_id
+ << ", reason=" << ret.to_string();
}
}
@@ -128,7 +130,7 @@ Status TaskGroupManager::upsert_cg_task_scheduler(taskgroup::TaskGroupInfo* tg_i
if (ret.ok()) {
_tg_sche_map.emplace(tg_id, std::move(pipeline_task_scheduler));
} else {
- return Status::InternalError<false>("task scheduler start failed, gid={}", tg_id);
+ LOG(INFO) << "[upsert wg thread pool] task scheduler start failed, gid= " << tg_id;
}
}
@@ -140,7 +142,7 @@ Status TaskGroupManager::upsert_cg_task_scheduler(taskgroup::TaskGroupInfo* tg_i
if (ret.ok()) {
_tg_scan_sche_map.emplace(tg_id, std::move(scan_scheduler));
} else {
- return Status::InternalError<false>("scan scheduler start failed, gid={}", tg_id);
+ LOG(INFO) << "[upsert wg thread pool] scan scheduler start failed, gid=" << tg_id;
}
}
if (scan_thread_num > 0 && _tg_scan_sche_map.find(tg_id) != _tg_scan_sche_map.end()) {
@@ -157,31 +159,33 @@ Status TaskGroupManager::upsert_cg_task_scheduler(taskgroup::TaskGroupInfo* tg_i
.set_cgroup_cpu_ctl(cg_cu_ctl_ptr)
.build(&thread_pool);
if (!ret.ok()) {
- LOG(INFO) << "create non-pipline thread pool failed";
+ LOG(INFO) << "[upsert wg thread pool] create non-pipline thread pool failed, gid="
+ << tg_id;
} else {
_non_pipe_thread_pool_map.emplace(tg_id, std::move(thread_pool));
}
}
// step 5: update cgroup cpu if needed
- if (enable_cpu_hard_limit) {
- if (cpu_hard_limit > 0) {
- _cgroup_ctl_map.at(tg_id)->update_cpu_hard_limit(cpu_hard_limit);
- _cgroup_ctl_map.at(tg_id)->update_cpu_soft_limit(CPU_SOFT_LIMIT_DEFAULT_VALUE);
+ if (_cgroup_ctl_map.find(tg_id) != _cgroup_ctl_map.end()) {
+ if (enable_cpu_hard_limit) {
+ if (cpu_hard_limit > 0) {
+ _cgroup_ctl_map.at(tg_id)->update_cpu_hard_limit(cpu_hard_limit);
+ _cgroup_ctl_map.at(tg_id)->update_cpu_soft_limit(CPU_SOFT_LIMIT_DEFAULT_VALUE);
+ } else {
+ LOG(INFO) << "[upsert wg thread pool] enable cpu hard limit but value is illegal: "
+ << cpu_hard_limit << ", gid=" << tg_id;
+ }
} else {
- return Status::InternalError<false>("enable cpu hard limit but value is illegal");
- }
- } else {
- if (config::enable_cgroup_cpu_soft_limit) {
- _cgroup_ctl_map.at(tg_id)->update_cpu_soft_limit(cpu_shares);
- _cgroup_ctl_map.at(tg_id)->update_cpu_hard_limit(
- CPU_HARD_LIMIT_DEFAULT_VALUE); // disable cpu hard limit
+ if (config::enable_cgroup_cpu_soft_limit) {
+ _cgroup_ctl_map.at(tg_id)->update_cpu_soft_limit(cpu_shares);
+ _cgroup_ctl_map.at(tg_id)->update_cpu_hard_limit(
+ CPU_HARD_LIMIT_DEFAULT_VALUE); // disable cpu hard limit
+ }
}
+ _cgroup_ctl_map.at(tg_id)->get_cgroup_cpu_info(&(tg_info->cgroup_cpu_shares),
+ &(tg_info->cgroup_cpu_hard_limit));
}
- _cgroup_ctl_map.at(tg_id)->get_cgroup_cpu_info(&(tg_info->cgroup_cpu_shares),
- &(tg_info->cgroup_cpu_hard_limit));
-
- return Status::OK();
}
void TaskGroupManager::delete_task_group_by_ids(std::set<uint64_t> used_wg_id) {
diff --git a/be/src/runtime/task_group/task_group_manager.h b/be/src/runtime/task_group/task_group_manager.h
index 1a1a614d068..29e5c30a596 100644
--- a/be/src/runtime/task_group/task_group_manager.h
+++ b/be/src/runtime/task_group/task_group_manager.h
@@ -51,7 +51,7 @@ public:
void get_related_taskgroups(const std::function<bool(const TaskGroupPtr& ptr)>& pred,
std::vector<TaskGroupPtr>* task_groups);
- Status upsert_cg_task_scheduler(taskgroup::TaskGroupInfo* tg_info, ExecEnv* exec_env);
+ void upsert_cg_task_scheduler(taskgroup::TaskGroupInfo* tg_info, ExecEnv* exec_env);
void delete_task_group_by_ids(std::set<uint64_t> id_set);
diff --git a/docs/en/docs/admin-manual/workload-group.md b/docs/en/docs/admin-manual/workload-group.md
index 93216a91fb9..01ebfdf156a 100644
--- a/docs/en/docs/admin-manual/workload-group.md
+++ b/docs/en/docs/admin-manual/workload-group.md
@@ -71,7 +71,9 @@ Notes:
Doris 2.0 version uses Doris scheduling to limit CPU resources, but since version 2.1, Doris defaults to using CGgroup v1 to limit CPU resources (CGgroup v2 is currently not supported). Therefore, if CPU resources are expected to be limited in version 2.1, it is necessary to have CGgroup v1 installed on the node where BE is located.
-If users use the Workload Group software limit in version 2.0 and upgrade to version 2.1, they also need to configure CGroup.
+If users use the Workload Group software limit in version 2.0 and upgrade to version 2.1, they also need to configure CGroup, Otherwise, cpu soft limit may not work.
+
+Without configuring cgroup, users can use all functions of the workload group except for CPU limitations.
1 Firstly, confirm that the CGgroup v1 version has been installed on the node where BE is located, and the path ```/sys/fs/cgroup/cpu/``` exists.
@@ -90,13 +92,11 @@ chonw -R doris:doris /sys/fs/cgroup/cpu/doris
4 Modify the configuration of BE and specify the path to cgroup
```
-1：modify be.conf in disk
doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris
-
-2 modify be conf in memory
-curl -X POST http://{be_ip}:{be_http_port}/api/update_config?doris_cgroup_cpu_path=/sys/fs/cgroup/cpu/doris
```
+5 restart BE, in the log (be. INFO), you can see the words "add thread xxx to group" indicating successful configuration.
+
It should be noted that the current workload group does not support the deployment of multiple BE on same machine.
## Workload group usage
diff --git a/docs/zh-CN/docs/admin-manual/workload-group.md b/docs/zh-CN/docs/admin-manual/workload-group.md
index d25fd26212e..2b36089fdeb 100644
--- a/docs/zh-CN/docs/admin-manual/workload-group.md
+++ b/docs/zh-CN/docs/admin-manual/workload-group.md
@@ -69,9 +69,11 @@ Workload Group是从2.0版本开始支持的功能，Workload Group在2.0版本
## 配置cgroup v1的环境
Doris的2.0版本使用基于Doris的调度实现CPU资源的限制，但是从2.1版本起，Doris默认使用基于CGroup v1版本对CPU资源进行限制（暂不支持CGroup v2），因此如果期望在2.1版本对CPU资源进行约束，那么需要BE所在的节点上已经安装好CGroup v1的环境。
-用户如果在2.0版本使用了Workload Group的软限并升级到了2.1版本，那么也需要配置CGroup。
+用户如果在2.0版本使用了Workload Group的软限并升级到了2.1版本，那么也需要配置CGroup，否则可能导致软限失效。
-1 首先确认BE所在节点已经安装好CGroup v1版本，确认存在路径```/sys/fs/cgroup/cpu/```即可。
+在不配置cgroup的情况下，用户可以使用workload group除CPU限制外的所有功能。
+
+1 首先确认BE所在节点已经安装好CGroup v1版本，确认存在路径```/sys/fs/cgroup/cpu/```即可
2 在cgroup的cpu路径下新建一个名为doris的目录，这个目录名用户可以自行指定
@@ -88,13 +90,11 @@ chonw -R doris:doris /sys/fs/cgroup/cpu/doris
4 修改BE的配置，指定cgroup的路径
```
-方法1：修改be.conf然后重启BE
doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris
-
-方法2：可以通过动态修改内存的方式
-curl -X POST http://{be_ip}:{be_http_port}/api/update_config?doris_cgroup_cpu_path=/sys/fs/cgroup/cpu/doris
```
+5 重启BE，在日志（be.INFO）可以看到"add thread xxx to group"的字样代表配置成功
+
需要注意的是，目前的workload group暂时不支持一个机器多个BE的部署方式。
## workload group使用
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 08/19: [fix](catalog recycle bin) Forbid recover partition if table schame is changed #31146
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1d6dc9a5f090ab26ad70111b981bc50519907fae
Author: meiyi <my...@gmail.com>
AuthorDate: Tue Feb 20 22:18:52 2024 +0800
[fix](catalog recycle bin) Forbid recover partition if table schame is changed #31146
---
.../apache/doris/catalog/CatalogRecycleBin.java | 11 +-
.../recover_with_schema_change.out | 39 +++++++
.../doris/regression/action/TestAction.groovy | 1 +
.../recover_with_schema_change.groovy | 127 +++++++++++++++++++++
4 files changed, 177 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
index 177d4641fc0..3ae63074454 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
@@ -810,8 +810,17 @@ public class CatalogRecycleBin extends MasterDaemon implements Writable {
throw new DdlException("Can not recover partition[" + partitionName + "]. Partition item conflict.");
}
- // check if partition name exists
+ // check if schema change
Partition recoverPartition = recoverPartitionInfo.getPartition();
+ Set<Long> tableIndex = table.getIndexIdToMeta().keySet();
+ Set<Long> partitionIndex = recoverPartition.getMaterializedIndices(IndexExtState.ALL).stream()
+ .map(i -> i.getId()).collect(Collectors.toSet());
+ if (!tableIndex.equals(partitionIndex)) {
+ throw new DdlException("table's index not equal with partition's index. table's index=" + tableIndex
+ + ", partition's index=" + partitionIndex);
+ }
+
+ // check if partition name exists
Preconditions.checkState(recoverPartition.getName().equalsIgnoreCase(partitionName));
if (!Strings.isNullOrEmpty(newPartitionName)) {
if (table.checkPartitionNameExist(newPartitionName)) {
diff --git a/regression-test/data/catalog_recycle_bin_p0/recover_with_schema_change.out b/regression-test/data/catalog_recycle_bin_p0/recover_with_schema_change.out
new file mode 100644
index 00000000000..11667bab84a
--- /dev/null
+++ b/regression-test/data/catalog_recycle_bin_p0/recover_with_schema_change.out
@@ -0,0 +1,39 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !sql --
+1 a \N 2022-01-02
+2 a \N 2023-01-02
+3 a \N 2024-01-02
+4 a 10 2022-01-02
+
+-- !sql --
+a
+a
+
+-- !sql --
+2 2023-01-02 a
+3 2024-01-02 a
+
+-- !sql --
+a
+a
+
+-- !sql --
+2 a 2023-01-02
+3 a 2024-01-02
+
+-- !sql --
+a
+a
+
+-- !sql --
+2 a 2023-01-02
+3 a 2024-01-02
+
+-- !sql --
+a
+a
+
+-- !sql --
+2 a 2023-01-02
+3 a 2024-01-02
+
diff --git a/regression-test/framework/src/main/groovy/org/apache/doris/regression/action/TestAction.groovy b/regression-test/framework/src/main/groovy/org/apache/doris/regression/action/TestAction.groovy
index c6b19bccd23..684cb9d5beb 100644
--- a/regression-test/framework/src/main/groovy/org/apache/doris/regression/action/TestAction.groovy
+++ b/regression-test/framework/src/main/groovy/org/apache/doris/regression/action/TestAction.groovy
@@ -68,6 +68,7 @@ class TestAction implements SuiteAction {
} else {
if (exception != null || result.exception != null) {
def msg = result.exception?.toString()
+ log.info("Exception: ${msg}")
Assert.assertTrue("Expect exception msg contains '${exception}', but meet '${msg}'",
msg != null && exception != null && msg.contains(exception))
}
diff --git a/regression-test/suites/catalog_recycle_bin_p0/recover_with_schema_change.groovy b/regression-test/suites/catalog_recycle_bin_p0/recover_with_schema_change.groovy
new file mode 100644
index 00000000000..a23c449ab95
--- /dev/null
+++ b/regression-test/suites/catalog_recycle_bin_p0/recover_with_schema_change.groovy
@@ -0,0 +1,127 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("recover_with_schema_change") {
+ def tableName = "recover_with_schema_change"
+
+ for (int i = 0; i <= 4; i++) {
+ def table = tableName + i
+
+ // create table and insert data
+ sql """ drop table if exists ${table} """
+ sql """
+ create table ${table} (
+ `id` int(11),
+ `name` varchar(128),
+ `da` date
+ )
+ engine=olap
+ duplicate key(id)
+ partition by range(da)(
+ PARTITION p3 VALUES LESS THAN ('2023-01-01'),
+ PARTITION p4 VALUES LESS THAN ('2024-01-01'),
+ PARTITION p5 VALUES LESS THAN ('2025-01-01')
+ )
+ distributed by hash(id) buckets 2
+ properties(
+ "replication_num"="1",
+ "light_schema_change"="true"
+ );
+ """
+
+ sql """ insert into ${table} values(1, 'a', '2022-01-02'); """
+ sql """ insert into ${table} values(2, 'a', '2023-01-02'); """
+ sql """ insert into ${table} values(3, 'a', '2024-01-02'); """
+
+ if (i == 3 || i == 4) {
+ // create mv
+ createMV """ create materialized view mv_${table} as select name from ${table}; """
+ }
+
+ // drop partition
+ sql """ ALTER TABLE ${table} DROP PARTITION p3; """
+
+ if (i == 0) {
+ // do light weight schema change
+ sql """ ALTER TABLE ${table} add column `age` int after name; """
+ waitForSchemaChangeDone {
+ sql """ SHOW ALTER TABLE COLUMN WHERE TableName='${table}' ORDER BY createtime DESC LIMIT 1 """
+ time 60
+ }
+
+ // recover partition should success
+ sql """ recover partition p3 from regression_test_catalog_recycle_bin_p0.${table}; """
+ } else if (i == 1) {
+ // do hard weight schema change
+ sql """ ALTER TABLE ${table} order by(`id`, `da`, `name`); """
+ waitForSchemaChangeDone {
+ sql """ SHOW ALTER TABLE COLUMN WHERE TableName='${table}' ORDER BY createtime DESC LIMIT 1 """
+ time 60
+ }
+
+ // recover partition should fail
+ test {
+ sql """ recover partition p3 from regression_test_catalog_recycle_bin_p0.${table}; """
+ exception "table's index not equal with partition's index"
+ }
+ } else if (i == 2) {
+ // create mv
+ createMV """ create materialized view mv_${table} as select name from ${table}; """
+
+ // recover partition should fail
+ test {
+ sql """ recover partition p3 from regression_test_catalog_recycle_bin_p0.${table}; """
+ exception "table's index not equal with partition's index"
+ }
+ } else if (i == 3) {
+ // drop mv
+ sql """ drop materialized view mv_${table} on ${table}; """
+
+ // recover partition should fail
+ test {
+ sql """ recover partition p3 from regression_test_catalog_recycle_bin_p0.${table}; """
+ exception "table's index not equal with partition's index"
+ }
+ } else if (i == 4) {
+ // drop mv
+ sql """ drop materialized view mv_${table} on ${table}; """
+
+ // create mv
+ createMV """ create materialized view mv_${table} as select name from ${table}; """
+
+ // recover partition should fail
+ test {
+ sql """ recover partition p3 from regression_test_catalog_recycle_bin_p0.${table}; """
+ exception "table's index not equal with partition's index"
+ }
+ }
+
+ // write data
+ if (i == 0) {
+ sql """ insert into ${table} values(4, 'a', 10, '2022-01-02'); """
+ } else {
+ test {
+ sql """ insert into ${table} values(4, 'b', '2022-01-02'); """
+ exception "Insert has filtered data in strict mode"
+ }
+ order_qt_sql """ select name from ${table}; """
+ }
+
+ // read data
+ order_qt_sql """ select * from ${table}; """
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 15/19: [Bug](json reader) object should stop processing when encounter error (#31159)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 278b232e76f215b214658b825f18c258a1768c98
Author: lihangyu <15...@163.com>
AuthorDate: Wed Feb 21 10:13:43 2024 +0800
[Bug](json reader) object should stop processing when encounter error (#31159)
If DATA_QUALITY_ERROR encountered we should stop processing this document any more.Otherwise there will be UB in simdjson.
---
be/src/exprs/json_functions.cpp | 2 +-
be/src/vec/exec/format/json/new_json_reader.cpp | 6 ++--
.../data/load_p0/stream_load/test_json_load.out | 4 +++
.../stream_load/test_malformed_json_with_path.json | 3 ++
.../load_p0/stream_load/test_json_load.groovy | 33 ++++++++++++++++++++--
5 files changed, 42 insertions(+), 6 deletions(-)
diff --git a/be/src/exprs/json_functions.cpp b/be/src/exprs/json_functions.cpp
index ff432c4655a..29c1596ed8f 100644
--- a/be/src/exprs/json_functions.cpp
+++ b/be/src/exprs/json_functions.cpp
@@ -261,7 +261,7 @@ Status JsonFunctions::extract_from_object(simdjson::ondemand::object& obj,
const std::string& _msg = msg; \
if (UNLIKELY(_err)) { \
if (_err == simdjson::NO_SUCH_FIELD || _err == simdjson::INDEX_OUT_OF_BOUNDS) { \
- return Status::DataQualityError( \
+ return Status::NotFound<false>( \
fmt::format("Not found target filed, err: {}, msg: {}", \
simdjson::error_message(_err), _msg)); \
} \
diff --git a/be/src/vec/exec/format/json/new_json_reader.cpp b/be/src/vec/exec/format/json/new_json_reader.cpp
index 97affdcd0bb..514a925cba4 100644
--- a/be/src/vec/exec/format/json/new_json_reader.cpp
+++ b/be/src/vec/exec/format/json/new_json_reader.cpp
@@ -1199,7 +1199,7 @@ Status NewJsonReader::_simdjson_handle_flat_array_complex_json_write_columns(
simdjson::ondemand::value val;
Status st = JsonFunctions::extract_from_object(cur, _parsed_json_root, &val);
if (UNLIKELY(!st.ok())) {
- if (st.is<DATA_QUALITY_ERROR>()) {
+ if (st.is<NOT_FOUND>()) {
RETURN_IF_ERROR(_append_error_msg(nullptr, st.to_string(), "", nullptr));
ADVANCE_ROW();
continue;
@@ -1630,11 +1630,11 @@ Status NewJsonReader::_simdjson_write_columns_by_jsonpath(
Status st;
if (i < _parsed_jsonpaths.size()) {
st = JsonFunctions::extract_from_object(*value, _parsed_jsonpaths[i], &json_value);
- if (!st.ok() && !st.is<DATA_QUALITY_ERROR>()) {
+ if (!st.ok() && !st.is<NOT_FOUND>()) {
return st;
}
}
- if (i >= _parsed_jsonpaths.size() || st.is<DATA_QUALITY_ERROR>()) {
+ if (i >= _parsed_jsonpaths.size() || st.is<NOT_FOUND>()) {
// not match in jsondata, filling with default value
RETURN_IF_ERROR(_fill_missing_column(slot_desc, column_ptr, valid));
if (!(*valid)) {
diff --git a/regression-test/data/load_p0/stream_load/test_json_load.out b/regression-test/data/load_p0/stream_load/test_json_load.out
index b9250608475..7351891633d 100644
--- a/regression-test/data/load_p0/stream_load/test_json_load.out
+++ b/regression-test/data/load_p0/stream_load/test_json_load.out
@@ -241,3 +241,7 @@ John 30 New York {"email":"john@example.com","phone":"+1-123-456-7890"}
100 2345676
200 755
+-- !select26 --
+android \N \N \N \N \N
+android \N \N \N \N \N
+
diff --git a/regression-test/data/load_p0/stream_load/test_malformed_json_with_path.json b/regression-test/data/load_p0/stream_load/test_malformed_json_with_path.json
new file mode 100644
index 00000000000..f87ebaa5d3b
--- /dev/null
+++ b/regression-test/data/load_p0/stream_load/test_malformed_json_with_path.json
@@ -0,0 +1,3 @@
+{"app_version":"v1.0.0","app_package":"com.fdf.listen","subject":"USER","ip":"45334","platform":"android","app_name":"听听","pro_brand":"图书","report_time":0,"user_id":"unknown","platform_ID":"1","action":"CLICK","event_name":"section_play","phone_num":"45645642692","pro_code":"unknown","event_value":"device_id":"gikj78675678","media_id":"67867","album_id":"1734","duration":"60","event_time":1706841911773,"object":"play_content"}
+{"app_version":"v1.0.0","app_package":"com.fdf.listen","subject":"USER","ip":"45334","platform":"android","app_name":"听听","pro_brand":"图书","report_time":0,"user_id":"unknown","platform_ID":"1","action":"CLICK","event_name":"section_play","phone_num":"45645642692","pro_code":"unknown","device_id":"gikj78675678","media_id":"67867","album_id":"1734","duration":"60","event_time":1706841911773,"object":"play_content"}
+{"app_version":"v1.0.0","app_package":"com.fdf.listen","subject":"USER","ip":"45334","platform":"android","app_name":"听听","pro_brand":"图书","report_time":0,"user_id":"unknown","platform_ID":"1","action":"CLICK","event_name":"section_play","phone_num":"45645642692","pro_code":"unknown","device_id":"gikj78675678","syscode":123, "media_id":"67867","album_id":"1734","duration":"60","event_time":1706841911773,"object":"play_content"}
\ No newline at end of file
diff --git a/regression-test/suites/load_p0/stream_load/test_json_load.groovy b/regression-test/suites/load_p0/stream_load/test_json_load.groovy
index 816765b1233..f41610f3ba2 100644
--- a/regression-test/suites/load_p0/stream_load/test_json_load.groovy
+++ b/regression-test/suites/load_p0/stream_load/test_json_load.groovy
@@ -727,6 +727,35 @@ suite("test_json_load", "p0") {
try_sql("DROP TABLE IF EXISTS ${testTable}")
}
+ // case27: import json with malformed json along with json path
+ try {
+ sql "DROP TABLE IF EXISTS ${testTable}"
+
+ sql """CREATE TABLE IF NOT EXISTS ${testTable}
+ (
+ `syscode` VARCHAR(20) NOT NULL COMMENT "",
+ `event_dt` DateTime NULL COMMENT "",
+ `pro_brand` VARCHAR(20) COMMENT "",
+ `app_package` VARCHAR(50) COMMENT "",
+ `platform` VARCHAR(20) COMMENT "",
+ `log_num` BIGINT DEFAULT "0" COMMENT ""
+ )
+ DUPLICATE KEY(`syscode`, `event_dt`,`pro_brand`,`app_package`,`platform`)
+ COMMENT ''
+ DISTRIBUTED BY RANDOM BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );"""
+
+ load_json_data.call("${testTable}", "${testTable}_case27_1", 'false', 'true', 'json', 'id= id * 10', '[\"$.platform\",\"$.app_package\",\"$.sysCode\",\"$.sys_code\",\"$.proBrand\",\"$.pro_brand\",\"$.event_time\"]',
+ '', '', '', 'test_malformed_json_with_path.json', false, 2)
+ sql "sync"
+ qt_select26 "select * from ${testTable}"
+
+ } finally {
+ try_sql("DROP TABLE IF EXISTS ${testTable}")
+ }
+
// test jsonpaths error
try {
sql "DROP TABLE IF EXISTS ${testTable}"
@@ -734,7 +763,7 @@ suite("test_json_load", "p0") {
create_json_test_table.call(testTable)
streamLoad {
table "${testTable}"
- set 'jsonpaths', '[\"Name\", \"Age\", \"Agent_id\"]'
+ set 'jsonpaths', '[\"$.Name\", \"$.Age\", \"$.Agent_id\"]'
set 'format', 'json'
file 'test_json_error.json' // import json file
time 10000 // limit inflight 10s
@@ -754,7 +783,7 @@ suite("test_json_load", "p0") {
def code = process.waitFor()
def out = process.text
log.info("result: ${out}".toString())
- def reason = "Reason: There is no column matching jsonpaths in the json file, columns:[name, age, agent_id, ], please check columns and jsonpaths:[\"Name\", \"Age\", \"Agent_id\"]. src line [{\"name\":\"Name1\",\"age\":21,\"agent_id\":\"1\"}]; \n"
+ def reason = "Reason: There is no column matching jsonpaths in the json file, columns:[name, age, agent_id, ], please check columns and jsonpaths:[\"\$.Name\", \"\$.Age\", \"\$.Agent_id\"]. src line [{\"name\":\"Name1\",\"age\":21,\"agent_id\":\"1\"}]; \n"
assertEquals("${reason}", "${out}")
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 07/19: [fix](backup) fix concurrent upload and release snapshot crash (#31144)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 872e1f6687cf8f8a0961b9a63687af469a78dfb6
Author: walter <w4...@gmail.com>
AuthorDate: Tue Feb 20 22:17:37 2024 +0800
[fix](backup) fix concurrent upload and release snapshot crash (#31144)
In upload implementation, filesystem::size throws an exception
if the specified file, which is removed by the release snapshot
task, does not exist.
---
be/src/io/fs/s3_file_system.cpp | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/be/src/io/fs/s3_file_system.cpp b/be/src/io/fs/s3_file_system.cpp
index 207bd9dff49..7b443ca6804 100644
--- a/be/src/io/fs/s3_file_system.cpp
+++ b/be/src/io/fs/s3_file_system.cpp
@@ -381,11 +381,10 @@ Status S3FileSystem::upload_impl(const Path& local_file, const Path& remote_file
local_file.native(), full_path(key)));
}
- auto file_size = std::filesystem::file_size(local_file);
+ auto size = handle->GetBytesTransferred();
LOG(INFO) << "Upload " << local_file.native() << " to s3, endpoint=" << _s3_conf.endpoint
<< ", bucket=" << _s3_conf.bucket << ", key=" << key
- << ", duration=" << duration.count() << ", capacity=" << file_size
- << ", tp=" << (file_size) / duration.count();
+ << ", duration=" << duration.count() << ", bytes=" << size;
return Status::OK();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 14/19: [pipelineX](refactor) refactor streaming agg structure (#31151)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 8b6d6d01657b020304599d08ac2cc2c5826506d8
Author: Gabriel <ga...@gmail.com>
AuthorDate: Wed Feb 21 10:01:52 2024 +0800
[pipelineX](refactor) refactor streaming agg structure (#31151)
---
be/src/pipeline/exec/aggregation_sink_operator.cpp | 172 +--
be/src/pipeline/exec/aggregation_sink_operator.h | 57 +-
.../pipeline/exec/aggregation_source_operator.cpp | 6 +-
be/src/pipeline/exec/aggregation_source_operator.h | 7 +-
be/src/pipeline/exec/data_queue.cpp | 14 +-
be/src/pipeline/exec/data_queue.h | 4 +-
.../distinct_streaming_aggregation_operator.cpp | 406 ++++++
.../exec/distinct_streaming_aggregation_operator.h | 128 ++
...istinct_streaming_aggregation_sink_operator.cpp | 151 ---
.../distinct_streaming_aggregation_sink_operator.h | 56 -
...tinct_streaming_aggregation_source_operator.cpp | 49 -
...istinct_streaming_aggregation_source_operator.h | 14 -
be/src/pipeline/exec/empty_source_operator.h | 2 +-
be/src/pipeline/exec/join_probe_operator.cpp | 6 +-
be/src/pipeline/exec/join_probe_operator.h | 2 +-
be/src/pipeline/exec/multi_cast_data_stream_sink.h | 2 +-
.../exec/multi_cast_data_stream_source.cpp | 2 +-
.../pipeline/exec/multi_cast_data_stream_source.h | 2 +-
.../exec/nested_loop_join_probe_operator.h | 2 +-
be/src/pipeline/exec/operator.cpp | 2 +-
be/src/pipeline/exec/operator.h | 8 +-
.../exec/streaming_aggregation_operator.cpp | 1302 ++++++++++++++++++++
.../pipeline/exec/streaming_aggregation_operator.h | 231 ++++
.../exec/streaming_aggregation_sink_operator.cpp | 298 -----
.../exec/streaming_aggregation_sink_operator.h | 61 -
.../exec/streaming_aggregation_source_operator.cpp | 39 -
.../exec/streaming_aggregation_source_operator.h | 13 -
be/src/pipeline/pipeline_x/dependency.h | 4 +-
.../local_exchange_source_operator.h | 2 +-
be/src/pipeline/pipeline_x/operator.cpp | 17 +-
be/src/pipeline/pipeline_x/operator.h | 2 +-
.../pipeline_x/pipeline_x_fragment_context.cpp | 39 +-
32 files changed, 2174 insertions(+), 926 deletions(-)
diff --git a/be/src/pipeline/exec/aggregation_sink_operator.cpp b/be/src/pipeline/exec/aggregation_sink_operator.cpp
index 4e6baf824ea..30c4d9d77a3 100644
--- a/be/src/pipeline/exec/aggregation_sink_operator.cpp
+++ b/be/src/pipeline/exec/aggregation_sink_operator.cpp
@@ -48,30 +48,25 @@ OPERATOR_CODE_GENERATOR(AggSinkOperator, StreamingOperator)
/// using the planner's estimated input cardinality and the assumption that input
/// is in a random order. This means that we assume that the reduction factor will
/// increase over time.
-template <typename DependencyType, typename Derived>
-AggSinkLocalState<DependencyType, Derived>::AggSinkLocalState(DataSinkOperatorXBase* parent,
- RuntimeState* state)
+AggSinkLocalState::AggSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
: Base(parent, state),
_hash_table_compute_timer(nullptr),
_hash_table_input_counter(nullptr),
_build_timer(nullptr),
_expr_timer(nullptr),
- _exec_timer(nullptr),
_serialize_key_timer(nullptr),
_merge_timer(nullptr),
_serialize_data_timer(nullptr),
_deserialize_data_timer(nullptr),
_max_row_size_counter(nullptr) {}
-template <typename DependencyType, typename Derived>
-Status AggSinkLocalState<DependencyType, Derived>::init(RuntimeState* state,
- LocalSinkStateInfo& info) {
+Status AggSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo& info) {
RETURN_IF_ERROR(Base::init(state, info));
SCOPED_TIMER(Base::exec_time_counter());
SCOPED_TIMER(Base::_open_timer);
_agg_data = Base::_shared_state->agg_data.get();
_agg_arena_pool = Base::_shared_state->agg_arena_pool.get();
- auto& p = Base::_parent->template cast<typename Derived::Parent>();
+ auto& p = Base::_parent->template cast<AggSinkOperatorX>();
Base::_shared_state->align_aggregate_states = p._align_aggregate_states;
Base::_shared_state->total_size_of_aggregate_states = p._total_size_of_aggregate_states;
Base::_shared_state->offsets_of_aggregate_states = p._offsets_of_aggregate_states;
@@ -80,9 +75,6 @@ Status AggSinkLocalState<DependencyType, Derived>::init(RuntimeState* state,
for (auto& evaluator : p._aggregate_evaluators) {
Base::_shared_state->aggregate_evaluators.push_back(evaluator->clone(state, p._pool));
}
- if (p._is_streaming) {
- Base::_shared_state->data_queue->set_sink_dependency(Base::_dependency, 0);
- }
Base::_shared_state->probe_expr_ctxs.resize(p._probe_expr_ctxs.size());
for (size_t i = 0; i < Base::_shared_state->probe_expr_ctxs.size(); i++) {
RETURN_IF_ERROR(
@@ -153,8 +145,7 @@ Status AggSinkLocalState<DependencyType, Derived>::init(RuntimeState* state,
return Status::OK();
}
-template <typename DependencyType, typename Derived>
-Status AggSinkLocalState<DependencyType, Derived>::open(RuntimeState* state) {
+Status AggSinkLocalState::open(RuntimeState* state) {
SCOPED_TIMER(Base::exec_time_counter());
SCOPED_TIMER(Base::_open_timer);
RETURN_IF_ERROR(Base::open(state));
@@ -169,9 +160,7 @@ Status AggSinkLocalState<DependencyType, Derived>::open(RuntimeState* state) {
return Status::OK();
}
-template <typename DependencyType, typename Derived>
-Status AggSinkLocalState<DependencyType, Derived>::_create_agg_status(
- vectorized::AggregateDataPtr data) {
+Status AggSinkLocalState::_create_agg_status(vectorized::AggregateDataPtr data) {
auto& shared_state = *Base::_shared_state;
for (int i = 0; i < shared_state.aggregate_evaluators.size(); ++i) {
try {
@@ -188,23 +177,20 @@ Status AggSinkLocalState<DependencyType, Derived>::_create_agg_status(
return Status::OK();
}
-template <typename DependencyType, typename Derived>
-Status AggSinkLocalState<DependencyType, Derived>::_execute_without_key(vectorized::Block* block) {
+Status AggSinkLocalState::_execute_without_key(vectorized::Block* block) {
DCHECK(_agg_data->without_key != nullptr);
SCOPED_TIMER(_build_timer);
for (int i = 0; i < Base::_shared_state->aggregate_evaluators.size(); ++i) {
RETURN_IF_ERROR(Base::_shared_state->aggregate_evaluators[i]->execute_single_add(
block,
- _agg_data->without_key + Base::_parent->template cast<typename Derived::Parent>()
+ _agg_data->without_key + Base::_parent->template cast<AggSinkOperatorX>()
._offsets_of_aggregate_states[i],
_agg_arena_pool));
}
return Status::OK();
}
-template <typename DependencyType, typename Derived>
-Status AggSinkLocalState<DependencyType, Derived>::_merge_with_serialized_key(
- vectorized::Block* block) {
+Status AggSinkLocalState::_merge_with_serialized_key(vectorized::Block* block) {
if (_reach_limit) {
return _merge_with_serialized_key_helper<true, false>(block);
} else {
@@ -212,8 +198,7 @@ Status AggSinkLocalState<DependencyType, Derived>::_merge_with_serialized_key(
}
}
-template <typename DependencyType, typename Derived>
-size_t AggSinkLocalState<DependencyType, Derived>::_memory_usage() const {
+size_t AggSinkLocalState::_memory_usage() const {
size_t usage = 0;
if (_agg_arena_pool) {
usage += _agg_arena_pool->size();
@@ -226,8 +211,7 @@ size_t AggSinkLocalState<DependencyType, Derived>::_memory_usage() const {
return usage;
}
-template <typename DependencyType, typename Derived>
-void AggSinkLocalState<DependencyType, Derived>::_update_memusage_with_serialized_key() {
+void AggSinkLocalState::_update_memusage_with_serialized_key() {
std::visit(
[&](auto&& agg_method) -> void {
auto& data = *agg_method.hash_table;
@@ -251,9 +235,7 @@ void AggSinkLocalState<DependencyType, Derived>::_update_memusage_with_serialize
_agg_data->method_variant);
}
-template <typename DependencyType, typename Derived>
-Status AggSinkLocalState<DependencyType, Derived>::_destroy_agg_status(
- vectorized::AggregateDataPtr data) {
+Status AggSinkLocalState::_destroy_agg_status(vectorized::AggregateDataPtr data) {
auto& shared_state = *Base::_shared_state;
for (int i = 0; i < shared_state.aggregate_evaluators.size(); ++i) {
shared_state.aggregate_evaluators[i]->function()->destroy(
@@ -262,10 +244,8 @@ Status AggSinkLocalState<DependencyType, Derived>::_destroy_agg_status(
return Status::OK();
}
-template <typename DependencyType, typename Derived>
template <bool limit, bool for_spill>
-Status AggSinkLocalState<DependencyType, Derived>::_merge_with_serialized_key_helper(
- vectorized::Block* block) {
+Status AggSinkLocalState::_merge_with_serialized_key_helper(vectorized::Block* block) {
SCOPED_TIMER(_merge_timer);
size_t key_size = Base::_shared_state->probe_expr_ctxs.size();
@@ -312,7 +292,7 @@ Status AggSinkLocalState<DependencyType, Derived>::_merge_with_serialized_key_he
->function()
->deserialize_and_merge_vec_selected(
_places.data(),
- Base::_parent->template cast<typename Derived::Parent>()
+ Base::_parent->template cast<AggSinkOperatorX>()
._offsets_of_aggregate_states[i],
_deserialize_buffer.data(),
(vectorized::ColumnString*)(column.get()), _agg_arena_pool,
@@ -322,7 +302,7 @@ Status AggSinkLocalState<DependencyType, Derived>::_merge_with_serialized_key_he
RETURN_IF_ERROR(
Base::_shared_state->aggregate_evaluators[i]->execute_batch_add_selected(
block,
- Base::_parent->template cast<typename Derived::Parent>()
+ Base::_parent->template cast<AggSinkOperatorX>()
._offsets_of_aggregate_states[i],
_places.data(), _agg_arena_pool));
}
@@ -356,7 +336,7 @@ Status AggSinkLocalState<DependencyType, Derived>::_merge_with_serialized_key_he
->function()
->deserialize_and_merge_vec(
_places.data(),
- Base::_parent->template cast<typename Derived::Parent>()
+ Base::_parent->template cast<AggSinkOperatorX>()
._offsets_of_aggregate_states[i],
_deserialize_buffer.data(),
(vectorized::ColumnString*)(column.get()), _agg_arena_pool,
@@ -365,7 +345,7 @@ Status AggSinkLocalState<DependencyType, Derived>::_merge_with_serialized_key_he
} else {
RETURN_IF_ERROR(Base::_shared_state->aggregate_evaluators[i]->execute_batch_add(
block,
- Base::_parent->template cast<typename Derived::Parent>()
+ Base::_parent->template cast<AggSinkOperatorX>()
._offsets_of_aggregate_states[i],
_places.data(), _agg_arena_pool));
}
@@ -373,7 +353,7 @@ Status AggSinkLocalState<DependencyType, Derived>::_merge_with_serialized_key_he
if (_should_limit_output) {
_reach_limit = _get_hash_table_size() >=
- Base::_parent->template cast<typename Derived::Parent>()._limit;
+ Base::_parent->template cast<AggSinkOperatorX>()._limit;
}
}
@@ -383,9 +363,7 @@ Status AggSinkLocalState<DependencyType, Derived>::_merge_with_serialized_key_he
// We should call this function only at 1st phase.
// 1st phase: is_merge=true, only have one SlotRef.
// 2nd phase: is_merge=false, maybe have multiple exprs.
-template <typename DependencyType, typename Derived>
-int AggSinkLocalState<DependencyType, Derived>::_get_slot_column_id(
- const vectorized::AggFnEvaluator* evaluator) {
+int AggSinkLocalState::_get_slot_column_id(const vectorized::AggFnEvaluator* evaluator) {
auto ctxs = evaluator->input_exprs_ctxs();
CHECK(ctxs.size() == 1 && ctxs[0]->root()->is_slot_ref())
<< "input_exprs_ctxs is invalid, input_exprs_ctx[0]="
@@ -393,8 +371,7 @@ int AggSinkLocalState<DependencyType, Derived>::_get_slot_column_id(
return ((vectorized::VSlotRef*)ctxs[0]->root().get())->column_id();
}
-template <typename DependencyType, typename Derived>
-Status AggSinkLocalState<DependencyType, Derived>::_merge_without_key(vectorized::Block* block) {
+Status AggSinkLocalState::_merge_without_key(vectorized::Block* block) {
SCOPED_TIMER(_merge_timer);
DCHECK(_agg_data->without_key != nullptr);
for (int i = 0; i < Base::_shared_state->aggregate_evaluators.size(); ++i) {
@@ -410,23 +387,21 @@ Status AggSinkLocalState<DependencyType, Derived>::_merge_without_key(vectorized
->function()
->deserialize_and_merge_from_column(
_agg_data->without_key +
- Base::_parent->template cast<typename Derived::Parent>()
+ Base::_parent->template cast<AggSinkOperatorX>()
._offsets_of_aggregate_states[i],
*column, _agg_arena_pool);
} else {
RETURN_IF_ERROR(Base::_shared_state->aggregate_evaluators[i]->execute_single_add(
block,
- _agg_data->without_key +
- Base::_parent->template cast<typename Derived::Parent>()
- ._offsets_of_aggregate_states[i],
+ _agg_data->without_key + Base::_parent->template cast<AggSinkOperatorX>()
+ ._offsets_of_aggregate_states[i],
_agg_arena_pool));
}
}
return Status::OK();
}
-template <typename DependencyType, typename Derived>
-void AggSinkLocalState<DependencyType, Derived>::_update_memusage_without_key() {
+void AggSinkLocalState::_update_memusage_without_key() {
auto arena_memory_usage =
_agg_arena_pool->size() - Base::_shared_state->mem_usage_record.used_in_arena;
Base::_mem_tracker->consume(arena_memory_usage);
@@ -434,9 +409,7 @@ void AggSinkLocalState<DependencyType, Derived>::_update_memusage_without_key()
Base::_shared_state->mem_usage_record.used_in_arena = _agg_arena_pool->size();
}
-template <typename DependencyType, typename Derived>
-Status AggSinkLocalState<DependencyType, Derived>::_execute_with_serialized_key(
- vectorized::Block* block) {
+Status AggSinkLocalState::_execute_with_serialized_key(vectorized::Block* block) {
if (_reach_limit) {
return _execute_with_serialized_key_helper<true>(block);
} else {
@@ -444,10 +417,8 @@ Status AggSinkLocalState<DependencyType, Derived>::_execute_with_serialized_key(
}
}
-template <typename DependencyType, typename Derived>
template <bool limit>
-Status AggSinkLocalState<DependencyType, Derived>::_execute_with_serialized_key_helper(
- vectorized::Block* block) {
+Status AggSinkLocalState::_execute_with_serialized_key_helper(vectorized::Block* block) {
SCOPED_TIMER(_build_timer);
DCHECK(!Base::_shared_state->probe_expr_ctxs.empty());
@@ -478,7 +449,7 @@ Status AggSinkLocalState<DependencyType, Derived>::_execute_with_serialized_key_
RETURN_IF_ERROR(
Base::_shared_state->aggregate_evaluators[i]->execute_batch_add_selected(
block,
- Base::_parent->template cast<typename Derived::Parent>()
+ Base::_parent->template cast<AggSinkOperatorX>()
._offsets_of_aggregate_states[i],
_places.data(), _agg_arena_pool));
}
@@ -488,16 +459,16 @@ Status AggSinkLocalState<DependencyType, Derived>::_execute_with_serialized_key_
for (int i = 0; i < Base::_shared_state->aggregate_evaluators.size(); ++i) {
RETURN_IF_ERROR(Base::_shared_state->aggregate_evaluators[i]->execute_batch_add(
block,
- Base::_parent->template cast<typename Derived::Parent>()
+ Base::_parent->template cast<AggSinkOperatorX>()
._offsets_of_aggregate_states[i],
_places.data(), _agg_arena_pool));
}
if (_should_limit_output) {
_reach_limit = _get_hash_table_size() >=
- Base::_parent->template cast<typename Derived::Parent>()._limit;
+ Base::_parent->template cast<AggSinkOperatorX>()._limit;
if (_reach_limit &&
- Base::_parent->template cast<typename Derived::Parent>()._can_short_circuit) {
+ Base::_parent->template cast<AggSinkOperatorX>()._can_short_circuit) {
Base::_dependency->set_ready_to_read();
return Status::Error<ErrorCode::END_OF_FILE>("");
}
@@ -507,16 +478,14 @@ Status AggSinkLocalState<DependencyType, Derived>::_execute_with_serialized_key_
return Status::OK();
}
-template <typename DependencyType, typename Derived>
-size_t AggSinkLocalState<DependencyType, Derived>::_get_hash_table_size() {
+size_t AggSinkLocalState::_get_hash_table_size() {
return std::visit([&](auto&& agg_method) { return agg_method.hash_table->size(); },
_agg_data->method_variant);
}
-template <typename DependencyType, typename Derived>
-void AggSinkLocalState<DependencyType, Derived>::_emplace_into_hash_table(
- vectorized::AggregateDataPtr* places, vectorized::ColumnRawPtrs& key_columns,
- const size_t num_rows) {
+void AggSinkLocalState::_emplace_into_hash_table(vectorized::AggregateDataPtr* places,
+ vectorized::ColumnRawPtrs& key_columns,
+ const size_t num_rows) {
std::visit(
[&](auto&& agg_method) -> void {
SCOPED_TIMER(_hash_table_compute_timer);
@@ -538,9 +507,9 @@ void AggSinkLocalState<DependencyType, Derived>::_emplace_into_hash_table(
auto creator_for_null_key = [&](auto& mapped) {
mapped = _agg_arena_pool->aligned_alloc(
- Base::_parent->template cast<typename Derived::Parent>()
+ Base::_parent->template cast<AggSinkOperatorX>()
._total_size_of_aggregate_states,
- Base::_parent->template cast<typename Derived::Parent>()
+ Base::_parent->template cast<AggSinkOperatorX>()
._align_aggregate_states);
auto st = _create_agg_status(mapped);
if (!st) {
@@ -558,10 +527,9 @@ void AggSinkLocalState<DependencyType, Derived>::_emplace_into_hash_table(
_agg_data->method_variant);
}
-template <typename DependencyType, typename Derived>
-void AggSinkLocalState<DependencyType, Derived>::_find_in_hash_table(
- vectorized::AggregateDataPtr* places, vectorized::ColumnRawPtrs& key_columns,
- size_t num_rows) {
+void AggSinkLocalState::_find_in_hash_table(vectorized::AggregateDataPtr* places,
+ vectorized::ColumnRawPtrs& key_columns,
+ size_t num_rows) {
std::visit(
[&](auto&& agg_method) -> void {
using HashMethodType = std::decay_t<decltype(agg_method)>;
@@ -583,9 +551,7 @@ void AggSinkLocalState<DependencyType, Derived>::_find_in_hash_table(
_agg_data->method_variant);
}
-template <typename DependencyType, typename Derived>
-void AggSinkLocalState<DependencyType, Derived>::_init_hash_method(
- const vectorized::VExprContextSPtrs& probe_exprs) {
+void AggSinkLocalState::_init_hash_method(const vectorized::VExprContextSPtrs& probe_exprs) {
DCHECK(probe_exprs.size() >= 1);
using Type = vectorized::AggregatedDataVariants::Type;
@@ -641,8 +607,7 @@ void AggSinkLocalState<DependencyType, Derived>::_init_hash_method(
_agg_data->init(
get_hash_key_type_with_phase(
- t,
- !Base::_parent->template cast<typename Derived::Parent>()._is_first_phase),
+ t, !Base::_parent->template cast<AggSinkOperatorX>()._is_first_phase),
is_nullable);
} else {
if (!try_get_hash_map_context_fixed<PHNormalHashMap, HashCRC32,
@@ -653,8 +618,7 @@ void AggSinkLocalState<DependencyType, Derived>::_init_hash_method(
}
}
-template <typename DependencyType, typename Derived>
-Status AggSinkLocalState<DependencyType, Derived>::_reset_hash_table() {
+Status AggSinkLocalState::_reset_hash_table() {
auto& ss = *Base::_shared_state;
return std::visit(
[&](auto&& agg_method) {
@@ -682,18 +646,15 @@ Status AggSinkLocalState<DependencyType, Derived>::_reset_hash_table() {
ss.agg_data->method_variant);
}
-template <typename DependencyType, typename Derived>
-Status AggSinkLocalState<DependencyType, Derived>::try_spill_disk(bool eos) {
- if (Base::_parent->template cast<typename Derived::Parent>()._external_agg_bytes_threshold ==
- 0) {
+Status AggSinkLocalState::try_spill_disk(bool eos) {
+ if (Base::_parent->template cast<AggSinkOperatorX>()._external_agg_bytes_threshold == 0) {
return Status::OK();
}
return std::visit(
[&](auto&& agg_method) -> Status {
auto& hash_table = *agg_method.hash_table;
- if (!eos &&
- _memory_usage() < Base::_parent->template cast<typename Derived::Parent>()
- ._external_agg_bytes_threshold) {
+ if (!eos && _memory_usage() < Base::_parent->template cast<AggSinkOperatorX>()
+ ._external_agg_bytes_threshold) {
return Status::OK();
}
@@ -707,11 +668,9 @@ Status AggSinkLocalState<DependencyType, Derived>::try_spill_disk(bool eos) {
_agg_data->method_variant);
}
-template <typename LocalStateType>
-AggSinkOperatorX<LocalStateType>::AggSinkOperatorX(ObjectPool* pool, int operator_id,
- const TPlanNode& tnode,
- const DescriptorTbl& descs, bool is_streaming)
- : DataSinkOperatorX<LocalStateType>(operator_id, tnode.node_id),
+AggSinkOperatorX::AggSinkOperatorX(ObjectPool* pool, int operator_id, const TPlanNode& tnode,
+ const DescriptorTbl& descs)
+ : DataSinkOperatorX<AggSinkLocalState>(operator_id, tnode.node_id),
_intermediate_tuple_id(tnode.agg_node.intermediate_tuple_id),
_intermediate_tuple_desc(nullptr),
_output_tuple_id(tnode.agg_node.output_tuple_id),
@@ -722,14 +681,12 @@ AggSinkOperatorX<LocalStateType>::AggSinkOperatorX(ObjectPool* pool, int operato
_pool(pool),
_limit(tnode.limit),
_have_conjuncts(tnode.__isset.vconjunct && !tnode.vconjunct.nodes.empty()),
- _is_streaming(is_streaming),
_partition_exprs(tnode.__isset.distribute_expr_lists ? tnode.distribute_expr_lists[0]
: std::vector<TExpr> {}),
_is_colocate(tnode.agg_node.__isset.is_colocate && tnode.agg_node.is_colocate) {}
-template <typename LocalStateType>
-Status AggSinkOperatorX<LocalStateType>::init(const TPlanNode& tnode, RuntimeState* state) {
- RETURN_IF_ERROR(DataSinkOperatorX<LocalStateType>::init(tnode, state));
+Status AggSinkOperatorX::init(const TPlanNode& tnode, RuntimeState* state) {
+ RETURN_IF_ERROR(DataSinkOperatorX<AggSinkLocalState>::init(tnode, state));
// ignore return status for now , so we need to introduce ExecNode::init()
RETURN_IF_ERROR(
vectorized::VExpr::create_expr_trees(tnode.agg_node.grouping_exprs, _probe_expr_ctxs));
@@ -767,13 +724,12 @@ Status AggSinkOperatorX<LocalStateType>::init(const TPlanNode& tnode, RuntimeSta
return Status::OK();
}
-template <typename LocalStateType>
-Status AggSinkOperatorX<LocalStateType>::prepare(RuntimeState* state) {
+Status AggSinkOperatorX::prepare(RuntimeState* state) {
_intermediate_tuple_desc = state->desc_tbl().get_tuple_descriptor(_intermediate_tuple_id);
_output_tuple_desc = state->desc_tbl().get_tuple_descriptor(_output_tuple_id);
DCHECK_EQ(_intermediate_tuple_desc->slots().size(), _output_tuple_desc->slots().size());
RETURN_IF_ERROR(vectorized::VExpr::prepare(
- _probe_expr_ctxs, state, DataSinkOperatorX<LocalStateType>::_child_x->row_desc()));
+ _probe_expr_ctxs, state, DataSinkOperatorX<AggSinkLocalState>::_child_x->row_desc()));
int j = _probe_expr_ctxs.size();
for (int i = 0; i < j; ++i) {
@@ -788,7 +744,7 @@ Status AggSinkOperatorX<LocalStateType>::prepare(RuntimeState* state) {
SlotDescriptor* intermediate_slot_desc = _intermediate_tuple_desc->slots()[j];
SlotDescriptor* output_slot_desc = _output_tuple_desc->slots()[j];
RETURN_IF_ERROR(_aggregate_evaluators[i]->prepare(
- state, DataSinkOperatorX<LocalStateType>::_child_x->row_desc(),
+ state, DataSinkOperatorX<AggSinkLocalState>::_child_x->row_desc(),
intermediate_slot_desc, output_slot_desc));
}
@@ -821,8 +777,7 @@ Status AggSinkOperatorX<LocalStateType>::prepare(RuntimeState* state) {
return Status::OK();
}
-template <typename LocalStateType>
-Status AggSinkOperatorX<LocalStateType>::open(RuntimeState* state) {
+Status AggSinkOperatorX::open(RuntimeState* state) {
RETURN_IF_ERROR(vectorized::VExpr::open(_probe_expr_ctxs, state));
for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
@@ -833,10 +788,8 @@ Status AggSinkOperatorX<LocalStateType>::open(RuntimeState* state) {
return Status::OK();
}
-template <typename LocalStateType>
-Status AggSinkOperatorX<LocalStateType>::sink(doris::RuntimeState* state,
- vectorized::Block* in_block,
- SourceState source_state) {
+Status AggSinkOperatorX::sink(doris::RuntimeState* state, vectorized::Block* in_block,
+ SourceState source_state) {
auto& local_state = get_local_state(state);
SCOPED_TIMER(local_state.exec_time_counter());
COUNTER_UPDATE(local_state.rows_input_counter(), (int64_t)in_block->rows());
@@ -856,8 +809,7 @@ Status AggSinkOperatorX<LocalStateType>::sink(doris::RuntimeState* state,
return Status::OK();
}
-template <typename DependencyType, typename Derived>
-Status AggSinkLocalState<DependencyType, Derived>::close(RuntimeState* state, Status exec_status) {
+Status AggSinkLocalState::close(RuntimeState* state, Status exec_status) {
SCOPED_TIMER(Base::exec_time_counter());
SCOPED_TIMER(Base::_close_timer);
if (Base::_closed) {
@@ -874,14 +826,4 @@ Status AggSinkLocalState<DependencyType, Derived>::close(RuntimeState* state, St
return Base::close(state, exec_status);
}
-class StreamingAggSinkLocalState;
-class DistinctStreamingAggSinkLocalState;
-
-template class AggSinkOperatorX<BlockingAggSinkLocalState>;
-template class AggSinkOperatorX<StreamingAggSinkLocalState>;
-template class AggSinkOperatorX<DistinctStreamingAggSinkLocalState>;
-template class AggSinkLocalState<AggSinkDependency, BlockingAggSinkLocalState>;
-template class AggSinkLocalState<StreamingAggSinkDependency, StreamingAggSinkLocalState>;
-template class AggSinkLocalState<DistinctStreamingAggSinkDependency,
- DistinctStreamingAggSinkLocalState>;
} // namespace doris::pipeline
diff --git a/be/src/pipeline/exec/aggregation_sink_operator.h b/be/src/pipeline/exec/aggregation_sink_operator.h
index d9014ec1325..2cbfe6c94e0 100644
--- a/be/src/pipeline/exec/aggregation_sink_operator.h
+++ b/be/src/pipeline/exec/aggregation_sink_operator.h
@@ -53,13 +53,13 @@ public:
~AggSinkDependency() override = default;
};
-template <typename LocalStateType>
class AggSinkOperatorX;
-template <typename DependencyType, typename Derived>
-class AggSinkLocalState : public PipelineXSinkLocalState<DependencyType> {
+class AggSinkLocalState : public PipelineXSinkLocalState<AggSinkDependency> {
public:
+ ENABLE_FACTORY_CREATOR(AggSinkLocalState);
using Base = PipelineXSinkLocalState<DependencyType>;
+ AggSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state);
~AggSinkLocalState() override = default;
Status init(RuntimeState* state, LocalSinkStateInfo& info) override;
@@ -69,21 +69,16 @@ public:
Status try_spill_disk(bool eos = false);
protected:
- AggSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state);
-
- template <typename LocalStateType>
friend class AggSinkOperatorX;
struct ExecutorBase {
- virtual Status execute(AggSinkLocalState<DependencyType, Derived>* local_state,
- vectorized::Block* block) = 0;
- virtual void update_memusage(AggSinkLocalState<DependencyType, Derived>* local_state) = 0;
+ virtual Status execute(AggSinkLocalState* local_state, vectorized::Block* block) = 0;
+ virtual void update_memusage(AggSinkLocalState* local_state) = 0;
virtual ~ExecutorBase() = default;
};
template <bool WithoutKey, bool NeedToMerge>
struct Executor final : public ExecutorBase {
- Status execute(AggSinkLocalState<DependencyType, Derived>* local_state,
- vectorized::Block* block) override {
+ Status execute(AggSinkLocalState* local_state, vectorized::Block* block) override {
if constexpr (WithoutKey) {
if constexpr (NeedToMerge) {
return local_state->_merge_without_key(block);
@@ -98,7 +93,7 @@ protected:
}
}
}
- void update_memusage(AggSinkLocalState<DependencyType, Derived>* local_state) override {
+ void update_memusage(AggSinkLocalState* local_state) override {
if constexpr (WithoutKey) {
local_state->_update_memusage_without_key();
} else {
@@ -309,7 +304,6 @@ protected:
RuntimeProfile::Counter* _hash_table_input_counter = nullptr;
RuntimeProfile::Counter* _build_timer = nullptr;
RuntimeProfile::Counter* _expr_timer = nullptr;
- RuntimeProfile::Counter* _exec_timer = nullptr;
RuntimeProfile::Counter* _build_table_convert_timer = nullptr;
RuntimeProfile::Counter* _serialize_key_timer = nullptr;
RuntimeProfile::Counter* _merge_timer = nullptr;
@@ -333,29 +327,14 @@ protected:
std::unique_ptr<ExecutorBase> _executor = nullptr;
};
-class BlockingAggSinkLocalState
- : public AggSinkLocalState<AggSinkDependency, BlockingAggSinkLocalState> {
-public:
- ENABLE_FACTORY_CREATOR(BlockingAggSinkLocalState);
- using Parent = AggSinkOperatorX<BlockingAggSinkLocalState>;
-
- BlockingAggSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state)
- : AggSinkLocalState<AggSinkDependency, BlockingAggSinkLocalState>(parent, state) {}
- ~BlockingAggSinkLocalState() override = default;
-};
-
-class StreamingAggSinkDependency;
-class StreamingAggSinkLocalState;
-
-template <typename LocalStateType = BlockingAggSinkLocalState>
-class AggSinkOperatorX : public DataSinkOperatorX<LocalStateType> {
+class AggSinkOperatorX final : public DataSinkOperatorX<AggSinkLocalState> {
public:
AggSinkOperatorX(ObjectPool* pool, int operator_id, const TPlanNode& tnode,
- const DescriptorTbl& descs, bool is_streaming = false);
+ const DescriptorTbl& descs);
~AggSinkOperatorX() override = default;
Status init(const TDataSink& tsink) override {
return Status::InternalError("{} should not init with TPlanNode",
- DataSinkOperatorX<LocalStateType>::_name);
+ DataSinkOperatorX<AggSinkLocalState>::_name);
}
Status init(const TPlanNode& tnode, RuntimeState* state) override;
@@ -368,25 +347,22 @@ public:
DataDistribution required_data_distribution() const override {
if (_probe_expr_ctxs.empty()) {
- return _needs_finalize || DataSinkOperatorX<LocalStateType>::_child_x
+ return _needs_finalize || DataSinkOperatorX<AggSinkLocalState>::_child_x
->ignore_data_distribution()
? DataDistribution(ExchangeType::PASSTHROUGH)
- : DataSinkOperatorX<LocalStateType>::required_data_distribution();
+ : DataSinkOperatorX<AggSinkLocalState>::required_data_distribution();
}
return _is_colocate ? DataDistribution(ExchangeType::BUCKET_HASH_SHUFFLE, _partition_exprs)
: DataDistribution(ExchangeType::HASH_SHUFFLE, _partition_exprs);
}
- using DataSinkOperatorX<LocalStateType>::id;
- using DataSinkOperatorX<LocalStateType>::operator_id;
- using DataSinkOperatorX<LocalStateType>::get_local_state;
+ using DataSinkOperatorX<AggSinkLocalState>::id;
+ using DataSinkOperatorX<AggSinkLocalState>::operator_id;
+ using DataSinkOperatorX<AggSinkLocalState>::get_local_state;
protected:
- using LocalState = LocalStateType;
- template <typename DependencyType, typename Derived>
+ using LocalState = AggSinkLocalState;
friend class AggSinkLocalState;
- friend class StreamingAggSinkLocalState;
- friend class DistinctStreamingAggSinkLocalState;
std::vector<vectorized::AggFnEvaluator*> _aggregate_evaluators;
bool _can_short_circuit = false;
@@ -415,7 +391,6 @@ protected:
size_t _spill_partition_count_bits;
int64_t _limit; // -1: no limit
bool _have_conjuncts;
- const bool _is_streaming;
const std::vector<TExpr> _partition_exprs;
const bool _is_colocate;
diff --git a/be/src/pipeline/exec/aggregation_source_operator.cpp b/be/src/pipeline/exec/aggregation_source_operator.cpp
index 7e2fca9bc67..5d27c2d33a2 100644
--- a/be/src/pipeline/exec/aggregation_source_operator.cpp
+++ b/be/src/pipeline/exec/aggregation_source_operator.cpp
@@ -50,9 +50,6 @@ Status AggLocalState::init(RuntimeState* state, LocalStateInfo& info) {
_serialize_data_timer = ADD_TIMER(profile(), "SerializeDataTime");
_hash_table_size_counter = ADD_COUNTER(profile(), "HashTableSize", TUnit::UNIT);
auto& p = _parent->template cast<AggSourceOperatorX>();
- if (p._is_streaming) {
- _shared_state->data_queue->set_source_dependency(info.dependency);
- }
if (p._without_key) {
if (p._needs_finalize) {
_executor.get_result = std::bind<Status>(&AggLocalState::_get_without_key_result, this,
@@ -529,9 +526,8 @@ Status AggLocalState::_get_without_key_result(RuntimeState* state, vectorized::B
}
AggSourceOperatorX::AggSourceOperatorX(ObjectPool* pool, const TPlanNode& tnode, int operator_id,
- const DescriptorTbl& descs, bool is_streaming)
+ const DescriptorTbl& descs)
: Base(pool, tnode, operator_id, descs),
- _is_streaming(is_streaming),
_needs_finalize(tnode.agg_node.need_finalize),
_without_key(tnode.agg_node.grouping_exprs.empty()) {}
diff --git a/be/src/pipeline/exec/aggregation_source_operator.h b/be/src/pipeline/exec/aggregation_source_operator.h
index 98fc72ecc97..d6aa19e57fd 100644
--- a/be/src/pipeline/exec/aggregation_source_operator.h
+++ b/be/src/pipeline/exec/aggregation_source_operator.h
@@ -72,10 +72,6 @@ public:
protected:
friend class AggSourceOperatorX;
- friend class StreamingAggSourceOperatorX;
- friend class StreamingAggSinkOperatorX;
- friend class DistinctStreamingAggSourceOperatorX;
- friend class DistinctStreamingAggSinkOperatorX;
Status _get_without_key_result(RuntimeState* state, vectorized::Block* block,
SourceState& source_state);
@@ -133,7 +129,7 @@ class AggSourceOperatorX : public OperatorX<AggLocalState> {
public:
using Base = OperatorX<AggLocalState>;
AggSourceOperatorX(ObjectPool* pool, const TPlanNode& tnode, int operator_id,
- const DescriptorTbl& descs, bool is_streaming = false);
+ const DescriptorTbl& descs);
~AggSourceOperatorX() = default;
Status get_block(RuntimeState* state, vectorized::Block* block,
@@ -143,7 +139,6 @@ public:
private:
friend class AggLocalState;
- const bool _is_streaming;
bool _needs_finalize;
bool _without_key;
diff --git a/be/src/pipeline/exec/data_queue.cpp b/be/src/pipeline/exec/data_queue.cpp
index d92a8907675..06c16c7dfa6 100644
--- a/be/src/pipeline/exec/data_queue.cpp
+++ b/be/src/pipeline/exec/data_queue.cpp
@@ -30,7 +30,7 @@
namespace doris {
namespace pipeline {
-DataQueue::DataQueue(int child_count, bool is_streaming_agg)
+DataQueue::DataQueue(int child_count)
: _queue_blocks_lock(child_count),
_queue_blocks(child_count),
_free_blocks_lock(child_count),
@@ -40,8 +40,7 @@ DataQueue::DataQueue(int child_count, bool is_streaming_agg)
_is_canceled(child_count),
_cur_bytes_in_queue(child_count),
_cur_blocks_nums_in_queue(child_count),
- _flag_queue_idx(0),
- _is_streaming_agg(is_streaming_agg) {
+ _flag_queue_idx(0) {
for (int i = 0; i < child_count; ++i) {
_queue_blocks_lock[i].reset(new std::mutex());
_free_blocks_lock[i].reset(new std::mutex());
@@ -123,11 +122,7 @@ Status DataQueue::get_block_from_queue(std::unique_ptr<vectorized::Block>* outpu
auto old_value = _cur_blocks_total_nums.fetch_sub(1);
if (old_value == 1 && _source_dependency) {
set_source_block();
- if (_is_streaming_agg && has_enough_space_to_push()) {
- _sink_dependencies[_flag_queue_idx]->set_ready();
- } else if (!_is_streaming_agg) {
- _sink_dependencies[_flag_queue_idx]->set_ready();
- }
+ _sink_dependencies[_flag_queue_idx]->set_ready();
}
} else {
if (_is_finished[_flag_queue_idx]) {
@@ -150,9 +145,6 @@ void DataQueue::push_block(std::unique_ptr<vectorized::Block> block, int child_i
_cur_blocks_total_nums++;
if (_source_dependency) {
set_source_ready();
- if (_is_streaming_agg && !has_enough_space_to_push()) {
- _sink_dependencies[child_idx]->block();
- }
}
//this only use to record the queue[0] for profile
_max_bytes_in_queue = std::max(_max_bytes_in_queue, _cur_bytes_in_queue[0].load());
diff --git a/be/src/pipeline/exec/data_queue.h b/be/src/pipeline/exec/data_queue.h
index 5fc58102ac6..144fe62728c 100644
--- a/be/src/pipeline/exec/data_queue.h
+++ b/be/src/pipeline/exec/data_queue.h
@@ -35,7 +35,7 @@ class Dependency;
class DataQueue {
public:
//always one is enough, but in union node it's has more children
- DataQueue(int child_count = 1, bool is_streaming_agg = true);
+ DataQueue(int child_count = 1);
~DataQueue() = default;
Status get_block_from_queue(std::unique_ptr<vectorized::Block>* block,
@@ -107,8 +107,6 @@ private:
std::shared_ptr<Dependency> _source_dependency = nullptr;
std::vector<Dependency*> _sink_dependencies;
SpinLock _source_lock;
-
- const bool _is_streaming_agg = true;
};
} // namespace doris::pipeline
diff --git a/be/src/pipeline/exec/distinct_streaming_aggregation_operator.cpp b/be/src/pipeline/exec/distinct_streaming_aggregation_operator.cpp
new file mode 100644
index 00000000000..36d5065087a
--- /dev/null
+++ b/be/src/pipeline/exec/distinct_streaming_aggregation_operator.cpp
@@ -0,0 +1,406 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "distinct_streaming_aggregation_operator.h"
+
+#include <gen_cpp/Metrics_types.h>
+
+#include <memory>
+#include <utility>
+
+#include "common/compiler_util.h" // IWYU pragma: keep
+
+namespace doris {
+class ExecNode;
+class RuntimeState;
+} // namespace doris
+
+namespace doris::pipeline {
+
+DistinctStreamingAggLocalState::DistinctStreamingAggLocalState(RuntimeState* state,
+ OperatorXBase* parent)
+ : PipelineXLocalState<FakeDependency>(state, parent),
+ dummy_mapped_data(std::make_shared<char>('A')),
+ _agg_arena_pool(std::make_unique<vectorized::Arena>()),
+ _agg_data(std::make_unique<vectorized::AggregatedDataVariants>()),
+ _agg_profile_arena(std::make_unique<vectorized::Arena>()),
+ _child_block(vectorized::Block::create_unique()),
+ _child_source_state(SourceState::DEPEND_ON_SOURCE),
+ _aggregated_block(vectorized::Block::create_unique()) {}
+
+Status DistinctStreamingAggLocalState::init(RuntimeState* state, LocalStateInfo& info) {
+ RETURN_IF_ERROR(Base::init(state, info));
+ SCOPED_TIMER(Base::exec_time_counter());
+ SCOPED_TIMER(Base::_open_timer);
+ auto& p = Base::_parent->template cast<DistinctStreamingAggOperatorX>();
+ for (auto& evaluator : p._aggregate_evaluators) {
+ _aggregate_evaluators.push_back(evaluator->clone(state, p._pool));
+ }
+ _probe_expr_ctxs.resize(p._probe_expr_ctxs.size());
+ for (size_t i = 0; i < _probe_expr_ctxs.size(); i++) {
+ RETURN_IF_ERROR(p._probe_expr_ctxs[i]->clone(state, _probe_expr_ctxs[i]));
+ }
+
+ _build_timer = ADD_TIMER(Base::profile(), "BuildTime");
+ _exec_timer = ADD_TIMER(Base::profile(), "ExecTime");
+ _hash_table_compute_timer = ADD_TIMER(Base::profile(), "HashTableComputeTime");
+ _hash_table_emplace_timer = ADD_TIMER(Base::profile(), "HashTableEmplaceTime");
+ _hash_table_input_counter = ADD_COUNTER(Base::profile(), "HashTableInputCount", TUnit::UNIT);
+
+ if (_probe_expr_ctxs.empty()) {
+ _agg_data->without_key = reinterpret_cast<vectorized::AggregateDataPtr>(
+ _agg_profile_arena->alloc(p._total_size_of_aggregate_states));
+ } else {
+ _init_hash_method(_probe_expr_ctxs);
+ }
+ return Status::OK();
+}
+
+void DistinctStreamingAggLocalState::_init_hash_method(
+ const vectorized::VExprContextSPtrs& probe_exprs) {
+ DCHECK(probe_exprs.size() >= 1);
+
+ using Type = vectorized::AggregatedDataVariants::Type;
+ Type t(Type::serialized);
+
+ if (probe_exprs.size() == 1) {
+ auto is_nullable = probe_exprs[0]->root()->is_nullable();
+ PrimitiveType type = probe_exprs[0]->root()->result_type();
+ switch (type) {
+ case TYPE_TINYINT:
+ case TYPE_BOOLEAN:
+ case TYPE_SMALLINT:
+ case TYPE_INT:
+ case TYPE_FLOAT:
+ case TYPE_DATEV2:
+ case TYPE_BIGINT:
+ case TYPE_DOUBLE:
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ case TYPE_DATETIMEV2:
+ case TYPE_LARGEINT:
+ case TYPE_DECIMALV2:
+ case TYPE_DECIMAL32:
+ case TYPE_DECIMAL64:
+ case TYPE_DECIMAL128I: {
+ size_t size = get_primitive_type_size(type);
+ if (size == 1) {
+ t = Type::int8_key;
+ } else if (size == 2) {
+ t = Type::int16_key;
+ } else if (size == 4) {
+ t = Type::int32_key;
+ } else if (size == 8) {
+ t = Type::int64_key;
+ } else if (size == 16) {
+ t = Type::int128_key;
+ } else {
+ throw Exception(ErrorCode::INTERNAL_ERROR,
+ "meet invalid type size, size={}, type={}", size,
+ type_to_string(type));
+ }
+ break;
+ }
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING: {
+ t = Type::string_key;
+ break;
+ }
+ default:
+ t = Type::serialized;
+ }
+
+ _agg_data->init(get_hash_key_type_with_phase(
+ t, !Base::_parent->template cast<DistinctStreamingAggOperatorX>()
+ ._is_first_phase),
+ is_nullable);
+ } else {
+ if (!try_get_hash_map_context_fixed<PHNormalHashMap, HashCRC32,
+ vectorized::AggregateDataPtr>(_agg_data->method_variant,
+ probe_exprs)) {
+ _agg_data->init(Type::serialized);
+ }
+ }
+}
+
+Status DistinctStreamingAggLocalState::_distinct_pre_agg_with_serialized_key(
+ doris::vectorized::Block* in_block, doris::vectorized::Block* out_block) {
+ SCOPED_TIMER(_build_timer);
+ DCHECK(!_probe_expr_ctxs.empty());
+
+ size_t key_size = _probe_expr_ctxs.size();
+ vectorized::ColumnRawPtrs key_columns(key_size);
+ {
+ SCOPED_TIMER(_expr_timer);
+ for (size_t i = 0; i < key_size; ++i) {
+ int result_column_id = -1;
+ RETURN_IF_ERROR(_probe_expr_ctxs[i]->execute(in_block, &result_column_id));
+ in_block->get_by_position(result_column_id).column =
+ in_block->get_by_position(result_column_id)
+ .column->convert_to_full_column_if_const();
+ key_columns[i] = in_block->get_by_position(result_column_id).column.get();
+ }
+ }
+
+ int rows = in_block->rows();
+ _distinct_row.clear();
+ _distinct_row.reserve(rows);
+
+ RETURN_IF_CATCH_EXCEPTION(
+ _emplace_into_hash_table_to_distinct(_distinct_row, key_columns, rows));
+
+ bool mem_reuse = _parent->cast<DistinctStreamingAggOperatorX>()._make_nullable_keys.empty() &&
+ out_block->mem_reuse();
+ if (mem_reuse) {
+ for (int i = 0; i < key_size; ++i) {
+ auto dst = out_block->get_by_position(i).column->assume_mutable();
+ key_columns[i]->append_data_by_selector(dst, _distinct_row);
+ }
+ } else {
+ vectorized::ColumnsWithTypeAndName columns_with_schema;
+ for (int i = 0; i < key_size; ++i) {
+ auto distinct_column = key_columns[i]->clone_empty();
+ key_columns[i]->append_data_by_selector(distinct_column, _distinct_row);
+ columns_with_schema.emplace_back(std::move(distinct_column),
+ _probe_expr_ctxs[i]->root()->data_type(),
+ _probe_expr_ctxs[i]->root()->expr_name());
+ }
+ out_block->swap(vectorized::Block(columns_with_schema));
+ }
+ return Status::OK();
+}
+
+void DistinctStreamingAggLocalState::_make_nullable_output_key(vectorized::Block* block) {
+ if (block->rows() != 0) {
+ for (auto cid : Base::_parent->cast<DistinctStreamingAggOperatorX>()._make_nullable_keys) {
+ block->get_by_position(cid).column = make_nullable(block->get_by_position(cid).column);
+ block->get_by_position(cid).type = make_nullable(block->get_by_position(cid).type);
+ }
+ }
+}
+
+void DistinctStreamingAggLocalState::_emplace_into_hash_table_to_distinct(
+ vectorized::IColumn::Selector& distinct_row, vectorized::ColumnRawPtrs& key_columns,
+ const size_t num_rows) {
+ std::visit(
+ [&](auto&& agg_method) -> void {
+ SCOPED_TIMER(_hash_table_compute_timer);
+ using HashMethodType = std::decay_t<decltype(agg_method)>;
+ using AggState = typename HashMethodType::State;
+ AggState state(key_columns);
+ agg_method.init_serialized_keys(key_columns, num_rows);
+ size_t row = 0;
+ auto creator = [&](const auto& ctor, auto& key, auto& origin) {
+ HashMethodType::try_presis_key(key, origin, _arena);
+ ctor(key, dummy_mapped_data.get());
+ distinct_row.push_back(row);
+ };
+ auto creator_for_null_key = [&](auto& mapped) {
+ mapped = dummy_mapped_data.get();
+ distinct_row.push_back(row);
+ };
+
+ SCOPED_TIMER(_hash_table_emplace_timer);
+ for (; row < num_rows; ++row) {
+ agg_method.lazy_emplace(state, row, creator, creator_for_null_key);
+ }
+
+ COUNTER_UPDATE(_hash_table_input_counter, num_rows);
+ },
+ _agg_data->method_variant);
+}
+
+DistinctStreamingAggOperatorX::DistinctStreamingAggOperatorX(ObjectPool* pool, int operator_id,
+ const TPlanNode& tnode,
+ const DescriptorTbl& descs)
+ : StatefulOperatorX<DistinctStreamingAggLocalState>(pool, tnode, operator_id, descs),
+ _intermediate_tuple_id(tnode.agg_node.intermediate_tuple_id),
+ _intermediate_tuple_desc(nullptr),
+ _output_tuple_id(tnode.agg_node.output_tuple_id),
+ _output_tuple_desc(nullptr),
+ _needs_finalize(tnode.agg_node.need_finalize),
+ _is_first_phase(tnode.agg_node.__isset.is_first_phase && tnode.agg_node.is_first_phase),
+ _partition_exprs(tnode.__isset.distribute_expr_lists ? tnode.distribute_expr_lists[0]
+ : std::vector<TExpr> {}),
+ _is_colocate(tnode.agg_node.__isset.is_colocate && tnode.agg_node.is_colocate) {
+ if (tnode.agg_node.__isset.use_streaming_preaggregation) {
+ _is_streaming_preagg = tnode.agg_node.use_streaming_preaggregation;
+ if (_is_streaming_preagg) {
+ DCHECK(!tnode.agg_node.grouping_exprs.empty()) << "Streaming preaggs do grouping";
+ DCHECK(_limit == -1) << "Preaggs have no limits";
+ }
+ } else {
+ _is_streaming_preagg = false;
+ }
+}
+
+Status DistinctStreamingAggOperatorX::init(const TPlanNode& tnode, RuntimeState* state) {
+ RETURN_IF_ERROR(StatefulOperatorX<DistinctStreamingAggLocalState>::init(tnode, state));
+ // ignore return status for now , so we need to introduce ExecNode::init()
+ RETURN_IF_ERROR(
+ vectorized::VExpr::create_expr_trees(tnode.agg_node.grouping_exprs, _probe_expr_ctxs));
+
+ // init aggregate functions
+ _aggregate_evaluators.reserve(tnode.agg_node.aggregate_functions.size());
+
+ TSortInfo dummy;
+ for (int i = 0; i < tnode.agg_node.aggregate_functions.size(); ++i) {
+ vectorized::AggFnEvaluator* evaluator = nullptr;
+ RETURN_IF_ERROR(vectorized::AggFnEvaluator::create(
+ _pool, tnode.agg_node.aggregate_functions[i],
+ tnode.agg_node.__isset.agg_sort_infos ? tnode.agg_node.agg_sort_infos[i] : dummy,
+ &evaluator));
+ _aggregate_evaluators.push_back(evaluator);
+ }
+
+ _op_name = "DISTINCT_STREAMING_AGGREGATION_OPERATOR";
+ return Status::OK();
+}
+
+Status DistinctStreamingAggOperatorX::prepare(RuntimeState* state) {
+ RETURN_IF_ERROR(StatefulOperatorX<DistinctStreamingAggLocalState>::prepare(state));
+ _intermediate_tuple_desc = state->desc_tbl().get_tuple_descriptor(_intermediate_tuple_id);
+ _output_tuple_desc = state->desc_tbl().get_tuple_descriptor(_output_tuple_id);
+ DCHECK_EQ(_intermediate_tuple_desc->slots().size(), _output_tuple_desc->slots().size());
+ RETURN_IF_ERROR(vectorized::VExpr::prepare(_probe_expr_ctxs, state, _child_x->row_desc()));
+
+ int j = _probe_expr_ctxs.size();
+ for (int i = 0; i < j; ++i) {
+ auto nullable_output = _output_tuple_desc->slots()[i]->is_nullable();
+ auto nullable_input = _probe_expr_ctxs[i]->root()->is_nullable();
+ if (nullable_output != nullable_input) {
+ DCHECK(nullable_output);
+ _make_nullable_keys.emplace_back(i);
+ }
+ }
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i, ++j) {
+ SlotDescriptor* intermediate_slot_desc = _intermediate_tuple_desc->slots()[j];
+ SlotDescriptor* output_slot_desc = _output_tuple_desc->slots()[j];
+ RETURN_IF_ERROR(_aggregate_evaluators[i]->prepare(
+ state, _child_x->row_desc(), intermediate_slot_desc, output_slot_desc));
+ }
+
+ for (size_t i = 0; i < _aggregate_evaluators.size(); ++i) {
+ const auto& agg_function = _aggregate_evaluators[i]->function();
+ _total_size_of_aggregate_states += agg_function->size_of_data();
+
+ // If not the last aggregate_state, we need pad it so that next aggregate_state will be aligned.
+ if (i + 1 < _aggregate_evaluators.size()) {
+ size_t alignment_of_next_state =
+ _aggregate_evaluators[i + 1]->function()->align_of_data();
+ if ((alignment_of_next_state & (alignment_of_next_state - 1)) != 0) {
+ return Status::RuntimeError("Logical error: align_of_data is not 2^N");
+ }
+
+ /// Extend total_size to next alignment requirement
+ /// Add padding by rounding up 'total_size_of_aggregate_states' to be a multiplier of alignment_of_next_state.
+ _total_size_of_aggregate_states =
+ (_total_size_of_aggregate_states + alignment_of_next_state - 1) /
+ alignment_of_next_state * alignment_of_next_state;
+ }
+ }
+
+ return Status::OK();
+}
+
+Status DistinctStreamingAggOperatorX::open(RuntimeState* state) {
+ RETURN_IF_ERROR(StatefulOperatorX<DistinctStreamingAggLocalState>::open(state));
+ RETURN_IF_ERROR(vectorized::VExpr::open(_probe_expr_ctxs, state));
+
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ RETURN_IF_ERROR(_aggregate_evaluators[i]->open(state));
+ _aggregate_evaluators[i]->set_version(state->be_exec_version());
+ }
+
+ return Status::OK();
+}
+
+Status DistinctStreamingAggOperatorX::push(RuntimeState* state, vectorized::Block* in_block,
+ SourceState source_state) const {
+ auto& local_state = get_local_state(state);
+ local_state._input_num_rows += in_block->rows();
+ Status ret = Status::OK();
+ if (in_block->rows() > 0) {
+ RETURN_IF_ERROR(local_state._distinct_pre_agg_with_serialized_key(
+ in_block, local_state._aggregated_block.get()));
+
+ // get enough data or reached limit rows, need push block to queue
+ if (_limit != -1 &&
+ (local_state._aggregated_block->rows() + local_state._output_distinct_rows) >= _limit) {
+ auto limit_rows = _limit - local_state._output_distinct_rows;
+ local_state._aggregated_block->set_num_rows(limit_rows);
+ local_state._output_distinct_rows += limit_rows;
+ } else if (local_state._aggregated_block->rows() >= state->batch_size()) {
+ local_state._output_distinct_rows += local_state._aggregated_block->rows();
+ }
+ }
+
+ // reach limit or source finish
+ if ((UNLIKELY(source_state == SourceState::FINISHED)) ||
+ (_limit != -1 && local_state._output_distinct_rows >= _limit)) {
+ local_state._output_distinct_rows += local_state._aggregated_block->rows();
+ return Status::OK(); // need given finish signal
+ }
+ return Status::OK();
+}
+
+Status DistinctStreamingAggOperatorX::pull(RuntimeState* state, vectorized::Block* block,
+ SourceState& source_state) const {
+ auto& local_state = get_local_state(state);
+ if (!local_state._aggregated_block->empty()) {
+ block->swap(*local_state._aggregated_block);
+ local_state._aggregated_block->clear_column_data(block->columns());
+ }
+
+ local_state._make_nullable_output_key(block);
+ if (_is_streaming_preagg == false) {
+ // dispose the having clause, should not be execute in prestreaming agg
+ RETURN_IF_ERROR(
+ vectorized::VExprContext::filter_block(_conjuncts, block, block->columns()));
+ }
+
+ if (UNLIKELY(local_state._child_source_state == SourceState::FINISHED ||
+ (_limit != -1 && local_state._output_distinct_rows >= _limit))) {
+ source_state = SourceState::FINISHED;
+ } else {
+ source_state = SourceState::DEPEND_ON_SOURCE;
+ }
+ return Status::OK();
+}
+
+bool DistinctStreamingAggOperatorX::need_more_input_data(RuntimeState* state) const {
+ auto& local_state = get_local_state(state);
+ return local_state._aggregated_block->empty() &&
+ local_state._child_source_state != SourceState::FINISHED &&
+ (_limit == -1 || local_state._output_distinct_rows < _limit);
+}
+
+Status DistinctStreamingAggLocalState::close(RuntimeState* state) {
+ if (_closed) {
+ return Status::OK();
+ }
+ SCOPED_TIMER(Base::exec_time_counter());
+ SCOPED_TIMER(Base::_close_timer);
+ if (Base::_closed) {
+ return Status::OK();
+ }
+ _aggregated_block->clear();
+ return Base::close(state);
+}
+
+} // namespace doris::pipeline
diff --git a/be/src/pipeline/exec/distinct_streaming_aggregation_operator.h b/be/src/pipeline/exec/distinct_streaming_aggregation_operator.h
new file mode 100644
index 00000000000..564c29b3e2a
--- /dev/null
+++ b/be/src/pipeline/exec/distinct_streaming_aggregation_operator.h
@@ -0,0 +1,128 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stdint.h>
+
+#include <cstdint>
+#include <memory>
+
+#include "common/status.h"
+#include "pipeline/pipeline_x/dependency.h"
+#include "pipeline/pipeline_x/operator.h"
+#include "util/runtime_profile.h"
+#include "vec/core/block.h"
+
+namespace doris {
+class ExecNode;
+class RuntimeState;
+
+namespace pipeline {
+
+class DistinctStreamingAggOperatorX;
+
+class DistinctStreamingAggLocalState final : public PipelineXLocalState<FakeDependency> {
+public:
+ using Parent = DistinctStreamingAggOperatorX;
+ using Base = PipelineXLocalState<FakeDependency>;
+ ENABLE_FACTORY_CREATOR(DistinctStreamingAggLocalState);
+ DistinctStreamingAggLocalState(RuntimeState* state, OperatorXBase* parent);
+
+ Status init(RuntimeState* state, LocalStateInfo& info) override;
+ Status close(RuntimeState* state) override;
+
+private:
+ friend class DistinctStreamingAggOperatorX;
+ template <typename LocalStateType>
+ friend class StatefulOperatorX;
+ Status _distinct_pre_agg_with_serialized_key(vectorized::Block* in_block,
+ vectorized::Block* out_block);
+ void _init_hash_method(const vectorized::VExprContextSPtrs& probe_exprs);
+ void _emplace_into_hash_table_to_distinct(vectorized::IColumn::Selector& distinct_row,
+ vectorized::ColumnRawPtrs& key_columns,
+ const size_t num_rows);
+ void _make_nullable_output_key(vectorized::Block* block);
+
+ std::shared_ptr<char> dummy_mapped_data;
+ vectorized::IColumn::Selector _distinct_row;
+ vectorized::Arena _arena;
+ int64_t _output_distinct_rows = 0;
+ size_t _input_num_rows = 0;
+
+ std::unique_ptr<vectorized::Arena> _agg_arena_pool = nullptr;
+ vectorized::AggregatedDataVariantsUPtr _agg_data = nullptr;
+ std::vector<vectorized::AggFnEvaluator*> _aggregate_evaluators;
+ // group by k1,k2
+ vectorized::VExprContextSPtrs _probe_expr_ctxs;
+ std::unique_ptr<vectorized::Arena> _agg_profile_arena = nullptr;
+ std::unique_ptr<vectorized::Block> _child_block = nullptr;
+ SourceState _child_source_state;
+ std::unique_ptr<vectorized::Block> _aggregated_block = nullptr;
+
+ RuntimeProfile::Counter* _build_timer = nullptr;
+ RuntimeProfile::Counter* _expr_timer = nullptr;
+ RuntimeProfile::Counter* _hash_table_compute_timer = nullptr;
+ RuntimeProfile::Counter* _hash_table_emplace_timer = nullptr;
+ RuntimeProfile::Counter* _hash_table_input_counter = nullptr;
+};
+
+class DistinctStreamingAggOperatorX final
+ : public StatefulOperatorX<DistinctStreamingAggLocalState> {
+public:
+ DistinctStreamingAggOperatorX(ObjectPool* pool, int operator_id, const TPlanNode& tnode,
+ const DescriptorTbl& descs);
+ Status init(const TPlanNode& tnode, RuntimeState* state) override;
+ Status prepare(RuntimeState* state) override;
+ Status open(RuntimeState* state) override;
+ Status pull(RuntimeState* state, vectorized::Block* block,
+ SourceState& source_state) const override;
+ Status push(RuntimeState* state, vectorized::Block* input_block,
+ SourceState source_state) const override;
+ bool need_more_input_data(RuntimeState* state) const override;
+
+ DataDistribution required_data_distribution() const override {
+ if (_needs_finalize) {
+ return _is_colocate
+ ? DataDistribution(ExchangeType::BUCKET_HASH_SHUFFLE, _partition_exprs)
+ : DataDistribution(ExchangeType::HASH_SHUFFLE, _partition_exprs);
+ }
+ return StatefulOperatorX<DistinctStreamingAggLocalState>::required_data_distribution();
+ }
+
+private:
+ friend class DistinctStreamingAggLocalState;
+ TupleId _intermediate_tuple_id;
+ TupleDescriptor* _intermediate_tuple_desc = nullptr;
+
+ TupleId _output_tuple_id;
+ TupleDescriptor* _output_tuple_desc = nullptr;
+ const bool _needs_finalize;
+ const bool _is_first_phase;
+ const std::vector<TExpr> _partition_exprs;
+ const bool _is_colocate;
+ // group by k1,k2
+ vectorized::VExprContextSPtrs _probe_expr_ctxs;
+ std::vector<vectorized::AggFnEvaluator*> _aggregate_evaluators;
+ std::vector<size_t> _make_nullable_keys;
+ /// The total size of the row from the aggregate functions.
+ size_t _total_size_of_aggregate_states = 0;
+ bool _is_streaming_preagg = false;
+};
+
+} // namespace pipeline
+} // namespace doris
diff --git a/be/src/pipeline/exec/distinct_streaming_aggregation_sink_operator.cpp b/be/src/pipeline/exec/distinct_streaming_aggregation_sink_operator.cpp
index 6efacc12a04..765055a004b 100644
--- a/be/src/pipeline/exec/distinct_streaming_aggregation_sink_operator.cpp
+++ b/be/src/pipeline/exec/distinct_streaming_aggregation_sink_operator.cpp
@@ -95,155 +95,4 @@ OperatorPtr DistinctStreamingAggSinkOperatorBuilder::build_operator() {
return std::make_shared<DistinctStreamingAggSinkOperator>(this, _node, _data_queue);
}
-DistinctStreamingAggSinkLocalState::DistinctStreamingAggSinkLocalState(
- DataSinkOperatorXBase* parent, RuntimeState* state)
- : AggSinkLocalState<DistinctStreamingAggSinkDependency, DistinctStreamingAggSinkLocalState>(
- parent, state),
- dummy_mapped_data(std::make_shared<char>('A')) {}
-
-Status DistinctStreamingAggSinkLocalState::_distinct_pre_agg_with_serialized_key(
- doris::vectorized::Block* in_block, doris::vectorized::Block* out_block) {
- SCOPED_TIMER(_build_timer);
- DCHECK(!_shared_state->probe_expr_ctxs.empty());
-
- size_t key_size = _shared_state->probe_expr_ctxs.size();
- vectorized::ColumnRawPtrs key_columns(key_size);
- {
- SCOPED_TIMER(_expr_timer);
- for (size_t i = 0; i < key_size; ++i) {
- int result_column_id = -1;
- RETURN_IF_ERROR(
- _shared_state->probe_expr_ctxs[i]->execute(in_block, &result_column_id));
- in_block->get_by_position(result_column_id).column =
- in_block->get_by_position(result_column_id)
- .column->convert_to_full_column_if_const();
- key_columns[i] = in_block->get_by_position(result_column_id).column.get();
- }
- }
-
- int rows = in_block->rows();
- _distinct_row.clear();
- _distinct_row.reserve(rows);
-
- RETURN_IF_CATCH_EXCEPTION(
- _emplace_into_hash_table_to_distinct(_distinct_row, key_columns, rows));
-
- bool mem_reuse = _shared_state->make_nullable_keys.empty() && out_block->mem_reuse();
- if (mem_reuse) {
- for (int i = 0; i < key_size; ++i) {
- auto dst = out_block->get_by_position(i).column->assume_mutable();
- key_columns[i]->append_data_by_selector(dst, _distinct_row);
- }
- } else {
- vectorized::ColumnsWithTypeAndName columns_with_schema;
- for (int i = 0; i < key_size; ++i) {
- auto distinct_column = key_columns[i]->clone_empty();
- key_columns[i]->append_data_by_selector(distinct_column, _distinct_row);
- columns_with_schema.emplace_back(
- std::move(distinct_column),
- _shared_state->probe_expr_ctxs[i]->root()->data_type(),
- _shared_state->probe_expr_ctxs[i]->root()->expr_name());
- }
- out_block->swap(vectorized::Block(columns_with_schema));
- }
- return Status::OK();
-}
-
-void DistinctStreamingAggSinkLocalState::_emplace_into_hash_table_to_distinct(
- vectorized::IColumn::Selector& distinct_row, vectorized::ColumnRawPtrs& key_columns,
- const size_t num_rows) {
- std::visit(
- [&](auto&& agg_method) -> void {
- SCOPED_TIMER(_hash_table_compute_timer);
- using HashMethodType = std::decay_t<decltype(agg_method)>;
- using AggState = typename HashMethodType::State;
- AggState state(key_columns);
- agg_method.init_serialized_keys(key_columns, num_rows);
- size_t row = 0;
- auto creator = [&](const auto& ctor, auto& key, auto& origin) {
- HashMethodType::try_presis_key(key, origin, _arena);
- ctor(key, dummy_mapped_data.get());
- distinct_row.push_back(row);
- };
- auto creator_for_null_key = [&](auto& mapped) {
- mapped = dummy_mapped_data.get();
- distinct_row.push_back(row);
- };
-
- SCOPED_TIMER(_hash_table_emplace_timer);
- for (; row < num_rows; ++row) {
- agg_method.lazy_emplace(state, row, creator, creator_for_null_key);
- }
-
- COUNTER_UPDATE(_hash_table_input_counter, num_rows);
- },
- _agg_data->method_variant);
-}
-
-DistinctStreamingAggSinkOperatorX::DistinctStreamingAggSinkOperatorX(ObjectPool* pool,
- int operator_id,
- const TPlanNode& tnode,
- const DescriptorTbl& descs)
- : AggSinkOperatorX<DistinctStreamingAggSinkLocalState>(pool, operator_id, tnode, descs,
- true) {}
-
-Status DistinctStreamingAggSinkOperatorX::init(const TPlanNode& tnode, RuntimeState* state) {
- RETURN_IF_ERROR(AggSinkOperatorX<DistinctStreamingAggSinkLocalState>::init(tnode, state));
- _name = "DISTINCT_STREAMING_AGGREGATION_SINK_OPERATOR";
- return Status::OK();
-}
-
-Status DistinctStreamingAggSinkOperatorX::sink(RuntimeState* state, vectorized::Block* in_block,
- SourceState source_state) {
- auto& local_state = get_local_state(state);
- SCOPED_TIMER(local_state.exec_time_counter());
- COUNTER_UPDATE(local_state.rows_input_counter(), (int64_t)in_block->rows());
- local_state._shared_state->input_num_rows += in_block->rows();
- Status ret = Status::OK();
- if (in_block && in_block->rows() > 0) {
- if (local_state._output_block == nullptr) {
- local_state._output_block = local_state._shared_state->data_queue->get_free_block();
- }
- RETURN_IF_ERROR(local_state._distinct_pre_agg_with_serialized_key(
- in_block, local_state._output_block.get()));
-
- // get enough data or reached limit rows, need push block to queue
- if (_limit != -1 &&
- (local_state._output_block->rows() + local_state._output_distinct_rows) >= _limit) {
- auto limit_rows = _limit - local_state._output_distinct_rows;
- local_state._output_block->set_num_rows(limit_rows);
- local_state._output_distinct_rows += limit_rows;
- local_state._shared_state->data_queue->push_block(std::move(local_state._output_block));
- } else if (local_state._output_block->rows() >= state->batch_size()) {
- local_state._output_distinct_rows += local_state._output_block->rows();
- local_state._shared_state->data_queue->push_block(std::move(local_state._output_block));
- }
- }
-
- // reach limit or source finish
- if ((UNLIKELY(source_state == SourceState::FINISHED)) ||
- (_limit != -1 && local_state._output_distinct_rows >= _limit)) {
- if (local_state._output_block != nullptr) { //maybe the last block with eos
- local_state._output_distinct_rows += local_state._output_block->rows();
- local_state._shared_state->data_queue->push_block(std::move(local_state._output_block));
- }
- local_state._shared_state->data_queue->set_finish();
- return Status::Error<ErrorCode::END_OF_FILE>(""); // need given finish signal
- }
- return Status::OK();
-}
-
-Status DistinctStreamingAggSinkLocalState::close(RuntimeState* state, Status exec_status) {
- if (_closed) {
- return Status::OK();
- }
- SCOPED_TIMER(exec_time_counter());
- SCOPED_TIMER(_close_timer);
- if (_shared_state->data_queue && !_shared_state->data_queue->is_finish()) {
- // finish should be set, if not set here means error.
- _shared_state->data_queue->set_canceled();
- }
- return Base::close(state, exec_status);
-}
-
} // namespace doris::pipeline
diff --git a/be/src/pipeline/exec/distinct_streaming_aggregation_sink_operator.h b/be/src/pipeline/exec/distinct_streaming_aggregation_sink_operator.h
index 13f48cad74a..c872a2b299e 100644
--- a/be/src/pipeline/exec/distinct_streaming_aggregation_sink_operator.h
+++ b/be/src/pipeline/exec/distinct_streaming_aggregation_sink_operator.h
@@ -75,61 +75,5 @@ private:
std::unique_ptr<vectorized::Block> _output_block = vectorized::Block::create_unique();
};
-class DistinctStreamingAggSinkOperatorX;
-
-class DistinctStreamingAggSinkDependency final : public Dependency {
-public:
- using SharedState = AggSharedState;
- DistinctStreamingAggSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "DistinctStreamingAggSinkDependency", true, query_ctx) {}
- ~DistinctStreamingAggSinkDependency() override = default;
-};
-
-class DistinctStreamingAggSinkLocalState final
- : public AggSinkLocalState<DistinctStreamingAggSinkDependency,
- DistinctStreamingAggSinkLocalState> {
-public:
- using Parent = DistinctStreamingAggSinkOperatorX;
- using Base = AggSinkLocalState<DistinctStreamingAggSinkDependency,
- DistinctStreamingAggSinkLocalState>;
- ENABLE_FACTORY_CREATOR(DistinctStreamingAggSinkLocalState);
- DistinctStreamingAggSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state);
-
- Status close(RuntimeState* state, Status exec_status) override;
- Status _distinct_pre_agg_with_serialized_key(vectorized::Block* in_block,
- vectorized::Block* out_block);
-
-private:
- friend class DistinctStreamingAggSinkOperatorX;
- void _emplace_into_hash_table_to_distinct(vectorized::IColumn::Selector& distinct_row,
- vectorized::ColumnRawPtrs& key_columns,
- const size_t num_rows);
-
- std::unique_ptr<vectorized::Block> _output_block = vectorized::Block::create_unique();
- std::shared_ptr<char> dummy_mapped_data;
- vectorized::IColumn::Selector _distinct_row;
- vectorized::Arena _arena;
- int64_t _output_distinct_rows = 0;
-};
-
-class DistinctStreamingAggSinkOperatorX final
- : public AggSinkOperatorX<DistinctStreamingAggSinkLocalState> {
-public:
- DistinctStreamingAggSinkOperatorX(ObjectPool* pool, int operator_id, const TPlanNode& tnode,
- const DescriptorTbl& descs);
- Status init(const TPlanNode& tnode, RuntimeState* state) override;
- Status sink(RuntimeState* state, vectorized::Block* in_block,
- SourceState source_state) override;
-
- DataDistribution required_data_distribution() const override {
- if (_needs_finalize) {
- return _is_colocate
- ? DataDistribution(ExchangeType::BUCKET_HASH_SHUFFLE, _partition_exprs)
- : DataDistribution(ExchangeType::HASH_SHUFFLE, _partition_exprs);
- }
- return DataSinkOperatorX<DistinctStreamingAggSinkLocalState>::required_data_distribution();
- }
-};
-
} // namespace pipeline
} // namespace doris
diff --git a/be/src/pipeline/exec/distinct_streaming_aggregation_source_operator.cpp b/be/src/pipeline/exec/distinct_streaming_aggregation_source_operator.cpp
index 722cc5b7956..fb653bdcbdb 100644
--- a/be/src/pipeline/exec/distinct_streaming_aggregation_source_operator.cpp
+++ b/be/src/pipeline/exec/distinct_streaming_aggregation_source_operator.cpp
@@ -88,54 +88,5 @@ OperatorPtr DistinctStreamingAggSourceOperatorBuilder::build_operator() {
return std::make_shared<DistinctStreamingAggSourceOperator>(this, _node, _data_queue);
}
-DistinctStreamingAggSourceOperatorX::DistinctStreamingAggSourceOperatorX(ObjectPool* pool,
- const TPlanNode& tnode,
- int operator_id,
- const DescriptorTbl& descs)
- : Base(pool, tnode, operator_id, descs, true) {
- if (tnode.agg_node.__isset.use_streaming_preaggregation) {
- _is_streaming_preagg = tnode.agg_node.use_streaming_preaggregation;
- if (_is_streaming_preagg) {
- DCHECK(!tnode.agg_node.grouping_exprs.empty()) << "Streaming preaggs do grouping";
- DCHECK(_limit == -1) << "Preaggs have no limits";
- }
- } else {
- _is_streaming_preagg = false;
- }
-}
-
-Status DistinctStreamingAggSourceOperatorX::get_block(RuntimeState* state, vectorized::Block* block,
- SourceState& source_state) {
- auto& local_state = get_local_state(state);
- SCOPED_TIMER(local_state.exec_time_counter());
- std::unique_ptr<vectorized::Block> agg_block;
- RETURN_IF_ERROR(local_state._shared_state->data_queue->get_block_from_queue(&agg_block));
- if (agg_block != nullptr) {
- block->swap(*agg_block);
- agg_block->clear_column_data(block->columns());
- local_state._shared_state->data_queue->push_free_block(std::move(agg_block));
- }
-
- local_state._make_nullable_output_key(block);
- if (_is_streaming_preagg == false) {
- // dispose the having clause, should not be execute in prestreaming agg
- RETURN_IF_ERROR(
- vectorized::VExprContext::filter_block(_conjuncts, block, block->columns()));
- }
-
- if (UNLIKELY(local_state._shared_state->data_queue->data_exhausted())) {
- source_state = SourceState::FINISHED;
- } else {
- source_state = SourceState::DEPEND_ON_SOURCE;
- }
- return Status::OK();
-}
-
-Status DistinctStreamingAggSourceOperatorX::init(const TPlanNode& tnode, RuntimeState* state) {
- RETURN_IF_ERROR(Base::init(tnode, state));
- _op_name = "DISTINCT_STREAMING_AGGREGATION_OPERATOR";
- return Status::OK();
-}
-
} // namespace pipeline
} // namespace doris
diff --git a/be/src/pipeline/exec/distinct_streaming_aggregation_source_operator.h b/be/src/pipeline/exec/distinct_streaming_aggregation_source_operator.h
index 81edf18567f..0283e31c41f 100644
--- a/be/src/pipeline/exec/distinct_streaming_aggregation_source_operator.h
+++ b/be/src/pipeline/exec/distinct_streaming_aggregation_source_operator.h
@@ -64,19 +64,5 @@ private:
std::shared_ptr<DataQueue> _data_queue;
};
-class DistinctStreamingAggSourceOperatorX final : public AggSourceOperatorX {
-public:
- using Base = AggSourceOperatorX;
- DistinctStreamingAggSourceOperatorX(ObjectPool* pool, const TPlanNode& tnode, int operator_id,
- const DescriptorTbl& descs);
- ~DistinctStreamingAggSourceOperatorX() = default;
-
- Status init(const TPlanNode& tnode, RuntimeState* state) override;
-
- Status get_block(RuntimeState* state, vectorized::Block* block,
- SourceState& source_state) override;
- bool _is_streaming_preagg = false;
-};
-
} // namespace pipeline
} // namespace doris
diff --git a/be/src/pipeline/exec/empty_source_operator.h b/be/src/pipeline/exec/empty_source_operator.h
index d76d3689253..b352f7cab29 100644
--- a/be/src/pipeline/exec/empty_source_operator.h
+++ b/be/src/pipeline/exec/empty_source_operator.h
@@ -46,7 +46,7 @@ public:
OperatorPtr build_operator() override;
- const RowDescriptor& row_desc() override { return _row_descriptor; }
+ const RowDescriptor& row_desc() const override { return _row_descriptor; }
private:
RowDescriptor _row_descriptor;
diff --git a/be/src/pipeline/exec/join_probe_operator.cpp b/be/src/pipeline/exec/join_probe_operator.cpp
index 3e71ec41dd4..c041fed5d7a 100644
--- a/be/src/pipeline/exec/join_probe_operator.cpp
+++ b/be/src/pipeline/exec/join_probe_operator.cpp
@@ -112,13 +112,15 @@ Status JoinProbeLocalState<DependencyType, Derived>::_build_output_block(
if (rows != 0) {
auto& mutable_columns = mutable_block.mutable_columns();
if (_output_expr_ctxs.empty()) {
- DCHECK(mutable_columns.size() == p.row_desc().num_materialized_slots());
+ DCHECK(mutable_columns.size() == p.row_desc().num_materialized_slots())
+ << mutable_columns.size() << " " << p.row_desc().num_materialized_slots();
for (int i = 0; i < mutable_columns.size(); ++i) {
insert_column_datas(mutable_columns[i], origin_block->get_by_position(i).column,
rows);
}
} else {
- DCHECK(mutable_columns.size() == p.row_desc().num_materialized_slots());
+ DCHECK(mutable_columns.size() == p.row_desc().num_materialized_slots())
+ << mutable_columns.size() << " " << p.row_desc().num_materialized_slots();
SCOPED_TIMER(Base::_projection_timer);
for (int i = 0; i < mutable_columns.size(); ++i) {
auto result_column_id = -1;
diff --git a/be/src/pipeline/exec/join_probe_operator.h b/be/src/pipeline/exec/join_probe_operator.h
index 5dc6089c348..6f379bc9749 100644
--- a/be/src/pipeline/exec/join_probe_operator.h
+++ b/be/src/pipeline/exec/join_probe_operator.h
@@ -73,7 +73,7 @@ public:
Status init(const TPlanNode& tnode, RuntimeState* state) override;
Status open(doris::RuntimeState* state) override;
- [[nodiscard]] const RowDescriptor& row_desc() override { return *_output_row_desc; }
+ [[nodiscard]] const RowDescriptor& row_desc() const override { return *_output_row_desc; }
[[nodiscard]] const RowDescriptor& intermediate_row_desc() const override {
return *_intermediate_row_desc;
diff --git a/be/src/pipeline/exec/multi_cast_data_stream_sink.h b/be/src/pipeline/exec/multi_cast_data_stream_sink.h
index 84a75720a66..7203ed0bf6f 100644
--- a/be/src/pipeline/exec/multi_cast_data_stream_sink.h
+++ b/be/src/pipeline/exec/multi_cast_data_stream_sink.h
@@ -99,7 +99,7 @@ public:
return Status::OK();
}
- RowDescriptor& row_desc() override { return _row_desc; }
+ const RowDescriptor& row_desc() const override { return _row_desc; }
std::shared_ptr<MultiCastSharedState> create_multi_cast_data_streamer() {
auto multi_cast_data_streamer =
diff --git a/be/src/pipeline/exec/multi_cast_data_stream_source.cpp b/be/src/pipeline/exec/multi_cast_data_stream_source.cpp
index 6ac06ee5f10..ccc0169c289 100644
--- a/be/src/pipeline/exec/multi_cast_data_stream_source.cpp
+++ b/be/src/pipeline/exec/multi_cast_data_stream_source.cpp
@@ -38,7 +38,7 @@ OperatorPtr MultiCastDataStreamerSourceOperatorBuilder::build_operator() {
this, _consumer_id, _multi_cast_data_streamer, _t_data_stream_sink);
}
-const RowDescriptor& MultiCastDataStreamerSourceOperatorBuilder::row_desc() {
+const RowDescriptor& MultiCastDataStreamerSourceOperatorBuilder::row_desc() const {
return _multi_cast_data_streamer->row_desc();
}
diff --git a/be/src/pipeline/exec/multi_cast_data_stream_source.h b/be/src/pipeline/exec/multi_cast_data_stream_source.h
index 5dae2be31f7..8eaa82520d1 100644
--- a/be/src/pipeline/exec/multi_cast_data_stream_source.h
+++ b/be/src/pipeline/exec/multi_cast_data_stream_source.h
@@ -48,7 +48,7 @@ public:
OperatorPtr build_operator() override;
- const RowDescriptor& row_desc() override;
+ const RowDescriptor& row_desc() const override;
private:
const int _consumer_id;
diff --git a/be/src/pipeline/exec/nested_loop_join_probe_operator.h b/be/src/pipeline/exec/nested_loop_join_probe_operator.h
index 1a3a0e9f9a1..35302434e70 100644
--- a/be/src/pipeline/exec/nested_loop_join_probe_operator.h
+++ b/be/src/pipeline/exec/nested_loop_join_probe_operator.h
@@ -234,7 +234,7 @@ public:
return {ExchangeType::ADAPTIVE_PASSTHROUGH};
}
- const RowDescriptor& row_desc() override {
+ const RowDescriptor& row_desc() const override {
return _old_version_flag
? (_output_row_descriptor ? *_output_row_descriptor : _row_descriptor)
: *_output_row_desc;
diff --git a/be/src/pipeline/exec/operator.cpp b/be/src/pipeline/exec/operator.cpp
index 11643bd4c36..4a01fa4578d 100644
--- a/be/src/pipeline/exec/operator.cpp
+++ b/be/src/pipeline/exec/operator.cpp
@@ -50,7 +50,7 @@ Status OperatorBase::close(RuntimeState* state) {
return Status::OK();
}
-const RowDescriptor& OperatorBase::row_desc() {
+const RowDescriptor& OperatorBase::row_desc() const {
return _operator_builder->row_desc();
}
diff --git a/be/src/pipeline/exec/operator.h b/be/src/pipeline/exec/operator.h
index 8755872a979..ffbe51d1b91 100644
--- a/be/src/pipeline/exec/operator.h
+++ b/be/src/pipeline/exec/operator.h
@@ -112,7 +112,7 @@ public:
std::string get_name() const { return _name; }
- virtual const RowDescriptor& row_desc() = 0;
+ virtual const RowDescriptor& row_desc() const = 0;
int32_t id() const { return _id; }
@@ -133,7 +133,7 @@ public:
~OperatorBuilder() override = default;
- const RowDescriptor& row_desc() override { return _node->row_desc(); }
+ const RowDescriptor& row_desc() const override { return _node->row_desc(); }
NodeType* exec_node() const { return _node; }
@@ -151,7 +151,7 @@ public:
bool is_sink() const override { return true; }
- const RowDescriptor& row_desc() override { return _sink->row_desc(); }
+ const RowDescriptor& row_desc() const override { return _sink->row_desc(); }
SinkType* exec_node() const { return _sink; }
@@ -243,7 +243,7 @@ public:
const OperatorBuilderBase* operator_builder() const { return _operator_builder; }
- virtual const RowDescriptor& row_desc();
+ virtual const RowDescriptor& row_desc() const;
virtual std::string debug_string() const;
virtual int32_t id() const { return _operator_builder->id(); }
diff --git a/be/src/pipeline/exec/streaming_aggregation_operator.cpp b/be/src/pipeline/exec/streaming_aggregation_operator.cpp
new file mode 100644
index 00000000000..f9ba600e3c0
--- /dev/null
+++ b/be/src/pipeline/exec/streaming_aggregation_operator.cpp
@@ -0,0 +1,1302 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "streaming_aggregation_operator.h"
+
+#include <gen_cpp/Metrics_types.h>
+
+#include <utility>
+
+#include "common/compiler_util.h" // IWYU pragma: keep
+#include "pipeline/exec/operator.h"
+
+namespace doris {
+class RuntimeState;
+} // namespace doris
+
+namespace doris::pipeline {
+/// The minimum reduction factor (input rows divided by output rows) to grow hash tables
+/// in a streaming preaggregation, given that the hash tables are currently the given
+/// size or above. The sizes roughly correspond to hash table sizes where the bucket
+/// arrays will fit in a cache level. Intuitively, we don't want the working set of the
+/// aggregation to expand to the next level of cache unless we're reducing the input
+/// enough to outweigh the increased memory latency we'll incur for each hash table
+/// lookup.
+///
+/// Note that the current reduction achieved is not always a good estimate of the
+/// final reduction. It may be biased either way depending on the ordering of the
+/// input. If the input order is random, we will underestimate the final reduction
+/// factor because the probability of a row having the same key as a previous row
+/// increases as more input is processed. If the input order is correlated with the
+/// key, skew may bias the estimate. If high cardinality keys appear first, we
+/// may overestimate and if low cardinality keys appear first, we underestimate.
+/// To estimate the eventual reduction achieved, we estimate the final reduction
+/// using the planner's estimated input cardinality and the assumption that input
+/// is in a random order. This means that we assume that the reduction factor will
+/// increase over time.
+struct StreamingHtMinReductionEntry {
+ // Use 'streaming_ht_min_reduction' if the total size of hash table bucket directories in
+ // bytes is greater than this threshold.
+ int min_ht_mem;
+ // The minimum reduction factor to expand the hash tables.
+ double streaming_ht_min_reduction;
+};
+
+// TODO: experimentally tune these values and also programmatically get the cache size
+// of the machine that we're running on.
+static constexpr StreamingHtMinReductionEntry STREAMING_HT_MIN_REDUCTION[] = {
+ // Expand up to L2 cache always.
+ {0, 0.0},
+ // Expand into L3 cache if we look like we're getting some reduction.
+ // At present, The L2 cache is generally 1024k or more
+ {1024 * 1024, 1.1},
+ // Expand into main memory if we're getting a significant reduction.
+ // The L3 cache is generally 16MB or more
+ {16 * 1024 * 1024, 2.0},
+};
+
+static constexpr int STREAMING_HT_MIN_REDUCTION_SIZE =
+ sizeof(STREAMING_HT_MIN_REDUCTION) / sizeof(STREAMING_HT_MIN_REDUCTION[0]);
+
+StreamingAggLocalState::StreamingAggLocalState(RuntimeState* state, OperatorXBase* parent)
+ : Base(state, parent),
+ _agg_arena_pool(std::make_unique<vectorized::Arena>()),
+ _agg_data(std::make_unique<vectorized::AggregatedDataVariants>()),
+ _agg_profile_arena(std::make_unique<vectorized::Arena>()),
+ _child_block(vectorized::Block::create_unique()),
+ _child_source_state(SourceState::DEPEND_ON_SOURCE),
+ _pre_aggregated_block(vectorized::Block::create_unique()) {}
+
+Status StreamingAggLocalState::init(RuntimeState* state, LocalStateInfo& info) {
+ RETURN_IF_ERROR(Base::init(state, info));
+ SCOPED_TIMER(Base::exec_time_counter());
+ SCOPED_TIMER(Base::_open_timer);
+ auto& p = Base::_parent->template cast<StreamingAggOperatorX>();
+ for (auto& evaluator : p._aggregate_evaluators) {
+ _aggregate_evaluators.push_back(evaluator->clone(state, p._pool));
+ }
+ _probe_expr_ctxs.resize(p._probe_expr_ctxs.size());
+ for (size_t i = 0; i < _probe_expr_ctxs.size(); i++) {
+ RETURN_IF_ERROR(p._probe_expr_ctxs[i]->clone(state, _probe_expr_ctxs[i]));
+ }
+ _hash_table_memory_usage = ADD_CHILD_COUNTER_WITH_LEVEL(Base::profile(), "HashTable",
+ TUnit::BYTES, "MemoryUsage", 1);
+ _serialize_key_arena_memory_usage = Base::profile()->AddHighWaterMarkCounter(
+ "SerializeKeyArena", TUnit::BYTES, "MemoryUsage", 1);
+
+ _build_timer = ADD_TIMER(Base::profile(), "BuildTime");
+ _build_table_convert_timer = ADD_TIMER(Base::profile(), "BuildConvertToPartitionedTime");
+ _serialize_key_timer = ADD_TIMER(Base::profile(), "SerializeKeyTime");
+ _exec_timer = ADD_TIMER(Base::profile(), "ExecTime");
+ _merge_timer = ADD_TIMER(Base::profile(), "MergeTime");
+ _expr_timer = ADD_TIMER(Base::profile(), "ExprTime");
+ _serialize_data_timer = ADD_TIMER(Base::profile(), "SerializeDataTime");
+ _deserialize_data_timer = ADD_TIMER(Base::profile(), "DeserializeAndMergeTime");
+ _hash_table_compute_timer = ADD_TIMER(Base::profile(), "HashTableComputeTime");
+ _hash_table_emplace_timer = ADD_TIMER(Base::profile(), "HashTableEmplaceTime");
+ _hash_table_input_counter = ADD_COUNTER(Base::profile(), "HashTableInputCount", TUnit::UNIT);
+ _max_row_size_counter = ADD_COUNTER(Base::profile(), "MaxRowSizeInBytes", TUnit::UNIT);
+ _hash_table_size_counter = ADD_COUNTER(profile(), "HashTableSize", TUnit::UNIT);
+ _queue_byte_size_counter = ADD_COUNTER(profile(), "MaxSizeInBlockQueue", TUnit::BYTES);
+ _queue_size_counter = ADD_COUNTER(profile(), "MaxSizeOfBlockQueue", TUnit::UNIT);
+ _streaming_agg_timer = ADD_TIMER(profile(), "StreamingAggTime");
+ _build_timer = ADD_TIMER(profile(), "BuildTime");
+ _expr_timer = ADD_TIMER(Base::profile(), "ExprTime");
+ _get_results_timer = ADD_TIMER(profile(), "GetResultsTime");
+ _serialize_result_timer = ADD_TIMER(profile(), "SerializeResultTime");
+ _hash_table_iterate_timer = ADD_TIMER(profile(), "HashTableIterateTime");
+ _insert_keys_to_column_timer = ADD_TIMER(profile(), "InsertKeysToColumnTime");
+ COUNTER_SET(_max_row_size_counter, (int64_t)0);
+
+ for (auto& evaluator : _aggregate_evaluators) {
+ evaluator->set_timer(_merge_timer, _expr_timer);
+ }
+
+ if (_probe_expr_ctxs.empty()) {
+ _agg_data->without_key = reinterpret_cast<vectorized::AggregateDataPtr>(
+ _agg_profile_arena->alloc(p._total_size_of_aggregate_states));
+
+ if (p._is_merge) {
+ if (p._needs_finalize) {
+ _executor = std::make_unique<Executor<true, true, true>>();
+ } else {
+ _executor = std::make_unique<Executor<true, true, false>>();
+ }
+ } else {
+ if (p._needs_finalize) {
+ _executor = std::make_unique<Executor<true, false, true>>();
+ } else {
+ _executor = std::make_unique<Executor<true, false, false>>();
+ }
+ }
+ } else {
+ _init_hash_method(_probe_expr_ctxs);
+
+ std::visit(
+ [&](auto&& agg_method) {
+ using HashTableType = std::decay_t<decltype(agg_method)>;
+ using KeyType = typename HashTableType::Key;
+
+ /// some aggregate functions (like AVG for decimal) have align issues.
+ _aggregate_data_container.reset(new vectorized::AggregateDataContainer(
+ sizeof(KeyType),
+ ((p._total_size_of_aggregate_states + p._align_aggregate_states - 1) /
+ p._align_aggregate_states) *
+ p._align_aggregate_states));
+ },
+ _agg_data->method_variant);
+ if (p._is_merge) {
+ if (p._needs_finalize) {
+ _executor = std::make_unique<Executor<false, true, true>>();
+ } else {
+ _executor = std::make_unique<Executor<false, true, false>>();
+ }
+ } else {
+ if (p._needs_finalize) {
+ _executor = std::make_unique<Executor<false, false, true>>();
+ } else {
+ _executor = std::make_unique<Executor<false, false, false>>();
+ }
+ }
+
+ _should_limit_output = p._limit != -1 && // has limit
+ (!p._have_conjuncts) && // no having conjunct
+ p._needs_finalize; // agg's finalize step
+ }
+ return Status::OK();
+}
+
+void StreamingAggLocalState::_make_nullable_output_key(vectorized::Block* block) {
+ if (block->rows() != 0) {
+ for (auto cid : Base::_parent->cast<StreamingAggOperatorX>()._make_nullable_keys) {
+ block->get_by_position(cid).column = make_nullable(block->get_by_position(cid).column);
+ block->get_by_position(cid).type = make_nullable(block->get_by_position(cid).type);
+ }
+ }
+}
+
+Status StreamingAggLocalState::_execute_without_key(vectorized::Block* block) {
+ DCHECK(_agg_data->without_key != nullptr);
+ SCOPED_TIMER(_build_timer);
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ RETURN_IF_ERROR(_aggregate_evaluators[i]->execute_single_add(
+ block,
+ _agg_data->without_key + Base::_parent->template cast<StreamingAggOperatorX>()
+ ._offsets_of_aggregate_states[i],
+ _agg_arena_pool.get()));
+ }
+ return Status::OK();
+}
+
+template <bool limit, bool for_spill>
+Status StreamingAggLocalState::_merge_with_serialized_key_helper(vectorized::Block* block) {
+ SCOPED_TIMER(_merge_timer);
+
+ size_t key_size = _probe_expr_ctxs.size();
+ vectorized::ColumnRawPtrs key_columns(key_size);
+
+ for (size_t i = 0; i < key_size; ++i) {
+ if constexpr (for_spill) {
+ key_columns[i] = block->get_by_position(i).column.get();
+ } else {
+ int result_column_id = -1;
+ RETURN_IF_ERROR(_probe_expr_ctxs[i]->execute(block, &result_column_id));
+ block->replace_by_position_if_const(result_column_id);
+ key_columns[i] = block->get_by_position(result_column_id).column.get();
+ }
+ }
+
+ int rows = block->rows();
+ if (_places.size() < rows) {
+ _places.resize(rows);
+ }
+
+ if constexpr (limit) {
+ _find_in_hash_table(_places.data(), key_columns, rows);
+
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ if (_aggregate_evaluators[i]->is_merge()) {
+ int col_id = _get_slot_column_id(_aggregate_evaluators[i]);
+ auto column = block->get_by_position(col_id).column;
+ if (column->is_nullable()) {
+ column = ((vectorized::ColumnNullable*)column.get())->get_nested_column_ptr();
+ }
+
+ size_t buffer_size = _aggregate_evaluators[i]->function()->size_of_data() * rows;
+ if (_deserialize_buffer.size() < buffer_size) {
+ _deserialize_buffer.resize(buffer_size);
+ }
+
+ {
+ SCOPED_TIMER(_deserialize_data_timer);
+ _aggregate_evaluators[i]->function()->deserialize_and_merge_vec_selected(
+ _places.data(),
+ Base::_parent->template cast<StreamingAggOperatorX>()
+ ._offsets_of_aggregate_states[i],
+ _deserialize_buffer.data(), (vectorized::ColumnString*)(column.get()),
+ _agg_arena_pool.get(), rows);
+ }
+ } else {
+ RETURN_IF_ERROR(_aggregate_evaluators[i]->execute_batch_add_selected(
+ block,
+ Base::_parent->template cast<StreamingAggOperatorX>()
+ ._offsets_of_aggregate_states[i],
+ _places.data(), _agg_arena_pool.get()));
+ }
+ }
+ } else {
+ _emplace_into_hash_table(_places.data(), key_columns, rows);
+
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ if (_aggregate_evaluators[i]->is_merge() || for_spill) {
+ int col_id = 0;
+ if constexpr (for_spill) {
+ col_id = _probe_expr_ctxs.size() + i;
+ } else {
+ col_id = _get_slot_column_id(_aggregate_evaluators[i]);
+ }
+ auto column = block->get_by_position(col_id).column;
+ if (column->is_nullable()) {
+ column = ((vectorized::ColumnNullable*)column.get())->get_nested_column_ptr();
+ }
+
+ size_t buffer_size = _aggregate_evaluators[i]->function()->size_of_data() * rows;
+ if (_deserialize_buffer.size() < buffer_size) {
+ _deserialize_buffer.resize(buffer_size);
+ }
+
+ {
+ SCOPED_TIMER(_deserialize_data_timer);
+ _aggregate_evaluators[i]->function()->deserialize_and_merge_vec(
+ _places.data(),
+ Base::_parent->template cast<StreamingAggOperatorX>()
+ ._offsets_of_aggregate_states[i],
+ _deserialize_buffer.data(), (vectorized::ColumnString*)(column.get()),
+ _agg_arena_pool.get(), rows);
+ }
+ } else {
+ RETURN_IF_ERROR(_aggregate_evaluators[i]->execute_batch_add(
+ block,
+ Base::_parent->template cast<StreamingAggOperatorX>()
+ ._offsets_of_aggregate_states[i],
+ _places.data(), _agg_arena_pool.get()));
+ }
+ }
+
+ if (_should_limit_output) {
+ _reach_limit = _get_hash_table_size() >=
+ Base::_parent->template cast<StreamingAggOperatorX>()._limit;
+ }
+ }
+
+ return Status::OK();
+}
+
+size_t StreamingAggLocalState::_get_hash_table_size() {
+ return std::visit([&](auto&& agg_method) { return agg_method.hash_table->size(); },
+ _agg_data->method_variant);
+}
+
+Status StreamingAggLocalState::_merge_without_key(vectorized::Block* block) {
+ SCOPED_TIMER(_merge_timer);
+ DCHECK(_agg_data->without_key != nullptr);
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ if (_aggregate_evaluators[i]->is_merge()) {
+ int col_id = _get_slot_column_id(_aggregate_evaluators[i]);
+ auto column = block->get_by_position(col_id).column;
+ if (column->is_nullable()) {
+ column = ((vectorized::ColumnNullable*)column.get())->get_nested_column_ptr();
+ }
+
+ SCOPED_TIMER(_deserialize_data_timer);
+ _aggregate_evaluators[i]->function()->deserialize_and_merge_from_column(
+ _agg_data->without_key + Base::_parent->template cast<StreamingAggOperatorX>()
+ ._offsets_of_aggregate_states[i],
+ *column, _agg_arena_pool.get());
+ } else {
+ RETURN_IF_ERROR(_aggregate_evaluators[i]->execute_single_add(
+ block,
+ _agg_data->without_key + Base::_parent->template cast<StreamingAggOperatorX>()
+ ._offsets_of_aggregate_states[i],
+ _agg_arena_pool.get()));
+ }
+ }
+ return Status::OK();
+}
+
+void StreamingAggLocalState::_update_memusage_without_key() {
+ auto arena_memory_usage = _agg_arena_pool->size() - _mem_usage_record.used_in_arena;
+ Base::_mem_tracker->consume(arena_memory_usage);
+ _serialize_key_arena_memory_usage->add(arena_memory_usage);
+ _mem_usage_record.used_in_arena = _agg_arena_pool->size();
+}
+
+Status StreamingAggLocalState::_execute_with_serialized_key(vectorized::Block* block) {
+ if (_reach_limit) {
+ return _execute_with_serialized_key_helper<true>(block);
+ } else {
+ return _execute_with_serialized_key_helper<false>(block);
+ }
+}
+
+void StreamingAggLocalState::_update_memusage_with_serialized_key() {
+ std::visit(
+ [&](auto&& agg_method) -> void {
+ auto& data = *agg_method.hash_table;
+ auto arena_memory_usage = _agg_arena_pool->size() +
+ _aggregate_data_container->memory_usage() -
+ _mem_usage_record.used_in_arena;
+ Base::_mem_tracker->consume(arena_memory_usage);
+ Base::_mem_tracker->consume(data.get_buffer_size_in_bytes() -
+ _mem_usage_record.used_in_state);
+ _serialize_key_arena_memory_usage->add(arena_memory_usage);
+ COUNTER_UPDATE(_hash_table_memory_usage,
+ data.get_buffer_size_in_bytes() - _mem_usage_record.used_in_state);
+ _mem_usage_record.used_in_state = data.get_buffer_size_in_bytes();
+ _mem_usage_record.used_in_arena =
+ _agg_arena_pool->size() + _aggregate_data_container->memory_usage();
+ },
+ _agg_data->method_variant);
+}
+
+template <bool limit>
+Status StreamingAggLocalState::_execute_with_serialized_key_helper(vectorized::Block* block) {
+ SCOPED_TIMER(_build_timer);
+ DCHECK(!_probe_expr_ctxs.empty());
+
+ size_t key_size = _probe_expr_ctxs.size();
+ vectorized::ColumnRawPtrs key_columns(key_size);
+ {
+ SCOPED_TIMER(_expr_timer);
+ for (size_t i = 0; i < key_size; ++i) {
+ int result_column_id = -1;
+ RETURN_IF_ERROR(_probe_expr_ctxs[i]->execute(block, &result_column_id));
+ block->get_by_position(result_column_id).column =
+ block->get_by_position(result_column_id)
+ .column->convert_to_full_column_if_const();
+ key_columns[i] = block->get_by_position(result_column_id).column.get();
+ }
+ }
+
+ int rows = block->rows();
+ if (_places.size() < rows) {
+ _places.resize(rows);
+ }
+
+ if constexpr (limit) {
+ _find_in_hash_table(_places.data(), key_columns, rows);
+
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ RETURN_IF_ERROR(_aggregate_evaluators[i]->execute_batch_add_selected(
+ block,
+ Base::_parent->template cast<StreamingAggOperatorX>()
+ ._offsets_of_aggregate_states[i],
+ _places.data(), _agg_arena_pool.get()));
+ }
+ } else {
+ _emplace_into_hash_table(_places.data(), key_columns, rows);
+
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ RETURN_IF_ERROR(_aggregate_evaluators[i]->execute_batch_add(
+ block,
+ Base::_parent->template cast<StreamingAggOperatorX>()
+ ._offsets_of_aggregate_states[i],
+ _places.data(), _agg_arena_pool.get()));
+ }
+
+ if (_should_limit_output) {
+ _reach_limit = _get_hash_table_size() >=
+ Base::_parent->template cast<StreamingAggOperatorX>()._limit;
+ if (_reach_limit &&
+ Base::_parent->template cast<StreamingAggOperatorX>()._can_short_circuit) {
+ Base::_dependency->set_ready_to_read();
+ return Status::Error<ErrorCode::END_OF_FILE>("");
+ }
+ }
+ }
+
+ return Status::OK();
+}
+
+// We should call this function only at 1st phase.
+// 1st phase: is_merge=true, only have one SlotRef.
+// 2nd phase: is_merge=false, maybe have multiple exprs.
+int StreamingAggLocalState::_get_slot_column_id(const vectorized::AggFnEvaluator* evaluator) {
+ auto ctxs = evaluator->input_exprs_ctxs();
+ CHECK(ctxs.size() == 1 && ctxs[0]->root()->is_slot_ref())
+ << "input_exprs_ctxs is invalid, input_exprs_ctx[0]="
+ << ctxs[0]->root()->debug_string();
+ return ((vectorized::VSlotRef*)ctxs[0]->root().get())->column_id();
+}
+
+void StreamingAggLocalState::_find_in_hash_table(vectorized::AggregateDataPtr* places,
+ vectorized::ColumnRawPtrs& key_columns,
+ size_t num_rows) {
+ std::visit(
+ [&](auto&& agg_method) -> void {
+ using HashMethodType = std::decay_t<decltype(agg_method)>;
+ using AggState = typename HashMethodType::State;
+ AggState state(key_columns);
+ agg_method.init_serialized_keys(key_columns, num_rows);
+
+ /// For all rows.
+ for (size_t i = 0; i < num_rows; ++i) {
+ auto find_result = agg_method.find(state, i);
+
+ if (find_result.is_found()) {
+ places[i] = find_result.get_mapped();
+ } else {
+ places[i] = nullptr;
+ }
+ }
+ },
+ _agg_data->method_variant);
+}
+
+Status StreamingAggLocalState::_merge_with_serialized_key(vectorized::Block* block) {
+ if (_reach_limit) {
+ return _merge_with_serialized_key_helper<true, false>(block);
+ } else {
+ return _merge_with_serialized_key_helper<false, false>(block);
+ }
+}
+
+void StreamingAggLocalState::_init_hash_method(const vectorized::VExprContextSPtrs& probe_exprs) {
+ DCHECK(probe_exprs.size() >= 1);
+
+ using Type = vectorized::AggregatedDataVariants::Type;
+ Type t(Type::serialized);
+
+ if (probe_exprs.size() == 1) {
+ auto is_nullable = probe_exprs[0]->root()->is_nullable();
+ PrimitiveType type = probe_exprs[0]->root()->result_type();
+ switch (type) {
+ case TYPE_TINYINT:
+ case TYPE_BOOLEAN:
+ case TYPE_SMALLINT:
+ case TYPE_INT:
+ case TYPE_FLOAT:
+ case TYPE_DATEV2:
+ case TYPE_BIGINT:
+ case TYPE_DOUBLE:
+ case TYPE_DATE:
+ case TYPE_DATETIME:
+ case TYPE_DATETIMEV2:
+ case TYPE_LARGEINT:
+ case TYPE_DECIMALV2:
+ case TYPE_DECIMAL32:
+ case TYPE_DECIMAL64:
+ case TYPE_DECIMAL128I: {
+ size_t size = get_primitive_type_size(type);
+ if (size == 1) {
+ t = Type::int8_key;
+ } else if (size == 2) {
+ t = Type::int16_key;
+ } else if (size == 4) {
+ t = Type::int32_key;
+ } else if (size == 8) {
+ t = Type::int64_key;
+ } else if (size == 16) {
+ t = Type::int128_key;
+ } else {
+ throw Exception(ErrorCode::INTERNAL_ERROR,
+ "meet invalid type size, size={}, type={}", size,
+ type_to_string(type));
+ }
+ break;
+ }
+ case TYPE_CHAR:
+ case TYPE_VARCHAR:
+ case TYPE_STRING: {
+ t = Type::string_key;
+ break;
+ }
+ default:
+ t = Type::serialized;
+ }
+
+ _agg_data->init(
+ get_hash_key_type_with_phase(
+ t, !Base::_parent->template cast<StreamingAggOperatorX>()._is_first_phase),
+ is_nullable);
+ } else {
+ if (!try_get_hash_map_context_fixed<PHNormalHashMap, HashCRC32,
+ vectorized::AggregateDataPtr>(_agg_data->method_variant,
+ probe_exprs)) {
+ _agg_data->init(Type::serialized);
+ }
+ }
+}
+
+Status StreamingAggLocalState::do_pre_agg(vectorized::Block* input_block,
+ vectorized::Block* output_block) {
+ RETURN_IF_ERROR(_pre_agg_with_serialized_key(input_block, output_block));
+
+ // pre stream agg need use _num_row_return to decide whether to do pre stream agg
+ _cur_num_rows_returned += output_block->rows();
+ _make_nullable_output_key(output_block);
+ // COUNTER_SET(_rows_returned_counter, _num_rows_returned);
+ _executor->update_memusage(this);
+ return Status::OK();
+}
+
+bool StreamingAggLocalState::_should_expand_preagg_hash_tables() {
+ if (!_should_expand_hash_table) {
+ return false;
+ }
+
+ return std::visit(
+ [&](auto&& agg_method) -> bool {
+ auto& hash_tbl = *agg_method.hash_table;
+ auto [ht_mem, ht_rows] =
+ std::pair {hash_tbl.get_buffer_size_in_bytes(), hash_tbl.size()};
+
+ // Need some rows in tables to have valid statistics.
+ if (ht_rows == 0) {
+ return true;
+ }
+
+ // Find the appropriate reduction factor in our table for the current hash table sizes.
+ int cache_level = 0;
+ while (cache_level + 1 < STREAMING_HT_MIN_REDUCTION_SIZE &&
+ ht_mem >= STREAMING_HT_MIN_REDUCTION[cache_level + 1].min_ht_mem) {
+ ++cache_level;
+ }
+
+ // Compare the number of rows in the hash table with the number of input rows that
+ // were aggregated into it. Exclude passed through rows from this calculation since
+ // they were not in hash tables.
+ const int64_t input_rows = _input_num_rows;
+ const int64_t aggregated_input_rows = input_rows - _cur_num_rows_returned;
+ // TODO chenhao
+ // const int64_t expected_input_rows = estimated_input_cardinality_ - num_rows_returned_;
+ double current_reduction = static_cast<double>(aggregated_input_rows) / ht_rows;
+
+ // TODO: workaround for IMPALA-2490: subplan node rows_returned counter may be
+ // inaccurate, which could lead to a divide by zero below.
+ if (aggregated_input_rows <= 0) {
+ return true;
+ }
+
+ // Extrapolate the current reduction factor (r) using the formula
+ // R = 1 + (N / n) * (r - 1), where R is the reduction factor over the full input data
+ // set, N is the number of input rows, excluding passed-through rows, and n is the
+ // number of rows inserted or merged into the hash tables. This is a very rough
+ // approximation but is good enough to be useful.
+ // TODO: consider collecting more statistics to better estimate reduction.
+ // double estimated_reduction = aggregated_input_rows >= expected_input_rows
+ // ? current_reduction
+ // : 1 + (expected_input_rows / aggregated_input_rows) * (current_reduction - 1);
+ double min_reduction =
+ STREAMING_HT_MIN_REDUCTION[cache_level].streaming_ht_min_reduction;
+
+ // COUNTER_SET(preagg_estimated_reduction_, estimated_reduction);
+ // COUNTER_SET(preagg_streaming_ht_min_reduction_, min_reduction);
+ // return estimated_reduction > min_reduction;
+ _should_expand_hash_table = current_reduction > min_reduction;
+ return _should_expand_hash_table;
+ },
+ _agg_data->method_variant);
+}
+
+size_t StreamingAggLocalState::_memory_usage() const {
+ size_t usage = 0;
+ if (_agg_arena_pool) {
+ usage += _agg_arena_pool->size();
+ }
+
+ if (_aggregate_data_container) {
+ usage += _aggregate_data_container->memory_usage();
+ }
+
+ return usage;
+}
+
+Status StreamingAggLocalState::_pre_agg_with_serialized_key(doris::vectorized::Block* in_block,
+ doris::vectorized::Block* out_block) {
+ SCOPED_TIMER(_build_timer);
+ DCHECK(!_probe_expr_ctxs.empty());
+
+ auto& p = Base::_parent->template cast<StreamingAggOperatorX>();
+
+ size_t key_size = _probe_expr_ctxs.size();
+ vectorized::ColumnRawPtrs key_columns(key_size);
+ {
+ SCOPED_TIMER(_expr_timer);
+ for (size_t i = 0; i < key_size; ++i) {
+ int result_column_id = -1;
+ RETURN_IF_ERROR(_probe_expr_ctxs[i]->execute(in_block, &result_column_id));
+ in_block->get_by_position(result_column_id).column =
+ in_block->get_by_position(result_column_id)
+ .column->convert_to_full_column_if_const();
+ key_columns[i] = in_block->get_by_position(result_column_id).column.get();
+ }
+ }
+
+ int rows = in_block->rows();
+ _places.resize(rows);
+
+ // Stop expanding hash tables if we're not reducing the input sufficiently. As our
+ // hash tables expand out of each level of cache hierarchy, every hash table lookup
+ // will take longer. We also may not be able to expand hash tables because of memory
+ // pressure. In either case we should always use the remaining space in the hash table
+ // to avoid wasting memory.
+ // But for fixed hash map, it never need to expand
+ bool ret_flag = false;
+ RETURN_IF_ERROR(std::visit(
+ [&](auto&& agg_method) -> Status {
+ if (auto& hash_tbl = *agg_method.hash_table;
+ hash_tbl.add_elem_size_overflow(rows)) {
+ /// If too much memory is used during the pre-aggregation stage,
+ /// it is better to output the data directly without performing further aggregation.
+ const bool used_too_much_memory =
+ (_parent->cast<StreamingAggOperatorX>()._external_agg_bytes_threshold >
+ 0 &&
+ _memory_usage() > _parent->cast<StreamingAggOperatorX>()
+ ._external_agg_bytes_threshold);
+ // do not try to do agg, just init and serialize directly return the out_block
+ if (!_should_expand_preagg_hash_tables() || used_too_much_memory) {
+ SCOPED_TIMER(_streaming_agg_timer);
+ ret_flag = true;
+
+ // will serialize value data to string column.
+ // non-nullable column(id in `_make_nullable_keys`)
+ // will be converted to nullable.
+ bool mem_reuse = p._make_nullable_keys.empty() && out_block->mem_reuse();
+
+ std::vector<vectorized::DataTypePtr> data_types;
+ vectorized::MutableColumns value_columns;
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ auto data_type =
+ _aggregate_evaluators[i]->function()->get_serialized_type();
+ if (mem_reuse) {
+ value_columns.emplace_back(
+ std::move(*out_block->get_by_position(i + key_size).column)
+ .mutate());
+ } else {
+ // slot type of value it should always be string type
+ value_columns.emplace_back(_aggregate_evaluators[i]
+ ->function()
+ ->create_serialize_column());
+ }
+ data_types.emplace_back(data_type);
+ }
+
+ for (int i = 0; i != _aggregate_evaluators.size(); ++i) {
+ SCOPED_TIMER(_serialize_data_timer);
+ RETURN_IF_ERROR(
+ _aggregate_evaluators[i]->streaming_agg_serialize_to_column(
+ in_block, value_columns[i], rows,
+ _agg_arena_pool.get()));
+ }
+
+ if (!mem_reuse) {
+ vectorized::ColumnsWithTypeAndName columns_with_schema;
+ for (int i = 0; i < key_size; ++i) {
+ columns_with_schema.emplace_back(
+ key_columns[i]->clone_resized(rows),
+ _probe_expr_ctxs[i]->root()->data_type(),
+ _probe_expr_ctxs[i]->root()->expr_name());
+ }
+ for (int i = 0; i < value_columns.size(); ++i) {
+ columns_with_schema.emplace_back(std::move(value_columns[i]),
+ data_types[i], "");
+ }
+ out_block->swap(vectorized::Block(columns_with_schema));
+ } else {
+ for (int i = 0; i < key_size; ++i) {
+ std::move(*out_block->get_by_position(i).column)
+ .mutate()
+ ->insert_range_from(*key_columns[i], 0, rows);
+ }
+ }
+ }
+ }
+ return Status::OK();
+ },
+ _agg_data->method_variant));
+
+ if (!ret_flag) {
+ RETURN_IF_CATCH_EXCEPTION(_emplace_into_hash_table(_places.data(), key_columns, rows));
+
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ RETURN_IF_ERROR(_aggregate_evaluators[i]->execute_batch_add(
+ in_block, p._offsets_of_aggregate_states[i], _places.data(),
+ _agg_arena_pool.get(), _should_expand_hash_table));
+ }
+ }
+
+ return Status::OK();
+}
+
+Status StreamingAggLocalState::_create_agg_status(vectorized::AggregateDataPtr data) {
+ auto& p = Base::_parent->template cast<StreamingAggOperatorX>();
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ try {
+ _aggregate_evaluators[i]->create(data + p._offsets_of_aggregate_states[i]);
+ } catch (...) {
+ for (int j = 0; j < i; ++j) {
+ _aggregate_evaluators[j]->destroy(data + p._offsets_of_aggregate_states[j]);
+ }
+ throw;
+ }
+ }
+ return Status::OK();
+}
+
+Status StreamingAggLocalState::_get_with_serialized_key_result(RuntimeState* state,
+ vectorized::Block* block,
+ SourceState& source_state) {
+ auto& p = _parent->cast<StreamingAggOperatorX>();
+ // non-nullable column(id in `_make_nullable_keys`) will be converted to nullable.
+ bool mem_reuse = p._make_nullable_keys.empty() && block->mem_reuse();
+
+ auto columns_with_schema =
+ vectorized::VectorizedUtils::create_columns_with_type_and_name(p._row_descriptor);
+ int key_size = _probe_expr_ctxs.size();
+
+ vectorized::MutableColumns key_columns;
+ for (int i = 0; i < key_size; ++i) {
+ if (!mem_reuse) {
+ key_columns.emplace_back(columns_with_schema[i].type->create_column());
+ } else {
+ key_columns.emplace_back(std::move(*block->get_by_position(i).column).mutate());
+ }
+ }
+ vectorized::MutableColumns value_columns;
+ for (int i = key_size; i < columns_with_schema.size(); ++i) {
+ if (!mem_reuse) {
+ value_columns.emplace_back(columns_with_schema[i].type->create_column());
+ } else {
+ value_columns.emplace_back(std::move(*block->get_by_position(i).column).mutate());
+ }
+ }
+
+ SCOPED_TIMER(_get_results_timer);
+ std::visit(
+ [&](auto&& agg_method) -> void {
+ auto& data = *agg_method.hash_table;
+ agg_method.init_iterator();
+ const auto size = std::min(data.size(), size_t(state->batch_size()));
+ using KeyType = std::decay_t<decltype(agg_method.iterator->get_first())>;
+ std::vector<KeyType> keys(size);
+ if (_values.size() < size) {
+ _values.resize(size);
+ }
+
+ size_t num_rows = 0;
+ _aggregate_data_container->init_once();
+ auto& iter = _aggregate_data_container->iterator;
+
+ {
+ SCOPED_TIMER(_hash_table_iterate_timer);
+ while (iter != _aggregate_data_container->end() &&
+ num_rows < state->batch_size()) {
+ keys[num_rows] = iter.template get_key<KeyType>();
+ _values[num_rows] = iter.get_aggregate_data();
+ ++iter;
+ ++num_rows;
+ }
+ }
+
+ {
+ SCOPED_TIMER(_insert_keys_to_column_timer);
+ agg_method.insert_keys_into_columns(keys, key_columns, num_rows);
+ }
+
+ for (size_t i = 0; i < _aggregate_evaluators.size(); ++i) {
+ _aggregate_evaluators[i]->insert_result_info_vec(
+ _values, p._offsets_of_aggregate_states[i], value_columns[i].get(),
+ num_rows);
+ }
+
+ if (iter == _aggregate_data_container->end()) {
+ if (agg_method.hash_table->has_null_key_data()) {
+ // only one key of group by support wrap null key
+ // here need additional processing logic on the null key / value
+ DCHECK(key_columns.size() == 1);
+ DCHECK(key_columns[0]->is_nullable());
+ if (key_columns[0]->size() < state->batch_size()) {
+ key_columns[0]->insert_data(nullptr, 0);
+ auto mapped = agg_method.hash_table->template get_null_key_data<
+ vectorized::AggregateDataPtr>();
+ for (size_t i = 0; i < _aggregate_evaluators.size(); ++i)
+ _aggregate_evaluators[i]->insert_result_info(
+ mapped + p._offsets_of_aggregate_states[i],
+ value_columns[i].get());
+ source_state = SourceState::FINISHED;
+ }
+ } else {
+ source_state = SourceState::FINISHED;
+ }
+ }
+ },
+ _agg_data->method_variant);
+
+ if (!mem_reuse) {
+ *block = columns_with_schema;
+ vectorized::MutableColumns columns(block->columns());
+ for (int i = 0; i < block->columns(); ++i) {
+ if (i < key_size) {
+ columns[i] = std::move(key_columns[i]);
+ } else {
+ columns[i] = std::move(value_columns[i - key_size]);
+ }
+ }
+ block->set_columns(std::move(columns));
+ }
+
+ return Status::OK();
+}
+
+Status StreamingAggLocalState::_serialize_without_key(RuntimeState* state, vectorized::Block* block,
+ SourceState& source_state) {
+ // 1. `child(0)->rows_returned() == 0` mean not data from child
+ // in level two aggregation node should return NULL result
+ // level one aggregation node set `eos = true` return directly
+ SCOPED_TIMER(_serialize_result_timer);
+ if (UNLIKELY(_input_num_rows == 0)) {
+ source_state = SourceState::FINISHED;
+ return Status::OK();
+ }
+ block->clear();
+
+ DCHECK(_agg_data->without_key != nullptr);
+ int agg_size = _aggregate_evaluators.size();
+
+ vectorized::MutableColumns value_columns(agg_size);
+ std::vector<vectorized::DataTypePtr> data_types(agg_size);
+ // will serialize data to string column
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ data_types[i] = _aggregate_evaluators[i]->function()->get_serialized_type();
+ value_columns[i] = _aggregate_evaluators[i]->function()->create_serialize_column();
+ }
+
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ _aggregate_evaluators[i]->function()->serialize_without_key_to_column(
+ _agg_data->without_key +
+ _parent->cast<StreamingAggOperatorX>()._offsets_of_aggregate_states[i],
+ *value_columns[i]);
+ }
+
+ {
+ vectorized::ColumnsWithTypeAndName data_with_schema;
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ vectorized::ColumnWithTypeAndName column_with_schema = {nullptr, data_types[i], ""};
+ data_with_schema.push_back(std::move(column_with_schema));
+ }
+ *block = vectorized::Block(data_with_schema);
+ }
+
+ block->set_columns(std::move(value_columns));
+ source_state = SourceState::FINISHED;
+ return Status::OK();
+}
+
+Status StreamingAggLocalState::_serialize_with_serialized_key_result(RuntimeState* state,
+ vectorized::Block* block,
+ SourceState& source_state) {
+ SCOPED_TIMER(_serialize_result_timer);
+ auto& p = _parent->cast<StreamingAggOperatorX>();
+ int key_size = _probe_expr_ctxs.size();
+ int agg_size = _aggregate_evaluators.size();
+ vectorized::MutableColumns value_columns(agg_size);
+ vectorized::DataTypes value_data_types(agg_size);
+
+ // non-nullable column(id in `_make_nullable_keys`) will be converted to nullable.
+ bool mem_reuse = p._make_nullable_keys.empty() && block->mem_reuse();
+
+ vectorized::MutableColumns key_columns;
+ for (int i = 0; i < key_size; ++i) {
+ if (mem_reuse) {
+ key_columns.emplace_back(std::move(*block->get_by_position(i).column).mutate());
+ } else {
+ key_columns.emplace_back(_probe_expr_ctxs[i]->root()->data_type()->create_column());
+ }
+ }
+
+ SCOPED_TIMER(_get_results_timer);
+ std::visit(
+ [&](auto&& agg_method) -> void {
+ agg_method.init_iterator();
+ auto& data = *agg_method.hash_table;
+ const auto size = std::min(data.size(), size_t(state->batch_size()));
+ using KeyType = std::decay_t<decltype(agg_method.iterator->get_first())>;
+ std::vector<KeyType> keys(size);
+ if (_values.size() < size + 1) {
+ _values.resize(size + 1);
+ }
+
+ size_t num_rows = 0;
+ _aggregate_data_container->init_once();
+ auto& iter = _aggregate_data_container->iterator;
+
+ {
+ SCOPED_TIMER(_hash_table_iterate_timer);
+ while (iter != _aggregate_data_container->end() &&
+ num_rows < state->batch_size()) {
+ keys[num_rows] = iter.template get_key<KeyType>();
+ _values[num_rows] = iter.get_aggregate_data();
+ ++iter;
+ ++num_rows;
+ }
+ }
+
+ {
+ SCOPED_TIMER(_insert_keys_to_column_timer);
+ agg_method.insert_keys_into_columns(keys, key_columns, num_rows);
+ }
+
+ if (iter == _aggregate_data_container->end()) {
+ if (agg_method.hash_table->has_null_key_data()) {
+ // only one key of group by support wrap null key
+ // here need additional processing logic on the null key / value
+ DCHECK(key_columns.size() == 1);
+ DCHECK(key_columns[0]->is_nullable());
+ if (agg_method.hash_table->has_null_key_data()) {
+ key_columns[0]->insert_data(nullptr, 0);
+ _values[num_rows] = agg_method.hash_table->template get_null_key_data<
+ vectorized::AggregateDataPtr>();
+ ++num_rows;
+ source_state = SourceState::FINISHED;
+ }
+ } else {
+ source_state = SourceState::FINISHED;
+ }
+ }
+
+ {
+ SCOPED_TIMER(_serialize_data_timer);
+ for (size_t i = 0; i < _aggregate_evaluators.size(); ++i) {
+ value_data_types[i] =
+ _aggregate_evaluators[i]->function()->get_serialized_type();
+ if (mem_reuse) {
+ value_columns[i] =
+ std::move(*block->get_by_position(i + key_size).column)
+ .mutate();
+ } else {
+ value_columns[i] =
+ _aggregate_evaluators[i]->function()->create_serialize_column();
+ }
+ _aggregate_evaluators[i]->function()->serialize_to_column(
+ _values, p._offsets_of_aggregate_states[i], value_columns[i],
+ num_rows);
+ }
+ }
+ },
+ _agg_data->method_variant);
+
+ if (!mem_reuse) {
+ vectorized::ColumnsWithTypeAndName columns_with_schema;
+ for (int i = 0; i < key_size; ++i) {
+ columns_with_schema.emplace_back(std::move(key_columns[i]),
+ _probe_expr_ctxs[i]->root()->data_type(),
+ _probe_expr_ctxs[i]->root()->expr_name());
+ }
+ for (int i = 0; i < agg_size; ++i) {
+ columns_with_schema.emplace_back(std::move(value_columns[i]), value_data_types[i], "");
+ }
+ *block = vectorized::Block(columns_with_schema);
+ }
+
+ return Status::OK();
+}
+
+void StreamingAggLocalState::make_nullable_output_key(vectorized::Block* block) {
+ if (block->rows() != 0) {
+ for (auto cid : _parent->cast<StreamingAggOperatorX>()._make_nullable_keys) {
+ block->get_by_position(cid).column = make_nullable(block->get_by_position(cid).column);
+ block->get_by_position(cid).type = make_nullable(block->get_by_position(cid).type);
+ }
+ }
+}
+
+Status StreamingAggLocalState::_get_without_key_result(RuntimeState* state,
+ vectorized::Block* block,
+ SourceState& source_state) {
+ DCHECK(_agg_data->without_key != nullptr);
+ block->clear();
+
+ auto& p = _parent->cast<StreamingAggOperatorX>();
+ *block = vectorized::VectorizedUtils::create_empty_columnswithtypename(p._row_descriptor);
+ int agg_size = _aggregate_evaluators.size();
+
+ vectorized::MutableColumns columns(agg_size);
+ std::vector<vectorized::DataTypePtr> data_types(agg_size);
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ data_types[i] = _aggregate_evaluators[i]->function()->get_return_type();
+ columns[i] = data_types[i]->create_column();
+ }
+
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ auto column = columns[i].get();
+ _aggregate_evaluators[i]->insert_result_info(
+ _agg_data->without_key + p._offsets_of_aggregate_states[i], column);
+ }
+
+ const auto& block_schema = block->get_columns_with_type_and_name();
+ DCHECK_EQ(block_schema.size(), columns.size());
+ for (int i = 0; i < block_schema.size(); ++i) {
+ const auto column_type = block_schema[i].type;
+ if (!column_type->equals(*data_types[i])) {
+ if (!vectorized::is_array(remove_nullable(column_type))) {
+ if (!column_type->is_nullable() || data_types[i]->is_nullable() ||
+ !remove_nullable(column_type)->equals(*data_types[i])) {
+ return Status::InternalError(
+ "node id = {}, column_type not match data_types, column_type={}, "
+ "data_types={}",
+ _parent->node_id(), column_type->get_name(), data_types[i]->get_name());
+ }
+ }
+
+ if (column_type->is_nullable() && !data_types[i]->is_nullable()) {
+ vectorized::ColumnPtr ptr = std::move(columns[i]);
+ // unless `count`, other aggregate function dispose empty set should be null
+ // so here check the children row return
+ ptr = make_nullable(ptr, _input_num_rows == 0);
+ columns[i] = ptr->assume_mutable();
+ }
+ }
+ }
+
+ block->set_columns(std::move(columns));
+ source_state = SourceState::FINISHED;
+ return Status::OK();
+}
+
+void StreamingAggLocalState::_emplace_into_hash_table(vectorized::AggregateDataPtr* places,
+ vectorized::ColumnRawPtrs& key_columns,
+ const size_t num_rows) {
+ std::visit(
+ [&](auto&& agg_method) -> void {
+ SCOPED_TIMER(_hash_table_compute_timer);
+ using HashMethodType = std::decay_t<decltype(agg_method)>;
+ using AggState = typename HashMethodType::State;
+ AggState state(key_columns);
+ agg_method.init_serialized_keys(key_columns, num_rows);
+
+ auto creator = [this](const auto& ctor, auto& key, auto& origin) {
+ HashMethodType::try_presis_key_and_origin(key, origin, *_agg_arena_pool);
+ auto mapped = _aggregate_data_container->append_data(origin);
+ auto st = _create_agg_status(mapped);
+ if (!st) {
+ throw Exception(st.code(), st.to_string());
+ }
+ ctor(key, mapped);
+ };
+
+ auto creator_for_null_key = [&](auto& mapped) {
+ mapped = _agg_arena_pool->aligned_alloc(
+ Base::_parent->template cast<StreamingAggOperatorX>()
+ ._total_size_of_aggregate_states,
+ Base::_parent->template cast<StreamingAggOperatorX>()
+ ._align_aggregate_states);
+ auto st = _create_agg_status(mapped);
+ if (!st) {
+ throw Exception(st.code(), st.to_string());
+ }
+ };
+
+ SCOPED_TIMER(_hash_table_emplace_timer);
+ for (size_t i = 0; i < num_rows; ++i) {
+ places[i] = agg_method.lazy_emplace(state, i, creator, creator_for_null_key);
+ }
+
+ COUNTER_UPDATE(_hash_table_input_counter, num_rows);
+ },
+ _agg_data->method_variant);
+}
+
+StreamingAggOperatorX::StreamingAggOperatorX(ObjectPool* pool, int operator_id,
+ const TPlanNode& tnode, const DescriptorTbl& descs)
+ : StatefulOperatorX<StreamingAggLocalState>(pool, tnode, operator_id, descs),
+ _intermediate_tuple_id(tnode.agg_node.intermediate_tuple_id),
+ _intermediate_tuple_desc(nullptr),
+ _output_tuple_id(tnode.agg_node.output_tuple_id),
+ _output_tuple_desc(nullptr),
+ _needs_finalize(tnode.agg_node.need_finalize),
+ _is_merge(false),
+ _is_first_phase(tnode.agg_node.__isset.is_first_phase && tnode.agg_node.is_first_phase),
+ _have_conjuncts(tnode.__isset.vconjunct && !tnode.vconjunct.nodes.empty()) {}
+
+Status StreamingAggOperatorX::init(const TPlanNode& tnode, RuntimeState* state) {
+ RETURN_IF_ERROR(StatefulOperatorX<StreamingAggLocalState>::init(tnode, state));
+ // ignore return status for now , so we need to introduce ExecNode::init()
+ RETURN_IF_ERROR(
+ vectorized::VExpr::create_expr_trees(tnode.agg_node.grouping_exprs, _probe_expr_ctxs));
+
+ // init aggregate functions
+ _aggregate_evaluators.reserve(tnode.agg_node.aggregate_functions.size());
+ // In case of : `select * from (select GoodEvent from hits union select CounterID from hits) as h limit 10;`
+ // only union with limit: we can short circuit query the pipeline exec engine.
+ _can_short_circuit =
+ tnode.agg_node.aggregate_functions.empty() && state->enable_pipeline_exec();
+
+ TSortInfo dummy;
+ for (int i = 0; i < tnode.agg_node.aggregate_functions.size(); ++i) {
+ vectorized::AggFnEvaluator* evaluator = nullptr;
+ RETURN_IF_ERROR(vectorized::AggFnEvaluator::create(
+ _pool, tnode.agg_node.aggregate_functions[i],
+ tnode.agg_node.__isset.agg_sort_infos ? tnode.agg_node.agg_sort_infos[i] : dummy,
+ &evaluator));
+ _aggregate_evaluators.push_back(evaluator);
+ }
+
+ const auto& agg_functions = tnode.agg_node.aggregate_functions;
+ _external_agg_bytes_threshold = state->external_agg_bytes_threshold();
+
+ if (_external_agg_bytes_threshold > 0) {
+ _spill_partition_count_bits = 4;
+ if (state->query_options().__isset.external_agg_partition_bits) {
+ _spill_partition_count_bits = state->query_options().external_agg_partition_bits;
+ }
+ }
+
+ _is_merge = std::any_of(agg_functions.cbegin(), agg_functions.cend(),
+ [](const auto& e) { return e.nodes[0].agg_expr.is_merge_agg; });
+ _op_name = "STREAMING_AGGREGATION_OPERATOR";
+ return Status::OK();
+}
+
+Status StreamingAggOperatorX::prepare(RuntimeState* state) {
+ RETURN_IF_ERROR(StatefulOperatorX<StreamingAggLocalState>::prepare(state));
+ _intermediate_tuple_desc = state->desc_tbl().get_tuple_descriptor(_intermediate_tuple_id);
+ _output_tuple_desc = state->desc_tbl().get_tuple_descriptor(_output_tuple_id);
+ DCHECK_EQ(_intermediate_tuple_desc->slots().size(), _output_tuple_desc->slots().size());
+ RETURN_IF_ERROR(vectorized::VExpr::prepare(_probe_expr_ctxs, state, _child_x->row_desc()));
+
+ int j = _probe_expr_ctxs.size();
+ for (int i = 0; i < j; ++i) {
+ auto nullable_output = _output_tuple_desc->slots()[i]->is_nullable();
+ auto nullable_input = _probe_expr_ctxs[i]->root()->is_nullable();
+ if (nullable_output != nullable_input) {
+ DCHECK(nullable_output);
+ _make_nullable_keys.emplace_back(i);
+ }
+ }
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i, ++j) {
+ SlotDescriptor* intermediate_slot_desc = _intermediate_tuple_desc->slots()[j];
+ SlotDescriptor* output_slot_desc = _output_tuple_desc->slots()[j];
+ RETURN_IF_ERROR(_aggregate_evaluators[i]->prepare(
+ state, _child_x->row_desc(), intermediate_slot_desc, output_slot_desc));
+ }
+
+ _offsets_of_aggregate_states.resize(_aggregate_evaluators.size());
+
+ for (size_t i = 0; i < _aggregate_evaluators.size(); ++i) {
+ _offsets_of_aggregate_states[i] = _total_size_of_aggregate_states;
+
+ const auto& agg_function = _aggregate_evaluators[i]->function();
+ // aggreate states are aligned based on maximum requirement
+ _align_aggregate_states = std::max(_align_aggregate_states, agg_function->align_of_data());
+ _total_size_of_aggregate_states += agg_function->size_of_data();
+
+ // If not the last aggregate_state, we need pad it so that next aggregate_state will be aligned.
+ if (i + 1 < _aggregate_evaluators.size()) {
+ size_t alignment_of_next_state =
+ _aggregate_evaluators[i + 1]->function()->align_of_data();
+ if ((alignment_of_next_state & (alignment_of_next_state - 1)) != 0) {
+ return Status::RuntimeError("Logical error: align_of_data is not 2^N");
+ }
+
+ /// Extend total_size to next alignment requirement
+ /// Add padding by rounding up 'total_size_of_aggregate_states' to be a multiplier of alignment_of_next_state.
+ _total_size_of_aggregate_states =
+ (_total_size_of_aggregate_states + alignment_of_next_state - 1) /
+ alignment_of_next_state * alignment_of_next_state;
+ }
+ }
+
+ return Status::OK();
+}
+
+Status StreamingAggOperatorX::open(RuntimeState* state) {
+ RETURN_IF_ERROR(StatefulOperatorX<StreamingAggLocalState>::open(state));
+ RETURN_IF_ERROR(vectorized::VExpr::open(_probe_expr_ctxs, state));
+
+ for (int i = 0; i < _aggregate_evaluators.size(); ++i) {
+ RETURN_IF_ERROR(_aggregate_evaluators[i]->open(state));
+ _aggregate_evaluators[i]->set_version(state->be_exec_version());
+ }
+
+ return Status::OK();
+}
+
+Status StreamingAggLocalState::close(RuntimeState* state) {
+ if (_closed) {
+ return Status::OK();
+ }
+ SCOPED_TIMER(Base::exec_time_counter());
+ SCOPED_TIMER(Base::_close_timer);
+ if (Base::_closed) {
+ return Status::OK();
+ }
+ _pre_aggregated_block->clear();
+ vectorized::PODArray<vectorized::AggregateDataPtr> tmp_places;
+ _places.swap(tmp_places);
+
+ std::vector<char> tmp_deserialize_buffer;
+ _deserialize_buffer.swap(tmp_deserialize_buffer);
+ Base::_mem_tracker->release(_mem_usage_record.used_in_state + _mem_usage_record.used_in_arena);
+
+ /// _hash_table_size_counter may be null if prepare failed.
+ if (_hash_table_size_counter) {
+ std::visit(
+ [&](auto&& agg_method) {
+ COUNTER_SET(_hash_table_size_counter, int64_t(agg_method.hash_table->size()));
+ },
+ _agg_data->method_variant);
+ }
+ return Base::close(state);
+}
+
+Status StreamingAggOperatorX::pull(RuntimeState* state, vectorized::Block* block,
+ SourceState& source_state) const {
+ auto& local_state = get_local_state(state);
+ if (!local_state._pre_aggregated_block->empty()) {
+ local_state._pre_aggregated_block->swap(*block);
+ } else {
+ RETURN_IF_ERROR(
+ local_state._executor->get_result(&local_state, state, block, source_state));
+ local_state.make_nullable_output_key(block);
+ // dispose the having clause, should not be execute in prestreaming agg
+ RETURN_IF_ERROR(
+ vectorized::VExprContext::filter_block(_conjuncts, block, block->columns()));
+ }
+ local_state.reached_limit(block, source_state);
+
+ return Status::OK();
+}
+
+Status StreamingAggOperatorX::push(RuntimeState* state, vectorized::Block* in_block,
+ SourceState source_state) const {
+ auto& local_state = get_local_state(state);
+ local_state._input_num_rows += in_block->rows();
+ if (in_block->rows() > 0) {
+ RETURN_IF_ERROR(local_state.do_pre_agg(in_block, local_state._pre_aggregated_block.get()));
+ }
+ in_block->clear_column_data(_child_x->row_desc().num_materialized_slots());
+ return Status::OK();
+}
+
+bool StreamingAggOperatorX::need_more_input_data(RuntimeState* state) const {
+ auto& local_state = get_local_state(state);
+ return local_state._pre_aggregated_block->empty() &&
+ local_state._child_source_state != SourceState::FINISHED;
+}
+
+} // namespace doris::pipeline
diff --git a/be/src/pipeline/exec/streaming_aggregation_operator.h b/be/src/pipeline/exec/streaming_aggregation_operator.h
new file mode 100644
index 00000000000..213e63459d5
--- /dev/null
+++ b/be/src/pipeline/exec/streaming_aggregation_operator.h
@@ -0,0 +1,231 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <stdint.h>
+
+#include <memory>
+
+#include "common/status.h"
+#include "pipeline/pipeline_x/operator.h"
+#include "util/runtime_profile.h"
+#include "vec/core/block.h"
+
+namespace doris {
+class RuntimeState;
+
+namespace pipeline {
+
+class StreamingAggOperatorX;
+
+class StreamingAggLocalState final : public PipelineXLocalState<FakeDependency> {
+public:
+ using Parent = StreamingAggOperatorX;
+ using Base = PipelineXLocalState<FakeDependency>;
+ ENABLE_FACTORY_CREATOR(StreamingAggLocalState);
+ StreamingAggLocalState(RuntimeState* state, OperatorXBase* parent);
+ ~StreamingAggLocalState() override = default;
+
+ Status init(RuntimeState* state, LocalStateInfo& info) override;
+ Status close(RuntimeState* state) override;
+ Status do_pre_agg(vectorized::Block* input_block, vectorized::Block* output_block);
+ void make_nullable_output_key(vectorized::Block* block);
+
+private:
+ friend class StreamingAggOperatorX;
+ template <typename LocalStateType>
+ friend class StatefulOperatorX;
+
+ size_t _memory_usage() const;
+ Status _pre_agg_with_serialized_key(doris::vectorized::Block* in_block,
+ doris::vectorized::Block* out_block);
+ bool _should_expand_preagg_hash_tables();
+ void _make_nullable_output_key(vectorized::Block* block);
+ Status _execute_without_key(vectorized::Block* block);
+ Status _merge_without_key(vectorized::Block* block);
+ void _update_memusage_without_key();
+ Status _execute_with_serialized_key(vectorized::Block* block);
+ Status _merge_with_serialized_key(vectorized::Block* block);
+ void _update_memusage_with_serialized_key();
+ void _init_hash_method(const vectorized::VExprContextSPtrs& probe_exprs);
+ Status _get_without_key_result(RuntimeState* state, vectorized::Block* block,
+ SourceState& source_state);
+ Status _serialize_without_key(RuntimeState* state, vectorized::Block* block,
+ SourceState& source_state);
+ Status _get_with_serialized_key_result(RuntimeState* state, vectorized::Block* block,
+ SourceState& source_state);
+ Status _serialize_with_serialized_key_result(RuntimeState* state, vectorized::Block* block,
+ SourceState& source_state);
+
+ template <bool limit, bool for_spill = false>
+ Status _merge_with_serialized_key_helper(vectorized::Block* block);
+ template <bool limit>
+ Status _execute_with_serialized_key_helper(vectorized::Block* block);
+ void _find_in_hash_table(vectorized::AggregateDataPtr* places,
+ vectorized::ColumnRawPtrs& key_columns, size_t num_rows);
+ int _get_slot_column_id(const vectorized::AggFnEvaluator* evaluator);
+ void _emplace_into_hash_table(vectorized::AggregateDataPtr* places,
+ vectorized::ColumnRawPtrs& key_columns, const size_t num_rows);
+ Status _create_agg_status(vectorized::AggregateDataPtr data);
+ size_t _get_hash_table_size();
+
+ RuntimeProfile::Counter* _queue_byte_size_counter = nullptr;
+ RuntimeProfile::Counter* _queue_size_counter = nullptr;
+ RuntimeProfile::Counter* _streaming_agg_timer = nullptr;
+ RuntimeProfile::Counter* _hash_table_compute_timer = nullptr;
+ RuntimeProfile::Counter* _hash_table_emplace_timer = nullptr;
+ RuntimeProfile::Counter* _hash_table_input_counter = nullptr;
+ RuntimeProfile::Counter* _build_timer = nullptr;
+ RuntimeProfile::Counter* _expr_timer = nullptr;
+ RuntimeProfile::Counter* _build_table_convert_timer = nullptr;
+ RuntimeProfile::Counter* _serialize_key_timer = nullptr;
+ RuntimeProfile::Counter* _merge_timer = nullptr;
+ RuntimeProfile::Counter* _serialize_data_timer = nullptr;
+ RuntimeProfile::Counter* _deserialize_data_timer = nullptr;
+ RuntimeProfile::Counter* _max_row_size_counter = nullptr;
+ RuntimeProfile::Counter* _hash_table_memory_usage = nullptr;
+ RuntimeProfile::HighWaterMarkCounter* _serialize_key_arena_memory_usage = nullptr;
+ RuntimeProfile::Counter* _hash_table_size_counter = nullptr;
+ RuntimeProfile::Counter* _get_results_timer = nullptr;
+ RuntimeProfile::Counter* _serialize_result_timer = nullptr;
+ RuntimeProfile::Counter* _hash_table_iterate_timer = nullptr;
+ RuntimeProfile::Counter* _insert_keys_to_column_timer = nullptr;
+
+ bool _should_expand_hash_table = true;
+ int64_t _cur_num_rows_returned = 0;
+ std::unique_ptr<vectorized::Arena> _agg_arena_pool = nullptr;
+ vectorized::AggregatedDataVariantsUPtr _agg_data = nullptr;
+ std::vector<vectorized::AggFnEvaluator*> _aggregate_evaluators;
+ // group by k1,k2
+ vectorized::VExprContextSPtrs _probe_expr_ctxs;
+ std::unique_ptr<vectorized::Arena> _agg_profile_arena = nullptr;
+ std::unique_ptr<vectorized::AggregateDataContainer> _aggregate_data_container = nullptr;
+ bool _should_limit_output = false;
+ bool _reach_limit = false;
+ size_t _input_num_rows = 0;
+
+ vectorized::PODArray<vectorized::AggregateDataPtr> _places;
+ std::vector<char> _deserialize_buffer;
+
+ struct ExecutorBase {
+ virtual Status execute(StreamingAggLocalState* local_state, vectorized::Block* block) = 0;
+ virtual void update_memusage(StreamingAggLocalState* local_state) = 0;
+ virtual Status get_result(StreamingAggLocalState* local_state, RuntimeState* state,
+ vectorized::Block* block, SourceState& source_state) = 0;
+ virtual ~ExecutorBase() = default;
+ };
+ template <bool WithoutKey, bool NeedToMerge, bool NeedFinalize>
+ struct Executor final : public ExecutorBase {
+ Status get_result(StreamingAggLocalState* local_state, RuntimeState* state,
+ vectorized::Block* block, SourceState& source_state) override {
+ if constexpr (WithoutKey) {
+ if constexpr (NeedFinalize) {
+ return local_state->_get_without_key_result(state, block, source_state);
+ } else {
+ return local_state->_serialize_without_key(state, block, source_state);
+ }
+ } else {
+ if constexpr (NeedFinalize) {
+ return local_state->_get_with_serialized_key_result(state, block, source_state);
+ } else {
+ return local_state->_serialize_with_serialized_key_result(state, block,
+ source_state);
+ }
+ }
+ }
+
+ Status execute(StreamingAggLocalState* local_state, vectorized::Block* block) override {
+ if constexpr (WithoutKey) {
+ if constexpr (NeedToMerge) {
+ return local_state->_merge_without_key(block);
+ } else {
+ return local_state->_execute_without_key(block);
+ }
+ } else {
+ if constexpr (NeedToMerge) {
+ return local_state->_merge_with_serialized_key(block);
+ } else {
+ return local_state->_execute_with_serialized_key(block);
+ }
+ }
+ }
+
+ void update_memusage(StreamingAggLocalState* local_state) override {
+ if constexpr (WithoutKey) {
+ local_state->_update_memusage_without_key();
+ } else {
+ local_state->_update_memusage_with_serialized_key();
+ }
+ }
+ };
+ std::unique_ptr<ExecutorBase> _executor = nullptr;
+
+ struct MemoryRecord {
+ MemoryRecord() : used_in_arena(0), used_in_state(0) {}
+ int64_t used_in_arena;
+ int64_t used_in_state;
+ };
+ MemoryRecord _mem_usage_record;
+ std::unique_ptr<vectorized::Block> _child_block = nullptr;
+ SourceState _child_source_state;
+ std::unique_ptr<vectorized::Block> _pre_aggregated_block = nullptr;
+ std::vector<vectorized::AggregateDataPtr> _values;
+};
+
+class StreamingAggOperatorX final : public StatefulOperatorX<StreamingAggLocalState> {
+public:
+ StreamingAggOperatorX(ObjectPool* pool, int operator_id, const TPlanNode& tnode,
+ const DescriptorTbl& descs);
+ ~StreamingAggOperatorX() override = default;
+ Status init(const TPlanNode& tnode, RuntimeState* state) override;
+ Status prepare(RuntimeState* state) override;
+ Status open(RuntimeState* state) override;
+ Status pull(RuntimeState* state, vectorized::Block* block,
+ SourceState& source_state) const override;
+ Status push(RuntimeState* state, vectorized::Block* input_block,
+ SourceState source_state) const override;
+ bool need_more_input_data(RuntimeState* state) const override;
+
+private:
+ friend class StreamingAggLocalState;
+ // may be we don't have to know the tuple id
+ TupleId _intermediate_tuple_id;
+ TupleDescriptor* _intermediate_tuple_desc = nullptr;
+
+ TupleId _output_tuple_id;
+ TupleDescriptor* _output_tuple_desc = nullptr;
+ bool _needs_finalize;
+ bool _is_merge;
+ const bool _is_first_phase;
+ size_t _align_aggregate_states = 1;
+ /// The offset to the n-th aggregate function in a row of aggregate functions.
+ vectorized::Sizes _offsets_of_aggregate_states;
+ /// The total size of the row from the aggregate functions.
+ size_t _total_size_of_aggregate_states = 0;
+ size_t _external_agg_bytes_threshold;
+ // group by k1,k2
+ vectorized::VExprContextSPtrs _probe_expr_ctxs;
+ std::vector<vectorized::AggFnEvaluator*> _aggregate_evaluators;
+ bool _can_short_circuit = false;
+ std::vector<size_t> _make_nullable_keys;
+ size_t _spill_partition_count_bits;
+ bool _have_conjuncts;
+};
+
+} // namespace pipeline
+} // namespace doris
diff --git a/be/src/pipeline/exec/streaming_aggregation_sink_operator.cpp b/be/src/pipeline/exec/streaming_aggregation_sink_operator.cpp
index 41d46e2f3ad..5805b209f4f 100644
--- a/be/src/pipeline/exec/streaming_aggregation_sink_operator.cpp
+++ b/be/src/pipeline/exec/streaming_aggregation_sink_operator.cpp
@@ -90,302 +90,4 @@ StreamingAggSinkOperatorBuilder::StreamingAggSinkOperatorBuilder(int32_t id, Exe
OperatorPtr StreamingAggSinkOperatorBuilder::build_operator() {
return std::make_shared<StreamingAggSinkOperator>(this, _node, _data_queue);
}
-
-/// The minimum reduction factor (input rows divided by output rows) to grow hash tables
-/// in a streaming preaggregation, given that the hash tables are currently the given
-/// size or above. The sizes roughly correspond to hash table sizes where the bucket
-/// arrays will fit in a cache level. Intuitively, we don't want the working set of the
-/// aggregation to expand to the next level of cache unless we're reducing the input
-/// enough to outweigh the increased memory latency we'll incur for each hash table
-/// lookup.
-///
-/// Note that the current reduction achieved is not always a good estimate of the
-/// final reduction. It may be biased either way depending on the ordering of the
-/// input. If the input order is random, we will underestimate the final reduction
-/// factor because the probability of a row having the same key as a previous row
-/// increases as more input is processed. If the input order is correlated with the
-/// key, skew may bias the estimate. If high cardinality keys appear first, we
-/// may overestimate and if low cardinality keys appear first, we underestimate.
-/// To estimate the eventual reduction achieved, we estimate the final reduction
-/// using the planner's estimated input cardinality and the assumption that input
-/// is in a random order. This means that we assume that the reduction factor will
-/// increase over time.
-struct StreamingHtMinReductionEntry {
- // Use 'streaming_ht_min_reduction' if the total size of hash table bucket directories in
- // bytes is greater than this threshold.
- int min_ht_mem;
- // The minimum reduction factor to expand the hash tables.
- double streaming_ht_min_reduction;
-};
-
-// TODO: experimentally tune these values and also programmatically get the cache size
-// of the machine that we're running on.
-static constexpr StreamingHtMinReductionEntry STREAMING_HT_MIN_REDUCTION[] = {
- // Expand up to L2 cache always.
- {0, 0.0},
- // Expand into L3 cache if we look like we're getting some reduction.
- // At present, The L2 cache is generally 1024k or more
- {1024 * 1024, 1.1},
- // Expand into main memory if we're getting a significant reduction.
- // The L3 cache is generally 16MB or more
- {16 * 1024 * 1024, 2.0},
-};
-
-static constexpr int STREAMING_HT_MIN_REDUCTION_SIZE =
- sizeof(STREAMING_HT_MIN_REDUCTION) / sizeof(STREAMING_HT_MIN_REDUCTION[0]);
-
-StreamingAggSinkLocalState::StreamingAggSinkLocalState(DataSinkOperatorXBase* parent,
- RuntimeState* state)
- : Base(parent, state),
- _queue_byte_size_counter(nullptr),
- _queue_size_counter(nullptr),
- _streaming_agg_timer(nullptr) {}
-
-Status StreamingAggSinkLocalState::init(RuntimeState* state, LocalSinkStateInfo& info) {
- RETURN_IF_ERROR(Base::init(state, info));
- _queue_byte_size_counter = ADD_COUNTER(profile(), "MaxSizeInBlockQueue", TUnit::BYTES);
- _queue_size_counter = ADD_COUNTER(profile(), "MaxSizeOfBlockQueue", TUnit::UNIT);
- _streaming_agg_timer = ADD_TIMER(profile(), "StreamingAggTime");
- return Status::OK();
-}
-
-Status StreamingAggSinkLocalState::do_pre_agg(vectorized::Block* input_block,
- vectorized::Block* output_block) {
- RETURN_IF_ERROR(_pre_agg_with_serialized_key(input_block, output_block));
-
- // pre stream agg need use _num_row_return to decide whether to do pre stream agg
- _num_rows_returned += output_block->rows();
- _make_nullable_output_key(output_block);
- // COUNTER_SET(_rows_returned_counter, _num_rows_returned);
- _executor->update_memusage(this);
- return Status::OK();
-}
-
-bool StreamingAggSinkLocalState::_should_expand_preagg_hash_tables() {
- if (!_should_expand_hash_table) {
- return false;
- }
-
- return std::visit(
- [&](auto&& agg_method) -> bool {
- auto& hash_tbl = *agg_method.hash_table;
- auto [ht_mem, ht_rows] =
- std::pair {hash_tbl.get_buffer_size_in_bytes(), hash_tbl.size()};
-
- // Need some rows in tables to have valid statistics.
- if (ht_rows == 0) {
- return true;
- }
-
- // Find the appropriate reduction factor in our table for the current hash table sizes.
- int cache_level = 0;
- while (cache_level + 1 < STREAMING_HT_MIN_REDUCTION_SIZE &&
- ht_mem >= STREAMING_HT_MIN_REDUCTION[cache_level + 1].min_ht_mem) {
- ++cache_level;
- }
-
- // Compare the number of rows in the hash table with the number of input rows that
- // were aggregated into it. Exclude passed through rows from this calculation since
- // they were not in hash tables.
- const int64_t input_rows = _shared_state->input_num_rows;
- const int64_t aggregated_input_rows = input_rows - _num_rows_returned;
- // TODO chenhao
- // const int64_t expected_input_rows = estimated_input_cardinality_ - num_rows_returned_;
- double current_reduction = static_cast<double>(aggregated_input_rows) / ht_rows;
-
- // TODO: workaround for IMPALA-2490: subplan node rows_returned counter may be
- // inaccurate, which could lead to a divide by zero below.
- if (aggregated_input_rows <= 0) {
- return true;
- }
-
- // Extrapolate the current reduction factor (r) using the formula
- // R = 1 + (N / n) * (r - 1), where R is the reduction factor over the full input data
- // set, N is the number of input rows, excluding passed-through rows, and n is the
- // number of rows inserted or merged into the hash tables. This is a very rough
- // approximation but is good enough to be useful.
- // TODO: consider collecting more statistics to better estimate reduction.
- // double estimated_reduction = aggregated_input_rows >= expected_input_rows
- // ? current_reduction
- // : 1 + (expected_input_rows / aggregated_input_rows) * (current_reduction - 1);
- double min_reduction =
- STREAMING_HT_MIN_REDUCTION[cache_level].streaming_ht_min_reduction;
-
- // COUNTER_SET(preagg_estimated_reduction_, estimated_reduction);
- // COUNTER_SET(preagg_streaming_ht_min_reduction_, min_reduction);
- // return estimated_reduction > min_reduction;
- _should_expand_hash_table = current_reduction > min_reduction;
- return _should_expand_hash_table;
- },
- _agg_data->method_variant);
-}
-
-Status StreamingAggSinkLocalState::_pre_agg_with_serialized_key(
- doris::vectorized::Block* in_block, doris::vectorized::Block* out_block) {
- SCOPED_TIMER(_build_timer);
- DCHECK(!_shared_state->probe_expr_ctxs.empty());
-
- size_t key_size = _shared_state->probe_expr_ctxs.size();
- vectorized::ColumnRawPtrs key_columns(key_size);
- {
- SCOPED_TIMER(_expr_timer);
- for (size_t i = 0; i < key_size; ++i) {
- int result_column_id = -1;
- RETURN_IF_ERROR(
- _shared_state->probe_expr_ctxs[i]->execute(in_block, &result_column_id));
- in_block->get_by_position(result_column_id).column =
- in_block->get_by_position(result_column_id)
- .column->convert_to_full_column_if_const();
- key_columns[i] = in_block->get_by_position(result_column_id).column.get();
- }
- }
-
- int rows = in_block->rows();
- _places.resize(rows);
-
- // Stop expanding hash tables if we're not reducing the input sufficiently. As our
- // hash tables expand out of each level of cache hierarchy, every hash table lookup
- // will take longer. We also may not be able to expand hash tables because of memory
- // pressure. In either case we should always use the remaining space in the hash table
- // to avoid wasting memory.
- // But for fixed hash map, it never need to expand
- bool ret_flag = false;
- RETURN_IF_ERROR(std::visit(
- [&](auto&& agg_method) -> Status {
- if (auto& hash_tbl = *agg_method.hash_table;
- hash_tbl.add_elem_size_overflow(rows)) {
- /// If too much memory is used during the pre-aggregation stage,
- /// it is better to output the data directly without performing further aggregation.
- const bool used_too_much_memory =
- (_parent->cast<StreamingAggSinkOperatorX>()
- ._external_agg_bytes_threshold > 0 &&
- _memory_usage() > _parent->cast<StreamingAggSinkOperatorX>()
- ._external_agg_bytes_threshold);
- // do not try to do agg, just init and serialize directly return the out_block
- if (!_should_expand_preagg_hash_tables() || used_too_much_memory) {
- SCOPED_TIMER(_streaming_agg_timer);
- ret_flag = true;
-
- // will serialize value data to string column.
- // non-nullable column(id in `_make_nullable_keys`)
- // will be converted to nullable.
- bool mem_reuse =
- _shared_state->make_nullable_keys.empty() && out_block->mem_reuse();
-
- std::vector<vectorized::DataTypePtr> data_types;
- vectorized::MutableColumns value_columns;
- for (int i = 0; i < _shared_state->aggregate_evaluators.size(); ++i) {
- auto data_type = _shared_state->aggregate_evaluators[i]
- ->function()
- ->get_serialized_type();
- if (mem_reuse) {
- value_columns.emplace_back(
- std::move(*out_block->get_by_position(i + key_size).column)
- .mutate());
- } else {
- // slot type of value it should always be string type
- value_columns.emplace_back(_shared_state->aggregate_evaluators[i]
- ->function()
- ->create_serialize_column());
- }
- data_types.emplace_back(data_type);
- }
-
- for (int i = 0; i != _shared_state->aggregate_evaluators.size(); ++i) {
- SCOPED_TIMER(_serialize_data_timer);
- RETURN_IF_ERROR(_shared_state->aggregate_evaluators[i]
- ->streaming_agg_serialize_to_column(
- in_block, value_columns[i], rows,
- _agg_arena_pool));
- }
-
- if (!mem_reuse) {
- vectorized::ColumnsWithTypeAndName columns_with_schema;
- for (int i = 0; i < key_size; ++i) {
- columns_with_schema.emplace_back(
- key_columns[i]->clone_resized(rows),
- _shared_state->probe_expr_ctxs[i]->root()->data_type(),
- _shared_state->probe_expr_ctxs[i]->root()->expr_name());
- }
- for (int i = 0; i < value_columns.size(); ++i) {
- columns_with_schema.emplace_back(std::move(value_columns[i]),
- data_types[i], "");
- }
- out_block->swap(vectorized::Block(columns_with_schema));
- } else {
- for (int i = 0; i < key_size; ++i) {
- std::move(*out_block->get_by_position(i).column)
- .mutate()
- ->insert_range_from(*key_columns[i], 0, rows);
- }
- }
- }
- }
- return Status::OK();
- },
- _agg_data->method_variant));
-
- if (!ret_flag) {
- RETURN_IF_CATCH_EXCEPTION(_emplace_into_hash_table(_places.data(), key_columns, rows));
-
- for (int i = 0; i < _shared_state->aggregate_evaluators.size(); ++i) {
- RETURN_IF_ERROR(_shared_state->aggregate_evaluators[i]->execute_batch_add(
- in_block, _shared_state->offsets_of_aggregate_states[i], _places.data(),
- _agg_arena_pool, _should_expand_hash_table));
- }
- }
-
- return Status::OK();
-}
-
-StreamingAggSinkOperatorX::StreamingAggSinkOperatorX(ObjectPool* pool, int operator_id,
- const TPlanNode& tnode,
- const DescriptorTbl& descs)
- : AggSinkOperatorX<StreamingAggSinkLocalState>(pool, operator_id, tnode, descs, true) {}
-
-Status StreamingAggSinkOperatorX::init(const TPlanNode& tnode, RuntimeState* state) {
- RETURN_IF_ERROR(AggSinkOperatorX<StreamingAggSinkLocalState>::init(tnode, state));
- _name = "STREAMING_AGGREGATION_SINK_OPERATOR";
- return Status::OK();
-}
-
-Status StreamingAggSinkOperatorX::sink(RuntimeState* state, vectorized::Block* in_block,
- SourceState source_state) {
- auto& local_state = get_local_state(state);
- SCOPED_TIMER(local_state.exec_time_counter());
- COUNTER_UPDATE(local_state.rows_input_counter(), (int64_t)in_block->rows());
- local_state._shared_state->input_num_rows += in_block->rows();
- Status ret = Status::OK();
- if (in_block && in_block->rows() > 0) {
- auto block_from_ctx = local_state._shared_state->data_queue->get_free_block();
- RETURN_IF_ERROR(local_state.do_pre_agg(in_block, block_from_ctx.get()));
- if (block_from_ctx->rows() == 0) {
- local_state._shared_state->data_queue->push_free_block(std::move(block_from_ctx));
- } else {
- local_state._shared_state->data_queue->push_block(std::move(block_from_ctx));
- }
- }
-
- if (UNLIKELY(source_state == SourceState::FINISHED)) {
- local_state._shared_state->data_queue->set_finish();
- }
- return Status::OK();
-}
-
-Status StreamingAggSinkLocalState::close(RuntimeState* state, Status exec_status) {
- if (_closed) {
- return Status::OK();
- }
- if (_shared_state->data_queue && !_shared_state->data_queue->is_finish()) {
- // finish should be set, if not set here means error.
- _shared_state->data_queue->set_canceled();
- }
- if (_shared_state->data_queue) {
- COUNTER_SET(_queue_size_counter, _shared_state->data_queue->max_size_of_queue());
- COUNTER_SET(_queue_byte_size_counter, _shared_state->data_queue->max_bytes_in_queue());
- }
- _preagg_block.clear();
- return Base::close(state, exec_status);
-}
-
} // namespace doris::pipeline
diff --git a/be/src/pipeline/exec/streaming_aggregation_sink_operator.h b/be/src/pipeline/exec/streaming_aggregation_sink_operator.h
index bd2640517cf..99e94e2d585 100644
--- a/be/src/pipeline/exec/streaming_aggregation_sink_operator.h
+++ b/be/src/pipeline/exec/streaming_aggregation_sink_operator.h
@@ -72,66 +72,5 @@ private:
std::shared_ptr<DataQueue> _data_queue;
};
-class StreamingAggSinkOperatorX;
-
-class StreamingAggSinkDependency final : public Dependency {
-public:
- using SharedState = AggSharedState;
- StreamingAggSinkDependency(int id, int node_id, QueryContext* query_ctx)
- : Dependency(id, node_id, "StreamingAggSinkDependency", true, query_ctx) {}
- ~StreamingAggSinkDependency() override = default;
-};
-
-class StreamingAggSinkLocalState final
- : public AggSinkLocalState<StreamingAggSinkDependency, StreamingAggSinkLocalState> {
-public:
- using Parent = StreamingAggSinkOperatorX;
- using Base = AggSinkLocalState<StreamingAggSinkDependency, StreamingAggSinkLocalState>;
- ENABLE_FACTORY_CREATOR(StreamingAggSinkLocalState);
- StreamingAggSinkLocalState(DataSinkOperatorXBase* parent, RuntimeState* state);
- ~StreamingAggSinkLocalState() override = default;
-
- Status init(RuntimeState* state, LocalSinkStateInfo& info) override;
- Status close(RuntimeState* state, Status exec_status) override;
- Status do_pre_agg(vectorized::Block* input_block, vectorized::Block* output_block);
-
-private:
- friend class StreamingAggSinkOperatorX;
-
- Status _pre_agg_with_serialized_key(doris::vectorized::Block* in_block,
- doris::vectorized::Block* out_block);
- bool _should_expand_preagg_hash_tables();
- void _make_nullable_output_key(vectorized::Block* block) {
- if (block->rows() != 0) {
- auto& shared_state = *Base ::_shared_state;
- for (auto cid : shared_state.make_nullable_keys) {
- block->get_by_position(cid).column =
- make_nullable(block->get_by_position(cid).column);
- block->get_by_position(cid).type = make_nullable(block->get_by_position(cid).type);
- }
- }
- }
-
- RuntimeProfile::Counter* _queue_byte_size_counter = nullptr;
- RuntimeProfile::Counter* _queue_size_counter = nullptr;
- RuntimeProfile::Counter* _streaming_agg_timer = nullptr;
-
- bool _should_expand_hash_table = true;
- int64_t _num_rows_returned = 0;
-};
-
-class StreamingAggSinkOperatorX final : public AggSinkOperatorX<StreamingAggSinkLocalState> {
-public:
- StreamingAggSinkOperatorX(ObjectPool* pool, int operator_id, const TPlanNode& tnode,
- const DescriptorTbl& descs);
- ~StreamingAggSinkOperatorX() override = default;
- Status init(const TPlanNode& tnode, RuntimeState* state) override;
- Status sink(RuntimeState* state, vectorized::Block* in_block,
- SourceState source_state) override;
- DataDistribution required_data_distribution() const override {
- return DataSinkOperatorX<StreamingAggSinkLocalState>::required_data_distribution();
- }
-};
-
} // namespace pipeline
} // namespace doris
diff --git a/be/src/pipeline/exec/streaming_aggregation_source_operator.cpp b/be/src/pipeline/exec/streaming_aggregation_source_operator.cpp
index 44991e2e14f..a6585720835 100644
--- a/be/src/pipeline/exec/streaming_aggregation_source_operator.cpp
+++ b/be/src/pipeline/exec/streaming_aggregation_source_operator.cpp
@@ -73,44 +73,5 @@ OperatorPtr StreamingAggSourceOperatorBuilder::build_operator() {
return std::make_shared<StreamingAggSourceOperator>(this, _node, _data_queue);
}
-StreamingAggSourceOperatorX::StreamingAggSourceOperatorX(ObjectPool* pool, const TPlanNode& tnode,
- int operator_id,
- const DescriptorTbl& descs)
- : Base(pool, tnode, operator_id, descs, true) {}
-
-Status StreamingAggSourceOperatorX::get_block(RuntimeState* state, vectorized::Block* block,
- SourceState& source_state) {
- auto& local_state = get_local_state(state);
- SCOPED_TIMER(local_state.exec_time_counter());
- if (!local_state._shared_state->data_queue->data_exhausted()) {
- std::unique_ptr<vectorized::Block> agg_block = nullptr;
- DCHECK(local_state._dependency->is_blocked_by() == nullptr);
- RETURN_IF_ERROR(local_state._shared_state->data_queue->get_block_from_queue(&agg_block));
-
- if (local_state._shared_state->data_queue->data_exhausted()) {
- RETURN_IF_ERROR(Base::get_block(state, block, source_state));
- } else if (agg_block) {
- block->swap(*agg_block);
- agg_block->clear_column_data(row_desc().num_materialized_slots());
- local_state._shared_state->data_queue->push_free_block(std::move(agg_block));
- } else if (local_state._shared_state->data_queue->is_all_finish()) {
- source_state = SourceState::FINISHED;
- } else {
- return Status::InternalError("Something wrong in StreamingAggSource: {}",
- Base::debug_string(0));
- }
- } else {
- RETURN_IF_ERROR(Base::get_block(state, block, source_state));
- }
-
- return Status::OK();
-}
-
-Status StreamingAggSourceOperatorX::init(const TPlanNode& tnode, RuntimeState* state) {
- RETURN_IF_ERROR(Base::init(tnode, state));
- _op_name = "STREAMING_AGGREGATION_OPERATOR";
- return Status::OK();
-}
-
} // namespace pipeline
} // namespace doris
diff --git a/be/src/pipeline/exec/streaming_aggregation_source_operator.h b/be/src/pipeline/exec/streaming_aggregation_source_operator.h
index 51e3c9ed972..89dbaab058a 100644
--- a/be/src/pipeline/exec/streaming_aggregation_source_operator.h
+++ b/be/src/pipeline/exec/streaming_aggregation_source_operator.h
@@ -60,18 +60,5 @@ private:
std::shared_ptr<DataQueue> _data_queue;
};
-class StreamingAggSourceOperatorX final : public AggSourceOperatorX {
-public:
- using Base = AggSourceOperatorX;
- StreamingAggSourceOperatorX(ObjectPool* pool, const TPlanNode& tnode, int operator_id,
- const DescriptorTbl& descs);
- ~StreamingAggSourceOperatorX() = default;
-
- Status init(const TPlanNode& tnode, RuntimeState* state) override;
-
- Status get_block(RuntimeState* state, vectorized::Block* block,
- SourceState& source_state) override;
-};
-
} // namespace pipeline
} // namespace doris
diff --git a/be/src/pipeline/pipeline_x/dependency.h b/be/src/pipeline/pipeline_x/dependency.h
index b2ca808e209..326185c9d2b 100644
--- a/be/src/pipeline/pipeline_x/dependency.h
+++ b/be/src/pipeline/pipeline_x/dependency.h
@@ -341,7 +341,6 @@ public:
size_t input_num_rows = 0;
std::vector<vectorized::AggregateDataPtr> values;
std::unique_ptr<vectorized::Arena> agg_profile_arena;
- std::unique_ptr<DataQueue> data_queue = std::make_unique<DataQueue>(1, true);
/// The total size of the row from the aggregate functions.
size_t total_size_of_aggregate_states = 0;
size_t align_aggregate_states = 1;
@@ -402,8 +401,7 @@ public:
struct UnionSharedState : public BasicSharedState {
public:
- UnionSharedState(int child_count = 1)
- : data_queue(child_count, false), _child_count(child_count) {};
+ UnionSharedState(int child_count = 1) : data_queue(child_count), _child_count(child_count) {};
int child_count() const { return _child_count; }
DataQueue data_queue;
const int _child_count;
diff --git a/be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.h b/be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.h
index 63d71bbe08b..b62ec1a8fb9 100644
--- a/be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.h
+++ b/be/src/pipeline/pipeline_x/local_exchange/local_exchange_source_operator.h
@@ -78,7 +78,7 @@ public:
return _child_x->intermediate_row_desc();
}
RowDescriptor& row_descriptor() override { return _child_x->row_descriptor(); }
- const RowDescriptor& row_desc() override { return _child_x->row_desc(); }
+ const RowDescriptor& row_desc() const override { return _child_x->row_desc(); }
Status get_block(RuntimeState* state, vectorized::Block* block,
SourceState& source_state) override;
diff --git a/be/src/pipeline/pipeline_x/operator.cpp b/be/src/pipeline/pipeline_x/operator.cpp
index 86c7f885792..890c4298bb0 100644
--- a/be/src/pipeline/pipeline_x/operator.cpp
+++ b/be/src/pipeline/pipeline_x/operator.cpp
@@ -29,7 +29,7 @@
#include "pipeline/exec/analytic_source_operator.h"
#include "pipeline/exec/assert_num_rows_operator.h"
#include "pipeline/exec/datagen_operator.h"
-#include "pipeline/exec/distinct_streaming_aggregation_sink_operator.h"
+#include "pipeline/exec/distinct_streaming_aggregation_operator.h"
#include "pipeline/exec/empty_set_operator.h"
#include "pipeline/exec/es_scan_operator.h"
#include "pipeline/exec/exchange_sink_operator.h"
@@ -59,8 +59,7 @@
#include "pipeline/exec/set_source_operator.h"
#include "pipeline/exec/sort_sink_operator.h"
#include "pipeline/exec/sort_source_operator.h"
-#include "pipeline/exec/streaming_aggregation_sink_operator.h"
-#include "pipeline/exec/streaming_aggregation_source_operator.h"
+#include "pipeline/exec/streaming_aggregation_operator.h"
#include "pipeline/exec/table_function_operator.h"
#include "pipeline/exec/union_sink_operator.h"
#include "pipeline/exec/union_source_operator.h"
@@ -72,10 +71,6 @@
namespace doris::pipeline {
-template class AggSinkLocalState<DistinctStreamingAggSinkDependency,
- DistinctStreamingAggSinkLocalState>;
-template class AggSinkLocalState<StreamingAggSinkDependency, StreamingAggSinkLocalState>;
-
template <typename DependencyType>
std::string PipelineXLocalState<DependencyType>::debug_string(int indentation_level) const {
fmt::memory_buffer debug_string_buffer;
@@ -601,9 +596,7 @@ DECLARE_OPERATOR_X(OlapTableSinkV2LocalState)
DECLARE_OPERATOR_X(AnalyticSinkLocalState)
DECLARE_OPERATOR_X(SortSinkLocalState)
DECLARE_OPERATOR_X(LocalExchangeSinkLocalState)
-DECLARE_OPERATOR_X(BlockingAggSinkLocalState)
-DECLARE_OPERATOR_X(StreamingAggSinkLocalState)
-DECLARE_OPERATOR_X(DistinctStreamingAggSinkLocalState)
+DECLARE_OPERATOR_X(AggSinkLocalState)
DECLARE_OPERATOR_X(ExchangeSinkLocalState)
DECLARE_OPERATOR_X(NestedLoopJoinBuildSinkLocalState)
DECLARE_OPERATOR_X(UnionSinkLocalState)
@@ -648,6 +641,8 @@ template class StreamingOperatorX<SelectLocalState>;
template class StatefulOperatorX<HashJoinProbeLocalState>;
template class StatefulOperatorX<RepeatLocalState>;
+template class StatefulOperatorX<StreamingAggLocalState>;
+template class StatefulOperatorX<DistinctStreamingAggLocalState>;
template class StatefulOperatorX<NestedLoopJoinProbeLocalState>;
template class StatefulOperatorX<TableFunctionLocalState>;
@@ -656,8 +651,6 @@ template class PipelineXSinkLocalState<SortSinkDependency>;
template class PipelineXSinkLocalState<NestedLoopJoinBuildSinkDependency>;
template class PipelineXSinkLocalState<AnalyticSinkDependency>;
template class PipelineXSinkLocalState<AggSinkDependency>;
-template class PipelineXSinkLocalState<StreamingAggSinkDependency>;
-template class PipelineXSinkLocalState<DistinctStreamingAggSinkDependency>;
template class PipelineXSinkLocalState<FakeDependency>;
template class PipelineXSinkLocalState<UnionSinkDependency>;
template class PipelineXSinkLocalState<PartitionSortSinkDependency>;
diff --git a/be/src/pipeline/pipeline_x/operator.h b/be/src/pipeline/pipeline_x/operator.h
index 61afe40e126..848ed5cd001 100644
--- a/be/src/pipeline/pipeline_x/operator.h
+++ b/be/src/pipeline/pipeline_x/operator.h
@@ -278,7 +278,7 @@ public:
[[nodiscard]] int64_t limit() const { return _limit; }
- [[nodiscard]] virtual const RowDescriptor& row_desc() override {
+ [[nodiscard]] const RowDescriptor& row_desc() const override {
return _output_row_descriptor ? *_output_row_descriptor : _row_descriptor;
}
diff --git a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
index 67c5da97063..38b7b7457fe 100644
--- a/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
+++ b/be/src/pipeline/pipeline_x/pipeline_x_fragment_context.cpp
@@ -44,8 +44,7 @@
#include "pipeline/exec/assert_num_rows_operator.h"
#include "pipeline/exec/data_queue.h"
#include "pipeline/exec/datagen_operator.h"
-#include "pipeline/exec/distinct_streaming_aggregation_sink_operator.h"
-#include "pipeline/exec/distinct_streaming_aggregation_source_operator.h"
+#include "pipeline/exec/distinct_streaming_aggregation_operator.h"
#include "pipeline/exec/empty_set_operator.h"
#include "pipeline/exec/es_scan_operator.h"
#include "pipeline/exec/exchange_sink_operator.h"
@@ -76,8 +75,7 @@
#include "pipeline/exec/set_source_operator.h"
#include "pipeline/exec/sort_sink_operator.h"
#include "pipeline/exec/sort_source_operator.h"
-#include "pipeline/exec/streaming_aggregation_sink_operator.h"
-#include "pipeline/exec/streaming_aggregation_source_operator.h"
+#include "pipeline/exec/streaming_aggregation_operator.h"
#include "pipeline/exec/table_function_operator.h"
#include "pipeline/exec/union_sink_operator.h"
#include "pipeline/exec/union_source_operator.h"
@@ -950,39 +948,12 @@ Status PipelineXFragmentContext::_create_operator(ObjectPool* pool, const TPlanN
}
case TPlanNodeType::AGGREGATION_NODE: {
if (tnode.agg_node.aggregate_functions.empty()) {
- op.reset(new DistinctStreamingAggSourceOperatorX(pool, tnode, next_operator_id(),
- descs));
+ op.reset(new DistinctStreamingAggOperatorX(pool, next_operator_id(), tnode, descs));
RETURN_IF_ERROR(cur_pipe->add_operator(op));
-
- const auto downstream_pipeline_id = cur_pipe->id();
- if (_dag.find(downstream_pipeline_id) == _dag.end()) {
- _dag.insert({downstream_pipeline_id, {}});
- }
- cur_pipe = add_pipeline(cur_pipe);
- _dag[downstream_pipeline_id].push_back(cur_pipe->id());
- DataSinkOperatorXPtr sink;
- sink.reset(new DistinctStreamingAggSinkOperatorX(pool, next_sink_operator_id(), tnode,
- descs));
- sink->set_dests_id({op->operator_id()});
- RETURN_IF_ERROR(cur_pipe->set_sink(sink));
- RETURN_IF_ERROR(cur_pipe->sink_x()->init(tnode, _runtime_state.get()));
} else if (tnode.agg_node.__isset.use_streaming_preaggregation &&
tnode.agg_node.use_streaming_preaggregation) {
- op.reset(new StreamingAggSourceOperatorX(pool, tnode, next_operator_id(), descs));
+ op.reset(new StreamingAggOperatorX(pool, next_operator_id(), tnode, descs));
RETURN_IF_ERROR(cur_pipe->add_operator(op));
-
- const auto downstream_pipeline_id = cur_pipe->id();
- if (_dag.find(downstream_pipeline_id) == _dag.end()) {
- _dag.insert({downstream_pipeline_id, {}});
- }
- cur_pipe = add_pipeline(cur_pipe);
- _dag[downstream_pipeline_id].push_back(cur_pipe->id());
- DataSinkOperatorXPtr sink;
- sink.reset(new StreamingAggSinkOperatorX(pool, next_sink_operator_id(), tnode, descs));
- sink->set_dests_id({op->operator_id()});
- RETURN_IF_ERROR(cur_pipe->set_sink(sink));
- RETURN_IF_ERROR(cur_pipe->sink_x()->init(tnode, _runtime_state.get()));
-
} else {
op.reset(new AggSourceOperatorX(pool, tnode, next_operator_id(), descs));
RETURN_IF_ERROR(cur_pipe->add_operator(op));
@@ -995,7 +966,7 @@ Status PipelineXFragmentContext::_create_operator(ObjectPool* pool, const TPlanN
_dag[downstream_pipeline_id].push_back(cur_pipe->id());
DataSinkOperatorXPtr sink;
- sink.reset(new AggSinkOperatorX<>(pool, next_sink_operator_id(), tnode, descs));
+ sink.reset(new AggSinkOperatorX(pool, next_sink_operator_id(), tnode, descs));
sink->set_dests_id({op->operator_id()});
RETURN_IF_ERROR(cur_pipe->set_sink(sink));
RETURN_IF_ERROR(cur_pipe->sink_x()->init(tnode, _runtime_state.get()));
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 19/19: [Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog (#31153)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit c27692fb3bd148b1a98a49d41ed1f6e7d99207c1
Author: zy-kkk <zh...@gmail.com>
AuthorDate: Wed Feb 21 12:07:31 2024 +0800
[Enhancement](jdbc catalog) Add security check on driver when creating Jdbc Catalog (#31153)
---
docs/en/docs/lakehouse/multi-catalog/jdbc.md | 24 ++++++++++++++++++++--
docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md | 24 ++++++++++++++++++++--
.../main/java/org/apache/doris/common/Config.java | 8 ++++++++
.../org/apache/doris/catalog/JdbcResource.java | 19 +++++++++++++++--
.../doris/datasource/jdbc/JdbcExternalCatalog.java | 24 ++++++++++++++--------
.../org/apache/doris/catalog/JdbcResourceTest.java | 15 ++++++++++++++
6 files changed, 99 insertions(+), 15 deletions(-)
diff --git a/docs/en/docs/lakehouse/multi-catalog/jdbc.md b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
index a2fbde91768..b0019373f72 100644
--- a/docs/en/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/en/docs/lakehouse/multi-catalog/jdbc.md
@@ -61,12 +61,32 @@ PROPERTIES ("key"="value", ...)
`driver_url` can be specified in three ways:
-1. File name. For example, `mysql-connector-java-5.1.47.jar`. Please place the Jar file package in `jdbc_drivers/` under the FE/BE deployment directory in advance so the system can locate the file. You can change the location of the file by modifying `jdbc_drivers_dir` in fe.conf and be.conf.
+1. File name. For example, `mysql-connector-java-8.0.25.jar`. Please place the Jar file package in `jdbc_drivers/` under the FE/BE deployment directory in advance so the system can locate the file. You can change the location of the file by modifying `jdbc_drivers_dir` in fe.conf and be.conf.
-2. Local absolute path. For example, `file:///path/to/mysql-connector-java-5.1.47.jar`. Please place the Jar file package in the specified paths of FE/BE node.
+2. Local absolute path. For example, `file:///path/to/mysql-connector-java-8.0.25.jar`. Please place the Jar file package in the specified paths of FE/BE node.
3. HTTP address. For example, `https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar`. The system will download the Driver file from the HTTP address. This only supports HTTP services with no authentication requirements.
+**Driver package security**
+
+In order to prevent the use of a Driver Jar package with an unallowed path when creating the Catalog, Doris will perform path management and checksum checking on the Jar package.
+
+1. For the above method 1, the `jdbc_drivers_dir` configured by the Doris default user and all Jar packages in its directory are safe and will not be path checked.
+
+2. For the above methods 2 and 3, Doris will check the source of the Jar package. The checking rules are as follows:
+
+ * Control the allowed driver package paths through the FE configuration item `jdbc_driver_secure_path`. This configuration item can configure multiple paths, separated by semicolons. When this item is configured, Doris will check whether the prefix of the driver_url path in the Catalog properties is in `jdbc_driver_secure_path`. If not, it will refuse to create the Catalog.
+ * This parameter defaults to `*`, which means Jar packages of all paths are allowed.
+ * If the configuration `jdbc_driver_secure_path` is empty, driver packages for all paths are not allowed, which means that the driver package can only be specified using method 1 above.
+
+ > If you configure `jdbc_driver_secure_path = "file:///path/to/jdbc_drivers;http://path/to/jdbc_drivers"`, only `file:///path/to/jdbc_drivers` or `http:// is allowed The driver package path starting with path/to/jdbc_drivers`.
+
+3. When creating a Catalog, you can specify the checksum of the driver package through the `checksum` parameter. Doris will verify the driver package after loading the driver package. If the verification fails, it will refuse to create the Catalog.
+
+:::warning
+The above verification will only be performed when the Catalog is created. For already created Catalogs, verification will not be performed again.
+:::
+
### Lowercase name synchronization
When `lower_case_meta_names` is set to `true`, Doris maintains the mapping of lowercase names to actual names in the remote system, enabling queries to use lowercase to query non-lowercase databases, tables and columns of external data sources.
diff --git a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
index c426098772b..00ef1afcf65 100644
--- a/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
+++ b/docs/zh-CN/docs/lakehouse/multi-catalog/jdbc.md
@@ -61,12 +61,32 @@ PROPERTIES ("key"="value", ...)
`driver_url` 可以通过以下三种方式指定：
-1. 文件名。如 `mysql-connector-java-5.1.47.jar`。需将 Jar 包预先存放在 FE 和 BE 部署目录的 `jdbc_drivers/` 目录下。系统会自动在这个目录下寻找。该目录的位置，也可以由 fe.conf 和 be.conf 中的 `jdbc_drivers_dir` 配置修改。
+1. 文件名。如 `mysql-connector-java-8.0.25.jar`。需将 Jar 包预先存放在 FE 和 BE 部署目录的 `jdbc_drivers/` 目录下。系统会自动在这个目录下寻找。该目录的位置，也可以由 fe.conf 和 be.conf 中的 `jdbc_drivers_dir` 配置修改。
-2. 本地绝对路径。如 `file:///path/to/mysql-connector-java-5.1.47.jar`。需将 Jar 包预先存放在所有 FE/BE 节点指定的路径下。
+2. 本地绝对路径。如 `file:///path/to/mysql-connector-java-8.0.25.jar`。需将 Jar 包预先存放在所有 FE/BE 节点指定的路径下。
3. Http 地址。如：`https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar`。系统会从这个 http 地址下载 Driver 文件。仅支持无认证的 http 服务。
+**驱动包安全性**
+
+为了防止在创建 Catalog 时使用了未允许路径的 Driver Jar 包，Doris 会对 Jar 包进行路径管理和校验和检查。
+
+1. 针对上述方式 1，Doris 默认用户配置的 `jdbc_drivers_dir` 和其目录下的所有 Jar 包都是安全的，不会对其进行路径检查。
+
+2. 针对上述方式 2、3 ，Doris 会对 Jar 包的来源进行检查，检查规则如下：
+
+ * 通过 FE 配置项 `jdbc_driver_secure_path` 来控制允许的驱动包路径，该配置项可配置多个路径，以分号分隔。当配置了该项时，Doris 会检查 Catalog properties 中 driver_url 的路径是的部分前缀是否在 `jdbc_driver_secure_path` 中，如果不在其中，则会拒绝创建 Catalog。
+ * 此参数默认为 `*` ，表示允许所有路径的 Jar 包。
+ * 如果配置 `jdbc_driver_secure_path` 为空，则不允许所有路径的驱动包，也就意味着只能使用上述方式 1 来指定驱动包。
+
+ > 如配置 `jdbc_driver_secure_path = "file:///path/to/jdbc_drivers;http://path/to/jdbc_drivers"`, 则只允许以 `file:///path/to/jdbc_drivers` 或 `http://path/to/jdbc_drivers` 开头的驱动包路径。
+
+3. 在创建 Catalog 时，可以通过 `checksum` 参数来指定驱动包的校验和，Doris 会在加载驱动包后，对驱动包进行校验，如果校验失败，则会拒绝创建 Catalog。
+
+:::warning
+上述的校验只会在创建 Catalog 时进行，对于已经创建的 Catalog，不会再次进行校验。
+:::
+
### 小写名称同步
当 `lower_case_meta_names` 设置为 `true` 时，Doris 通过维护小写名称到远程系统中实际名称的映射，使查询时能够使用小写去查询外部数据源非小写的数据库和表以及列。
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 2bb6173acd8..86ed91d3c08 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -142,6 +142,14 @@ public class Config extends ConfigBase {
+ "if the specified driver file path is not an absolute path, Doris will find jars from this path"})
public static String jdbc_drivers_dir = System.getenv("DORIS_HOME") + "/jdbc_drivers";
+ @ConfField(description = {"JDBC 驱动的安全路径。在创建 JDBC Catalog 时，允许使用的文件或者网络路径，可配置多个，使用分号分隔"
+ + "默认为 * 全部允许，如果设置为空责全部不允许",
+ "The safe path of the JDBC driver. When creating a JDBC Catalog,"
+ + "you can configure multiple files or network paths that are allowed to be used,"
+ + "separated by semicolons"
+ + "The default is * to allow all, if set to empty, all are not allowed"})
+ public static String jdbc_driver_secure_path = "*";
+
@ConfField(mutable = true, masterOnly = true, description = {"broker load 时，单个节点上 load 执行计划的默认并行度",
"The default parallelism of the load execution plan on a single node when the broker load is submitted"})
public static int default_load_parallelism = 8;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
index 03bdd3d2f30..6b19137dff0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/JdbcResource.java
@@ -42,6 +42,7 @@ import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.time.LocalDateTime;
import java.time.ZoneId;
+import java.util.Arrays;
import java.util.Map;
@@ -277,14 +278,28 @@ public class JdbcResource extends Resource {
}
}
- public static String getFullDriverUrl(String driverUrl) {
+ public static String getFullDriverUrl(String driverUrl) throws IllegalArgumentException {
try {
URI uri = new URI(driverUrl);
String schema = uri.getScheme();
if (schema == null && !driverUrl.startsWith("/")) {
return "file://" + Config.jdbc_drivers_dir + "/" + driverUrl;
+ } else {
+ if ("*".equals(Config.jdbc_driver_secure_path)) {
+ return driverUrl;
+ } else if (Config.jdbc_driver_secure_path.trim().isEmpty()) {
+ throw new IllegalArgumentException(
+ "jdbc_driver_secure_path is set to empty, disallowing all driver URLs.");
+ } else {
+ boolean isAllowed = Arrays.stream(Config.jdbc_driver_secure_path.split(";"))
+ .anyMatch(allowedPath -> driverUrl.startsWith(allowedPath.trim()));
+ if (!isAllowed) {
+ throw new IllegalArgumentException("Driver URL does not match any allowed paths: " + driverUrl);
+ } else {
+ return driverUrl;
+ }
+ }
}
- return driverUrl;
} catch (URISyntaxException e) {
LOG.warn("invalid jdbc driver url: " + driverUrl);
return driverUrl;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
index ee2ca9e0e00..93ebb1b2134 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java
@@ -119,11 +119,6 @@ public class JdbcExternalCatalog extends ExternalCatalog {
jdbcUrl = JdbcResource.handleJdbcUrl(jdbcUrl);
properties.put(JdbcResource.JDBC_URL, jdbcUrl);
}
-
- if (properties.containsKey(JdbcResource.DRIVER_URL) && !properties.containsKey(JdbcResource.CHECK_SUM)) {
- properties.put(JdbcResource.CHECK_SUM,
- JdbcResource.computeObjectChecksum(properties.get(JdbcResource.DRIVER_URL)));
- }
return properties;
}
@@ -246,10 +241,21 @@ public class JdbcExternalCatalog extends ExternalCatalog {
if (isReplay) {
return;
}
- Map<String, String> properties = Maps.newHashMap();
- if (properties.containsKey(JdbcResource.DRIVER_URL) && !properties.containsKey(JdbcResource.CHECK_SUM)) {
- properties.put(JdbcResource.CHECK_SUM,
- JdbcResource.computeObjectChecksum(properties.get(JdbcResource.DRIVER_URL)));
+ Map<String, String> properties = catalogProperty.getProperties();
+ if (properties.containsKey(JdbcResource.DRIVER_URL)) {
+ String computedChecksum = JdbcResource.computeObjectChecksum(properties.get(JdbcResource.DRIVER_URL));
+ if (properties.containsKey(JdbcResource.CHECK_SUM)) {
+ String providedChecksum = properties.get(JdbcResource.CHECK_SUM);
+ if (!providedChecksum.equals(computedChecksum)) {
+ throw new DdlException(
+ "The provided checksum (" + providedChecksum
+ + ") does not match the computed checksum (" + computedChecksum
+ + ") for the driver_url."
+ );
+ }
+ } else {
+ catalogProperty.addProperty(JdbcResource.CHECK_SUM, computedChecksum);
+ }
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java
index 87dcecc79dd..8e004d4b236 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/JdbcResourceTest.java
@@ -20,6 +20,7 @@ package org.apache.doris.catalog;
import org.apache.doris.analysis.AccessTestUtil;
import org.apache.doris.analysis.Analyzer;
import org.apache.doris.analysis.CreateResourceStmt;
+import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.UserException;
@@ -201,4 +202,18 @@ public class JdbcResourceTest {
// Ensure the result URL still contains ';'
Assert.assertTrue(resultUrl.contains(";"));
}
+
+ @Test
+ public void testJdbcDriverPtah() {
+ String driverPath = "postgresql-42.5.0.jar";
+ Config.jdbc_driver_secure_path = "";
+ String fullPath = JdbcResource.getFullDriverUrl(driverPath);
+ Assert.assertEquals(fullPath, "file://" + Config.jdbc_drivers_dir + "/" + driverPath);
+ Config.jdbc_driver_secure_path = "file:///jdbc/;http://jdbc";
+ String driverPath2 = "file:///postgresql-42.5.0.jar";
+ Exception exception = Assert.assertThrows(IllegalArgumentException.class, () -> {
+ JdbcResource.getFullDriverUrl(driverPath2);
+ });
+ Assert.assertEquals("Driver URL does not match any allowed paths: file:///postgresql-42.5.0.jar", exception.getMessage());
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 02/19: [fix](memory) Fix jemalloc lib name for thirdparty arrow (#28843)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 99cdbc7e12a7379dcaca559791a10304cbbb18ea
Author: Xinyi Zou <zo...@gmail.com>
AuthorDate: Tue Feb 20 16:41:55 2024 +0800
[fix](memory) Fix jemalloc lib name for thirdparty arrow (#28843)
after #28429
---
be/cmake/thirdparty.cmake | 2 +-
thirdparty/build-thirdparty.sh | 2 --
2 files changed, 1 insertion(+), 3 deletions(-)
diff --git a/be/cmake/thirdparty.cmake b/be/cmake/thirdparty.cmake
index 3dca9252faa..0d485b2466e 100644
--- a/be/cmake/thirdparty.cmake
+++ b/be/cmake/thirdparty.cmake
@@ -78,7 +78,7 @@ add_thirdparty(crypto)
add_thirdparty(openssl LIBNAME "lib/libssl.a")
add_thirdparty(leveldb)
add_thirdparty(jemalloc LIBNAME "lib/libjemalloc_doris.a")
-add_thirdparty(jemalloc_arrow LIBNAME "lib/libjemalloc.a")
+add_thirdparty(jemalloc_arrow LIBNAME "lib/libjemalloc_arrow.a")
if (WITH_MYSQL)
add_thirdparty(mysql LIBNAME "lib/libmysqlclient.a")
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index 68e29eac154..242667cbd10 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -1074,8 +1074,6 @@ build_arrow() {
strip_lib libarrow.a
strip_lib libjemalloc_arrow.a
strip_lib libparquet.a
-
- cp -rf "${TP_INSTALL_DIR}/lib64/libjemalloc_arrow.a" "${TP_INSTALL_DIR}/lib64/libjemalloc.a" # TODO delete
}
# abseil
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 13/19: [fix](nereids)push more than one runtime filters into cte (#30901)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit cd7230885f213908999a26f695ead9f0bb67079a
Author: minghong <en...@gmail.com>
AuthorDate: Wed Feb 21 09:55:30 2024 +0800
[fix](nereids)push more than one runtime filters into cte (#30901)
* push rf into cte, used by tpcds95
---
.../glue/translator/PhysicalPlanTranslator.java | 6 +--
.../glue/translator/RuntimeFilterTranslator.java | 7 ---
.../processor/post/RuntimeFilterContext.java | 41 ++++++++--------
.../processor/post/RuntimeFilterGenerator.java | 57 ++++++++++++----------
.../nereids/postprocess/RuntimeFilterTest.java | 2 +-
.../noStatsRfPrune/query95.out | 2 +-
.../no_stats_shape/query95.out | 2 +-
7 files changed, 58 insertions(+), 59 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
index e1571778a15..c9cb534c9c2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java
@@ -1284,8 +1284,7 @@ public class PhysicalPlanTranslator extends DefaultPlanVisitor<PlanFragment, Pla
.forEach(s -> rightChildOutputMap.put(s.getExprId(), s));
// translate runtime filter
- context.getRuntimeTranslator().ifPresent(runtimeFilterTranslator -> runtimeFilterTranslator
- .getRuntimeFilterOfHashJoinNode(physicalHashJoin)
+ context.getRuntimeTranslator().ifPresent(runtimeFilterTranslator -> physicalHashJoin.getRuntimeFilters()
.forEach(filter -> runtimeFilterTranslator.createLegacyRuntimeFilter(filter, hashJoinNode, context)));
// make intermediate tuple
@@ -1484,8 +1483,7 @@ public class PhysicalPlanTranslator extends DefaultPlanVisitor<PlanFragment, Pla
}
// translate runtime filter
context.getRuntimeTranslator().ifPresent(runtimeFilterTranslator -> {
- Set<RuntimeFilter> filters = runtimeFilterTranslator
- .getRuntimeFilterOfHashJoinNode(nestedLoopJoin);
+ List<RuntimeFilter> filters = nestedLoopJoin.getRuntimeFilters();
filters.forEach(filter -> runtimeFilterTranslator
.createLegacyRuntimeFilter(filter, nestedLoopJoinNode, context));
if (filters.stream().anyMatch(filter -> filter.getType() == TRuntimeFilterType.BITMAP)) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java
index a2fbc28ecd8..787a2bd8181 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/RuntimeFilterTranslator.java
@@ -28,7 +28,6 @@ import org.apache.doris.nereids.trees.expressions.ExprId;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.SlotReference;
-import org.apache.doris.nereids.trees.plans.physical.AbstractPhysicalJoin;
import org.apache.doris.nereids.trees.plans.physical.RuntimeFilter;
import org.apache.doris.planner.CTEScanNode;
import org.apache.doris.planner.DataStreamSink;
@@ -47,7 +46,6 @@ import com.google.common.collect.ImmutableMap;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
-import java.util.Set;
/**
* translate runtime filter
@@ -58,11 +56,6 @@ public class RuntimeFilterTranslator {
public RuntimeFilterTranslator(RuntimeFilterContext context) {
this.context = context;
- context.generatePhysicalHashJoinToRuntimeFilter();
- }
-
- public Set<RuntimeFilter> getRuntimeFilterOfHashJoinNode(AbstractPhysicalJoin join) {
- return context.getRuntimeFilterOnHashJoinNode(join);
}
public RuntimeFilterContext getContext() {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterContext.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterContext.java
index eb5767fc1f6..bf137e3c580 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterContext.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterContext.java
@@ -110,8 +110,6 @@ public class RuntimeFilterContext {
// exprId to olap scan node slotRef because the slotRef will be changed when translating.
private final Map<ExprId, SlotRef> exprIdToOlapScanNodeSlotRef = Maps.newHashMap();
- private final Map<AbstractPhysicalJoin, Set<RuntimeFilter>> runtimeFilterOnHashJoinNode = Maps.newHashMap();
-
// alias -> alias's child, if there's a key that is alias's child, the key-value will change by this way
// Alias(A) = B, now B -> A in map, and encounter Alias(B) -> C, the kv will be C -> A.
// you can see disjoint set data structure to learn the processing detailed.
@@ -191,19 +189,31 @@ public class RuntimeFilterContext {
public void removeFilter(ExprId targetId, PhysicalHashJoin builderNode) {
List<RuntimeFilter> filters = targetExprIdToFilter.get(targetId);
if (filters != null) {
- Iterator<RuntimeFilter> iter = filters.iterator();
- while (iter.hasNext()) {
- RuntimeFilter rf = iter.next();
+ Iterator<RuntimeFilter> filterIter = filters.iterator();
+ while (filterIter.hasNext()) {
+ RuntimeFilter rf = filterIter.next();
if (rf.getBuilderNode().equals(builderNode)) {
- builderNode.getRuntimeFilters().remove(rf);
- for (int i = 0; i < rf.getTargetSlots().size(); i++) {
- Slot targetSlot = rf.getTargetSlots().get(i);
+ Iterator<Slot> targetSlotIter = rf.getTargetSlots().listIterator();
+ Iterator<PhysicalRelation> targetScanIter = rf.getTargetScans().iterator();
+ Iterator<Expression> targetExpressionIter = rf.getTargetExpressions().iterator();
+ Slot targetSlot;
+ PhysicalRelation targetScan;
+ while (targetScanIter.hasNext() && targetSlotIter.hasNext() && targetExpressionIter.hasNext()) {
+ targetExpressionIter.next();
+ targetScan = targetScanIter.next();
+ targetSlot = targetSlotIter.next();
if (targetSlot.getExprId().equals(targetId)) {
- rf.getTargetScans().get(i).removeAppliedRuntimeFilter(rf);
+ targetScan.removeAppliedRuntimeFilter(rf);
+ targetExpressionIter.remove();
+ targetScanIter.remove();
+ targetSlotIter.remove();
}
}
- iter.remove();
- prunedRF.add(rf);
+ if (rf.getTargetSlots().isEmpty()) {
+ builderNode.getRuntimeFilters().remove(rf);
+ filterIter.remove();
+ prunedRF.add(rf);
+ }
}
}
}
@@ -255,15 +265,6 @@ public class RuntimeFilterContext {
return scanNodeOfLegacyRuntimeFilterTarget;
}
- public Set<RuntimeFilter> getRuntimeFilterOnHashJoinNode(AbstractPhysicalJoin join) {
- return runtimeFilterOnHashJoinNode.getOrDefault(join, Collections.emptySet());
- }
-
- public void generatePhysicalHashJoinToRuntimeFilter() {
- targetExprIdToFilter.values().forEach(filters -> filters.forEach(filter -> runtimeFilterOnHashJoinNode
- .computeIfAbsent(filter.getBuilderNode(), k -> Sets.newHashSet()).add(filter)));
- }
-
public Map<ExprId, List<RuntimeFilter>> getTargetExprIdToFilter() {
return targetExprIdToFilter;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
index d4c5a96d3c5..5a7fbecb6ec 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/processor/post/RuntimeFilterGenerator.java
@@ -158,7 +158,7 @@ public class RuntimeFilterGenerator extends PlanPostProcessor {
// the most right deep buildNode from rfsToPushDown is used as buildNode for pushDown rf
// since the srcExpr are the same, all buildNodes of rfToPushDown are in the same tree path
// the longest ancestors means its corresponding rf build node is the most right deep one.
- RuntimeFilter rightDeep = rfsToPushDown.get(0);
+ List<RuntimeFilter> rightDeepRfs = Lists.newArrayList();
List<Plan> rightDeepAncestors = rfsToPushDown.get(0).getBuilderNode().getAncestors();
int rightDeepAncestorsSize = rightDeepAncestors.size();
RuntimeFilter leftTop = rfsToPushDown.get(0);
@@ -166,10 +166,15 @@ public class RuntimeFilterGenerator extends PlanPostProcessor {
for (RuntimeFilter rf : rfsToPushDown) {
List<Plan> ancestors = rf.getBuilderNode().getAncestors();
int currentAncestorsSize = ancestors.size();
- if (currentAncestorsSize > rightDeepAncestorsSize) {
- rightDeep = rf;
- rightDeepAncestorsSize = currentAncestorsSize;
- rightDeepAncestors = ancestors;
+ if (currentAncestorsSize >= rightDeepAncestorsSize) {
+ if (currentAncestorsSize == rightDeepAncestorsSize) {
+ rightDeepRfs.add(rf);
+ } else {
+ rightDeepAncestorsSize = currentAncestorsSize;
+ rightDeepAncestors = ancestors;
+ rightDeepRfs.clear();
+ rightDeepRfs.add(rf);
+ }
}
if (currentAncestorsSize < leftTopAncestorsSize) {
leftTopAncestorsSize = currentAncestorsSize;
@@ -187,7 +192,7 @@ public class RuntimeFilterGenerator extends PlanPostProcessor {
if (cursor instanceof AbstractPhysicalJoin) {
AbstractPhysicalJoin cursorJoin = (AbstractPhysicalJoin) cursor;
valid = (!RuntimeFilterGenerator.DENIED_JOIN_TYPES
- .contains(cursorJoin.getJoinType())
+ .contains(cursorJoin.getJoinType())
|| cursorJoin.isMarkJoin()) && valid;
}
if (!valid) {
@@ -199,27 +204,29 @@ public class RuntimeFilterGenerator extends PlanPostProcessor {
break;
}
- Expression rightDeepTargetExpressionOnCTE = null;
- int targetCount = rightDeep.getTargetExpressions().size();
- for (int i = 0; i < targetCount; i++) {
- PhysicalRelation rel = rightDeep.getTargetScans().get(i);
- if (rel instanceof PhysicalCTEConsumer
- && ((PhysicalCTEConsumer) rel).getCteId().equals(cteId)) {
- rightDeepTargetExpressionOnCTE = rightDeep.getTargetExpressions().get(i);
- break;
+ for (RuntimeFilter rfToPush : rightDeepRfs) {
+ Expression rightDeepTargetExpressionOnCTE = null;
+ int targetCount = rfToPush.getTargetExpressions().size();
+ for (int i = 0; i < targetCount; i++) {
+ PhysicalRelation rel = rfToPush.getTargetScans().get(i);
+ if (rel instanceof PhysicalCTEConsumer
+ && ((PhysicalCTEConsumer) rel).getCteId().equals(cteId)) {
+ rightDeepTargetExpressionOnCTE = rfToPush.getTargetExpressions().get(i);
+ break;
+ }
}
- }
- boolean pushedDown = doPushDownIntoCTEProducerInternal(
- rightDeep,
- rightDeepTargetExpressionOnCTE,
- rfCtx,
- rfCtx.getCteProduceMap().get(cteId)
- );
- if (pushedDown) {
- rfCtx.removeFilter(
- rightDeepTargetExpressionOnCTE.getInputSlotExprIds().iterator().next(),
- (PhysicalHashJoin) rightDeep.getBuilderNode());
+ boolean pushedDown = doPushDownIntoCTEProducerInternal(
+ rfToPush,
+ rightDeepTargetExpressionOnCTE,
+ rfCtx,
+ rfCtx.getCteProduceMap().get(cteId)
+ );
+ if (pushedDown) {
+ rfCtx.removeFilter(
+ rightDeepTargetExpressionOnCTE.getInputSlotExprIds().iterator().next(),
+ (PhysicalHashJoin) rfToPush.getBuilderNode());
+ }
}
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/RuntimeFilterTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/RuntimeFilterTest.java
index ce647427d21..f354ed5b02f 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/RuntimeFilterTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/postprocess/RuntimeFilterTest.java
@@ -295,7 +295,7 @@ public class RuntimeFilterTest extends SSBTestBase {
.rewrite()
.implement();
PhysicalPlan plan = checker.getPhysicalPlan();
- new PlanPostProcessors(checker.getCascadesContext()).process(plan);
+ plan = new PlanPostProcessors(checker.getCascadesContext()).process(plan);
System.out.println(plan.treeString());
new PhysicalPlanTranslator(new PlanTranslatorContext(checker.getCascadesContext())).translatePlan(plan);
RuntimeFilterContext context = checker.getCascadesContext().getRuntimeFilterContext();
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query95.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query95.out
index 6e12e33b54d..82a42a4795c 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query95.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/noStatsRfPrune/query95.out
@@ -26,7 +26,7 @@ PhysicalCteAnchor ( cteId=CTEId#0 )
----------------------hashJoin[INNER_JOIN] hashCondition=((ws1.ws_web_site_sk = web_site.web_site_sk)) otherCondition=() build RFs:RF5 web_site_sk->[ws_web_site_sk]
------------------------hashJoin[INNER_JOIN] hashCondition=((ws1.ws_ship_addr_sk = customer_address.ca_address_sk)) otherCondition=() build RFs:RF4 ca_address_sk->[ws_ship_addr_sk]
--------------------------hashJoin[INNER_JOIN] hashCondition=((ws1.ws_ship_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF3 d_date_sk->[ws_ship_date_sk]
-----------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((ws1.ws_order_number = web_returns.wr_order_number)) otherCondition=() build RFs:RF7 ws_order_number->[ws_order_number,ws_order_number]
+----------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((ws1.ws_order_number = web_returns.wr_order_number)) otherCondition=() build RFs:RF2 ws_order_number->[wr_order_number];RF7 ws_order_number->[ws_order_number,ws_order_number]
------------------------------PhysicalProject
--------------------------------hashJoin[INNER_JOIN] hashCondition=((web_returns.wr_order_number = ws_wh.ws_order_number)) otherCondition=()
----------------------------------PhysicalDistribute[DistributionSpecHash]
diff --git a/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query95.out b/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query95.out
index 262c9046490..bf426f81483 100644
--- a/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query95.out
+++ b/regression-test/data/nereids_tpcds_shape_sf100_p0/no_stats_shape/query95.out
@@ -26,7 +26,7 @@ PhysicalCteAnchor ( cteId=CTEId#0 )
----------------------hashJoin[INNER_JOIN] hashCondition=((ws1.ws_web_site_sk = web_site.web_site_sk)) otherCondition=() build RFs:RF5 web_site_sk->[ws_web_site_sk]
------------------------hashJoin[INNER_JOIN] hashCondition=((ws1.ws_ship_addr_sk = customer_address.ca_address_sk)) otherCondition=() build RFs:RF4 ca_address_sk->[ws_ship_addr_sk]
--------------------------hashJoin[INNER_JOIN] hashCondition=((ws1.ws_ship_date_sk = date_dim.d_date_sk)) otherCondition=() build RFs:RF3 d_date_sk->[ws_ship_date_sk]
-----------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((ws1.ws_order_number = web_returns.wr_order_number)) otherCondition=() build RFs:RF7 ws_order_number->[ws_order_number,ws_order_number]
+----------------------------hashJoin[RIGHT_SEMI_JOIN] hashCondition=((ws1.ws_order_number = web_returns.wr_order_number)) otherCondition=() build RFs:RF2 ws_order_number->[wr_order_number];RF7 ws_order_number->[ws_order_number,ws_order_number]
------------------------------PhysicalProject
--------------------------------hashJoin[INNER_JOIN] hashCondition=((web_returns.wr_order_number = ws_wh.ws_order_number)) otherCondition=() build RFs:RF1 wr_order_number->[ws_order_number]
----------------------------------PhysicalDistribute[DistributionSpecHash]
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 16/19: [regression-test](point query) make test stable (#31140)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 728b12cfdf5eb20903f136242256e22ff3885cb7
Author: lihangyu <15...@163.com>
AuthorDate: Wed Feb 21 10:22:39 2024 +0800
[regression-test](point query) make test stable (#31140)
---
.../suites/point_query_p0/test_point_query.groovy | 18 ++++++++++--------
.../test_point_query_cluster_key.groovy | 19 ++++++++++---------
2 files changed, 20 insertions(+), 17 deletions(-)
diff --git a/regression-test/suites/point_query_p0/test_point_query.groovy b/regression-test/suites/point_query_p0/test_point_query.groovy
index e919bb7a026..f7df5ab820f 100644
--- a/regression-test/suites/point_query_p0/test_point_query.groovy
+++ b/regression-test/suites/point_query_p0/test_point_query.groovy
@@ -18,15 +18,14 @@
import java.math.BigDecimal;
suite("test_point_query") {
+ def backendId_to_backendIP = [:]
+ def backendId_to_backendHttpPort = [:]
+ getBackendIpHttpPort(backendId_to_backendIP, backendId_to_backendHttpPort);
def set_be_config = { key, value ->
- String backend_id;
- def backendId_to_backendIP = [:]
- def backendId_to_backendHttpPort = [:]
- getBackendIpHttpPort(backendId_to_backendIP, backendId_to_backendHttpPort);
-
- backend_id = backendId_to_backendIP.keySet()[0]
- def (code, out, err) = update_be_config(backendId_to_backendIP.get(backend_id), backendId_to_backendHttpPort.get(backend_id), key, value)
- logger.info("update config: code=" + code + ", out=" + out + ", err=" + err)
+ for (String backend_id: backendId_to_backendIP.keySet()) {
+ def (code, out, err) = update_be_config(backendId_to_backendIP.get(backend_id), backendId_to_backendHttpPort.get(backend_id), key, value)
+ logger.info("update config: code=" + code + ", out=" + out + ", err=" + err)
+ }
}
try {
set_be_config.call("disable_storage_row_cache", "false")
@@ -65,6 +64,8 @@ suite("test_point_query") {
def nprep_sql = { sql_str ->
def url_without_prep = "jdbc:mysql://" + sql_ip + ":" + sql_port + "/" + realDb
connect(user = user, password = password, url = url_without_prep) {
+ // set to false to invalid cache correcly
+ sql "set enable_memtable_on_sink_node = false"
sql sql_str
}
}
@@ -198,6 +199,7 @@ suite("test_point_query") {
qe_point_select stmt
qe_point_select stmt
// invalidate cache
+ sql "sync"
nprep_sql """ INSERT INTO ${tableName} VALUES(1235, 120939.11130, "a ddd", "xxxxxx", "2030-01-02", "2020-01-01 12:36:38", 22.822, "7022-01-01 11:30:38", 0, 1929111.1111,[119291.19291], ["111", "222", "333"], 2) """
qe_point_select stmt
qe_point_select stmt
diff --git a/regression-test/suites/point_query_p0/test_point_query_cluster_key.groovy b/regression-test/suites/point_query_p0/test_point_query_cluster_key.groovy
index 700f19b53d1..a117199297d 100644
--- a/regression-test/suites/point_query_p0/test_point_query_cluster_key.groovy
+++ b/regression-test/suites/point_query_p0/test_point_query_cluster_key.groovy
@@ -18,16 +18,15 @@
import java.math.BigDecimal;
suite("test_point_query_cluster_key") {
+ def backendId_to_backendIP = [:]
+ def backendId_to_backendHttpPort = [:]
+ getBackendIpHttpPort(backendId_to_backendIP, backendId_to_backendHttpPort);
def set_be_config = { key, value ->
- String backend_id;
- def backendId_to_backendIP = [:]
- def backendId_to_backendHttpPort = [:]
- getBackendIpHttpPort(backendId_to_backendIP, backendId_to_backendHttpPort);
-
- backend_id = backendId_to_backendIP.keySet()[0]
- def (code, out, err) = update_be_config(backendId_to_backendIP.get(backend_id), backendId_to_backendHttpPort.get(backend_id), key, value)
- logger.info("update config: code=" + code + ", out=" + out + ", err=" + err)
- }
+ for (String backend_id: backendId_to_backendIP.keySet()) {
+ def (code, out, err) = update_be_config(backendId_to_backendIP.get(backend_id), backendId_to_backendHttpPort.get(backend_id), key, value)
+ logger.info("update config: code=" + code + ", out=" + out + ", err=" + err)
+ }
+ }
try {
set_be_config.call("disable_storage_row_cache", "false")
// nereids do not support point query now
@@ -65,6 +64,8 @@ suite("test_point_query_cluster_key") {
def nprep_sql = { sql_str ->
def url_without_prep = "jdbc:mysql://" + sql_ip + ":" + sql_port + "/" + realDb
connect(user = user, password = password, url = url_without_prep) {
+ // set to false to invalid cache correcly
+ sql "set enable_memtable_on_sink_node = false"
sql sql_str
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 05/19: Fix load stats cache FE ut case. (#31162)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e2514c950c68092595a369f151cbff77878d4688
Author: Jibing-Li <64...@users.noreply.github.com>
AuthorDate: Tue Feb 20 21:20:42 2024 +0800
Fix load stats cache FE ut case. (#31162)
---
.../org/apache/doris/statistics/CacheTest.java | 28 +++++++++++++++-------
1 file changed, 19 insertions(+), 9 deletions(-)
diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/CacheTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/CacheTest.java
index a30a8f2d3fd..00d233ff54b 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/CacheTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/CacheTest.java
@@ -249,15 +249,25 @@ public class CacheTest extends TestWithFeService {
try {
StatisticsCache statisticsCache = new StatisticsCache();
ColumnStatistic columnStatistic = statisticsCache.getColumnStatistics(1, 1, 1, -1, "col");
- Thread.sleep(3000);
- columnStatistic = statisticsCache.getColumnStatistics(1, 1, 1, -1, "col");
- Assertions.assertEquals(1, columnStatistic.count);
- Assertions.assertEquals(2, columnStatistic.ndv);
- Assertions.assertEquals(3, columnStatistic.avgSizeByte);
- Assertions.assertEquals(4, columnStatistic.numNulls);
- Assertions.assertEquals(5, columnStatistic.dataSize);
- Assertions.assertEquals(6, columnStatistic.minValue);
- Assertions.assertEquals(7, columnStatistic.maxValue);
+ for (int i = 0; i < 15; i++) {
+ columnStatistic = statisticsCache.getColumnStatistics(1, 1, 1, -1, "col");
+ if (columnStatistic != ColumnStatistic.UNKNOWN) {
+ break;
+ }
+ System.out.println("Not ready yet.");
+ Thread.sleep(1000);
+ }
+ if (columnStatistic != ColumnStatistic.UNKNOWN) {
+ Assertions.assertEquals(1, columnStatistic.count);
+ Assertions.assertEquals(2, columnStatistic.ndv);
+ Assertions.assertEquals(3, columnStatistic.avgSizeByte);
+ Assertions.assertEquals(4, columnStatistic.numNulls);
+ Assertions.assertEquals(5, columnStatistic.dataSize);
+ Assertions.assertEquals(6, columnStatistic.minValue);
+ Assertions.assertEquals(7, columnStatistic.maxValue);
+ } else {
+ System.out.println("Cached is not loaded, skip test.");
+ }
} catch (Throwable t) {
t.printStackTrace();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 17/19: [fix](invert index) fix the inaccurate rows_inverted_index_filtered in the profile (#31158)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f121ee907f6c93b266f3298e32d8e4878c3e00fa
Author: zzzxl <33...@users.noreply.github.com>
AuthorDate: Wed Feb 21 10:42:38 2024 +0800
[fix](invert index) fix the inaccurate rows_inverted_index_filtered in the profile (#31158)
---
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 2 ++
1 file changed, 2 insertions(+)
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index 3eab4c0dca1..98e1fe0d82e 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -490,6 +490,7 @@ Status SegmentIterator::_get_row_ranges_by_column_conditions() {
}
if (config::enable_index_apply_preds_except_leafnode_of_andnode) {
+ size_t input_rows = _row_bitmap.cardinality();
RETURN_IF_ERROR(_apply_index_except_leafnode_of_andnode());
if (_can_filter_by_preds_except_leafnode_of_andnode()) {
for (auto it = _remaining_conjunct_roots.begin();
@@ -509,6 +510,7 @@ Status SegmentIterator::_get_row_ranges_by_column_conditions() {
}
}
}
+ _opts.stats->rows_inverted_index_filtered += (input_rows - _row_bitmap.cardinality());
}
RETURN_IF_ERROR(_apply_bitmap_index());
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 11/19: [fix](group_commit) Need to wait wal to be deleted when creating MaterializedView (#30956)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1e3968fe7eeed7a246e7cc0185b4034023fa789d
Author: huanghaibin <28...@qq.com>
AuthorDate: Wed Feb 21 08:58:31 2024 +0800
[fix](group_commit) Need to wait wal to be deleted when creating MaterializedView (#30956)
---
be/src/olap/wal/wal_manager.cpp | 2 +-
.../doris/alter/MaterializedViewHandler.java | 10 ++
.../org/apache/doris/alter/SchemaChangeJobV2.java | 33 +-----
.../org/apache/doris/load/GroupCommitManager.java | 28 +++++
.../data/insert_p0/insert_group_commit_into.out | 18 +++
.../insert_p0/insert_group_commit_into.groovy | 126 +++++++++++++++++++++
6 files changed, 186 insertions(+), 31 deletions(-)
diff --git a/be/src/olap/wal/wal_manager.cpp b/be/src/olap/wal/wal_manager.cpp
index 06e1bd37d72..10a979f89dd 100644
--- a/be/src/olap/wal/wal_manager.cpp
+++ b/be/src/olap/wal/wal_manager.cpp
@@ -289,6 +289,7 @@ Status WalManager::_load_wals() {
continue;
}
}
+ _exec_env->wal_mgr()->add_wal_queue(wal.tb_id, wal.wal_id);
WARN_IF_ERROR(add_recover_wal(wal.db_id, wal.tb_id, wal.wal_id, wal.wal_path),
fmt::format("Failed to add recover wal={}", wal.wal_path));
}
@@ -406,7 +407,6 @@ Status WalManager::_replay_background() {
Status WalManager::add_recover_wal(int64_t db_id, int64_t table_id, int64_t wal_id,
std::string wal) {
- add_wal_queue(table_id, wal_id);
std::lock_guard<std::shared_mutex> wrlock(_table_lock);
std::shared_ptr<WalTable> table_ptr;
auto it = _table_map.find(table_id);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/MaterializedViewHandler.java b/fe/fe-core/src/main/java/org/apache/doris/alter/MaterializedViewHandler.java
index c5e7f33f99d..f5d6d944cc0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/MaterializedViewHandler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/MaterializedViewHandler.java
@@ -218,6 +218,10 @@ public class MaterializedViewHandler extends AlterHandler {
olapTable.setState(OlapTableState.ROLLUP);
+ // wait wal delete
+ Env.getCurrentEnv().getGroupCommitManager().blockTable(olapTable.getId());
+ Env.getCurrentEnv().getGroupCommitManager().waitWalFinished(olapTable.getId());
+
Env.getCurrentEnv().getEditLog().logAlterJob(rollupJobV2);
LOG.info("finished to create materialized view job: {}", rollupJobV2.getJobId());
} finally {
@@ -300,6 +304,11 @@ public class MaterializedViewHandler extends AlterHandler {
// ATTN: This order is not mandatory, because database lock will protect us,
// but this order is more reasonable
olapTable.setState(OlapTableState.ROLLUP);
+
+ // wait wal delete
+ Env.getCurrentEnv().getGroupCommitManager().blockTable(olapTable.getId());
+ Env.getCurrentEnv().getGroupCommitManager().waitWalFinished(olapTable.getId());
+
// 2 batch submit rollup job
List<AlterJobV2> rollupJobV2List = new ArrayList<>(rollupNameJobMap.values());
batchAddAlterJobV2(rollupJobV2List);
@@ -1169,6 +1178,7 @@ public class MaterializedViewHandler extends AlterHandler {
private void onJobDone(AlterJobV2 alterJob) {
removeJobFromRunningQueue(alterJob);
if (removeAlterJobV2FromTableNotFinalStateJobMap(alterJob)) {
+ Env.getCurrentEnv().getGroupCommitManager().unblockTable(alterJob.getTableId());
changeTableStatus(alterJob.getDbId(), alterJob.getTableId(), OlapTableState.NORMAL);
LOG.info("set table's state to NORMAL, table id: {}, job id: {}", alterJob.getTableId(),
alterJob.getJobId());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
index 200532c6d4b..4a8588a0ff2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
@@ -40,7 +40,6 @@ import org.apache.doris.catalog.Tablet;
import org.apache.doris.catalog.TabletInvertedIndex;
import org.apache.doris.catalog.TabletMeta;
import org.apache.doris.common.AnalysisException;
-import org.apache.doris.common.Config;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.MarkedCountDownLatch;
import org.apache.doris.common.MetaNotFoundException;
@@ -537,7 +536,9 @@ public class SchemaChangeJobV2 extends AlterJobV2 {
}
return;
}
- waitWalFinished();
+ Env.getCurrentEnv().getGroupCommitManager().blockTable(tableId);
+ Env.getCurrentEnv().getGroupCommitManager().waitWalFinished(tableId);
+ Env.getCurrentEnv().getGroupCommitManager().unblockTable(tableId);
/*
* all tasks are finished. check the integrity.
* we just check whether all new replicas are healthy.
@@ -600,34 +601,6 @@ public class SchemaChangeJobV2 extends AlterJobV2 {
LOG.info("set table's state to NORMAL, table id: {}, job id: {}", tableId, jobId);
}
- private void waitWalFinished() {
- // wait wal done here
- Env.getCurrentEnv().getGroupCommitManager().blockTable(tableId);
- LOG.info("block group commit for table={} when schema change", tableId);
- List<Long> aliveBeIds = Env.getCurrentSystemInfo().getAllBackendIds(true);
- long expireTime = System.currentTimeMillis() + Config.check_wal_queue_timeout_threshold;
- while (true) {
- LOG.info("wait for wal queue size to be empty");
- boolean walFinished = Env.getCurrentEnv().getGroupCommitManager()
- .isPreviousWalFinished(tableId, aliveBeIds);
- if (walFinished) {
- LOG.info("all wal is finished for table={}", tableId);
- break;
- } else if (System.currentTimeMillis() > expireTime) {
- LOG.warn("waitWalFinished time out for table={}", tableId);
- break;
- } else {
- try {
- Thread.sleep(100);
- } catch (InterruptedException ie) {
- LOG.warn("failed to wait for wal for table={} when schema change", tableId, ie);
- }
- }
- }
- Env.getCurrentEnv().getGroupCommitManager().unblockTable(tableId);
- LOG.info("unblock group commit for table={} when schema change", tableId);
- }
-
private void onFinished(OlapTable tbl) {
// replace the origin index with shadow index, set index state as NORMAL
for (Partition partition : tbl.getPartitions()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/GroupCommitManager.java b/fe/fe-core/src/main/java/org/apache/doris/load/GroupCommitManager.java
index 12410945e9f..c4bf1e03c9c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/GroupCommitManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/GroupCommitManager.java
@@ -45,11 +45,39 @@ public class GroupCommitManager {
}
public void blockTable(long tableId) {
+ LOG.info("block group commit for table={} when schema change", tableId);
blockedTableIds.add(tableId);
}
public void unblockTable(long tableId) {
blockedTableIds.remove(tableId);
+ LOG.info("unblock group commit for table={} when schema change", tableId);
+ }
+
+ /**
+ * Waiting All WAL files to be deleted.
+ */
+ public void waitWalFinished(long tableId) {
+ List<Long> aliveBeIds = Env.getCurrentSystemInfo().getAllBackendIds(true);
+ long expireTime = System.currentTimeMillis() + Config.check_wal_queue_timeout_threshold;
+ while (true) {
+ LOG.info("wait for wal queue size to be empty");
+ boolean walFinished = Env.getCurrentEnv().getGroupCommitManager()
+ .isPreviousWalFinished(tableId, aliveBeIds);
+ if (walFinished) {
+ LOG.info("all wal is finished for table={}", tableId);
+ break;
+ } else if (System.currentTimeMillis() > expireTime) {
+ LOG.warn("waitWalFinished time out for table={}", tableId);
+ break;
+ } else {
+ try {
+ Thread.sleep(100);
+ } catch (InterruptedException ie) {
+ LOG.warn("failed to wait for wal for table={} when schema change", tableId, ie);
+ }
+ }
+ }
}
/**
diff --git a/regression-test/data/insert_p0/insert_group_commit_into.out b/regression-test/data/insert_p0/insert_group_commit_into.out
index dfb3a3b41c0..71a1473e1e6 100644
--- a/regression-test/data/insert_p0/insert_group_commit_into.out
+++ b/regression-test/data/insert_p0/insert_group_commit_into.out
@@ -94,6 +94,15 @@ q 50
-- !sql --
0 service_46da0dab-e27d-4820-aea2-9bfc15741615 1697032066304 0 3229b7cd-f3a2-4359-aa24-946388c9cc54 0 CgEwEiQzMjI5YjdjZC1mM2EyLTQzNTktYWEyNC05NDYzODhjOWNjNTQaggQY/6n597ExIP+p+fexMWIWCgh0YWdLZXlfMBIKdGFnVmFsdWVfMGIWCgh0YWdLZXlfMRIKdGFnVmFsdWVfMWIWCgh0YWdLZXlfMhIKdGFnVmFsdWVfMmIWCgh0YWdLZXlfMxIKdGFnVmFsdWVfM2IWCgh0YWdLZXlfNBIKdGFnVmFsdWVfNGIWCgh0YWdLZXlfNRIKdGFnVmFsdWVfNWIWCgh0YWdLZXlfNhIKdGFnVmFsdWVfNmIWCgh0YWdLZXlfNxIKdGFnVmFsdWVfN2IWCgh0YWdLZXlfOBIKdGFnVmFsdWVfOGIWCgh0YWdLZXlfORIKdGFnVm [...]
+-- !order --
+2023-06-10 cib2205045_1_1s 0.0000 168939.0 0.0000 0.0 0.0000 0.0 0.0000 day
+
+-- !order2 --
+1 1 1
+2 2 2
+3 3 3
+4 4 4
+
-- !sql --
1 a 10
2 b -1
@@ -189,3 +198,12 @@ q 50
-- !sql --
0 service_46da0dab-e27d-4820-aea2-9bfc15741615 1697032066304 0 3229b7cd-f3a2-4359-aa24-946388c9cc54 0 CgEwEiQzMjI5YjdjZC1mM2EyLTQzNTktYWEyNC05NDYzODhjOWNjNTQaggQY/6n597ExIP+p+fexMWIWCgh0YWdLZXlfMBIKdGFnVmFsdWVfMGIWCgh0YWdLZXlfMRIKdGFnVmFsdWVfMWIWCgh0YWdLZXlfMhIKdGFnVmFsdWVfMmIWCgh0YWdLZXlfMxIKdGFnVmFsdWVfM2IWCgh0YWdLZXlfNBIKdGFnVmFsdWVfNGIWCgh0YWdLZXlfNRIKdGFnVmFsdWVfNWIWCgh0YWdLZXlfNhIKdGFnVmFsdWVfNmIWCgh0YWdLZXlfNxIKdGFnVmFsdWVfN2IWCgh0YWdLZXlfOBIKdGFnVmFsdWVfOGIWCgh0YWdLZXlfORIKdGFnVm [...]
+-- !order --
+2023-06-10 cib2205045_1_1s 0.0000 168939.0 0.0000 0.0 0.0000 0.0 0.0000 day
+
+-- !order2 --
+1 1 1
+2 2 2
+3 3 3
+4 4 4
+
diff --git a/regression-test/suites/insert_p0/insert_group_commit_into.groovy b/regression-test/suites/insert_p0/insert_group_commit_into.groovy
index 391e106aa2f..b7c6cc4e3c4 100644
--- a/regression-test/suites/insert_p0/insert_group_commit_into.groovy
+++ b/regression-test/suites/insert_p0/insert_group_commit_into.groovy
@@ -318,5 +318,131 @@ suite("insert_group_commit_into") {
} finally {
// try_sql("DROP TABLE ${table}")
}
+
+ // table with MaterializedView
+ tableName = "insert_group_commit_into_mv"
+ table = dbName + "." + tableName
+ def table_tmp = dbName + ".test_table_tmp"
+ try {
+ // create table
+ sql """ drop table if exists ${table}; """
+ sql """CREATE table ${table} (
+ `ordernum` varchar(65533) NOT NULL ,
+ `dnt` datetime NOT NULL ,
+ `data` json NULL
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`ordernum`, `dnt`)
+ COMMENT 'OLAP'
+ DISTRIBUTED BY HASH(`ordernum`) BUCKETS 3
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ );"""
+ sql """drop table if exists ${table_tmp};"""
+ sql """CREATE TABLE ${table_tmp} (
+ `dnt` varchar(200) NULL,
+ `ordernum` varchar(200) NULL,
+ `type` varchar(20) NULL,
+ `powers` double SUM NULL,
+ `p0` double REPLACE NULL,
+ `heatj` double SUM NULL,
+ `j0` double REPLACE NULL,
+ `heatg` double SUM NULL,
+ `g0` double REPLACE NULL,
+ `solar` double SUM NULL
+ ) ENGINE=OLAP
+ AGGREGATE KEY(`dnt`, `ordernum`, `type`)
+ COMMENT 'OLAP'
+ DISTRIBUTED BY HASH(`ordernum`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1"
+ ); """
+ sql """DROP MATERIALIZED VIEW IF EXISTS ods_zn_dnt_max1 ON ${table};"""
+ sql """create materialized view ods_zn_dnt_max1 as
+ select ordernum,max(dnt) as dnt from ${table}
+ group by ordernum
+ ORDER BY ordernum;"""
+ connect(user = context.config.jdbcUser, password = context.config.jdbcPassword, url = context.config.jdbcUrl) {
+ sql """ set group_commit = async_mode; """
+ if (item == "nereids") {
+ sql """ set enable_nereids_dml = true; """
+ sql """ set enable_nereids_planner=true; """
+ //sql """ set enable_fallback_to_original_planner=false; """
+ } else {
+ sql """ set enable_nereids_dml = false; """
+ }
+
+ // 1. insert into
+ int count = 0;
+ while (count < 30) {
+ try {
+ group_commit_insert """
+ insert into ${table} values('cib2205045_1_1s','2023/6/10 3:55:33','{"DB1":168939,"DNT":"2023-06-10 03:55:33"}');""", 1
+ break
+ } catch (Exception e) {
+ logger.info("got exception:" + e)
+ if (e.getMessage().contains("is blocked on schema change")) {
+ Thread.sleep(1000)
+ }
+ count++
+ }
+ }
+ group_commit_insert """insert into ${table} values('cib2205045_1_1s','2023/6/10 3:56:33','{"DB1":168939,"DNT":"2023-06-10 03:56:33"}');""", 1
+ group_commit_insert """insert into ${table} values('cib2205045_1_1s','2023/6/10 3:57:33','{"DB1":168939,"DNT":"2023-06-10 03:57:33"}');""", 1
+ group_commit_insert """insert into ${table} values('cib2205045_1_1s','2023/6/10 3:58:33','{"DB1":168939,"DNT":"2023-06-10 03:58:33"}');""", 1
+
+ getRowCount(4)
+
+ qt_order """select
+ '2023-06-10',
+ tmp.ordernum,
+ cast(nvl(if(tmp.p0-tmp1.p0>0,tmp.p0-tmp1.p0,tmp.p0-tmp.p1),0) as decimal(10,4)),
+ nvl(tmp.p0,0),
+ cast(nvl(if(tmp.j0-tmp1.j0>0,tmp.j0-tmp1.j0,tmp.j0-tmp.j1)*277.78,0) as decimal(10,4)),
+ nvl(tmp.j0,0),
+ cast(nvl(if(tmp.g0-tmp1.g0>0,tmp.g0-tmp1.g0,tmp.g0-tmp.g1)*277.78,0) as decimal(10,4)),
+ nvl(tmp.g0,0),
+ cast(nvl(tmp.solar,0) as decimal(20,4)),
+ 'day'
+ from
+ (
+ select
+ ordernum,
+ max(ljrl1) g0,min(ljrl1) g1,
+ max(ljrl2) j0,min(ljrl2) j1,
+ max(db1) p0,min(db1) p1,
+ max(fzl)*1600*0.278 solar
+ from(
+ select ordernum,dnt,
+ cast(if(json_extract(data,'\$.LJRL1')=0 or json_extract(data,'\$.LJRL1') like '%E%',null,json_extract(data,'\$.LJRL1')) as double) ljrl1,
+ cast(if(json_extract(data,'\$.LJRL2')=0 or json_extract(data,'\$.LJRL2') like '%E%',null,json_extract(data,'\$.LJRL2')) as double) ljrl2,
+ first_value(cast(if(json_extract(data,'\$.FZL')=0 or json_extract(data,'\$.FZL') like '%E%',null,
+ json_extract(data,'\$.FZL')) as double)) over (partition by ordernum order by dnt desc) fzl,
+ cast(if(json_extract(data,'\$.DB1')=0 or json_extract(data,'\$.DB1') like '%E%',null,json_extract(data,'\$.DB1')) as double) db1
+ from ${table}
+ )a1
+ group by ordernum
+ )tmp left join (
+ select
+ ordernum,MAX(p0) p0,MAX(j0) j0,MAX(g0) g0
+ from ${table_tmp}
+ group by ordernum
+ )tmp1
+ on tmp.ordernum=tmp1.ordernum;"""
+ qt_order2 """
+ SELECT
+ row_number() over(partition by add_date order by pc_num desc)
+ ,row_number() over(partition by add_date order by vc_num desc)
+ ,row_number() over(partition by add_date order by vt_num desc)
+ FROM (
+ SELECT
+ cast(dnt as datev2) add_date
+ ,row_number() over(order by dnt) pc_num
+ ,row_number() over(order by dnt) vc_num
+ ,row_number() over(order by dnt) vt_num
+ FROM ${table}
+ ) t;"""
+ }
+ } finally {
+ }
}
}
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 18/19: [chore](third-party) Link protobuf with zlib statically (#31173)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit a4f9eec810c86c760d0169fdb3e8f564394f9af4
Author: Adonis Ling <ad...@gmail.com>
AuthorDate: Wed Feb 21 11:00:24 2024 +0800
[chore](third-party) Link protobuf with zlib statically (#31173)
We should link protobuf with libz.a explicitly, otherwise protoc may not be executable due to the missing libz.1.dylib.
---
.github/workflows/build-thirdparty.yml | 4 +++-
thirdparty/build-thirdparty.sh | 2 +-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/build-thirdparty.yml b/.github/workflows/build-thirdparty.yml
index 978bed3a513..cf3837185cc 100644
--- a/.github/workflows/build-thirdparty.yml
+++ b/.github/workflows/build-thirdparty.yml
@@ -166,10 +166,12 @@ jobs:
'llvm@16'
)
- brew install "${packages[@]}"
+ brew install "${packages[@]}" || true
- name: Build
run: |
+ export MACOSX_DEPLOYMENT_TARGET=12.0
+
cd thirdparty
./build-thirdparty.sh -j "$(nproc)"
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index 242667cbd10..61ff2a2a657 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -434,7 +434,7 @@ build_protobuf() {
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
-Dprotobuf_BUILD_SHARED_LIBS=OFF \
-Dprotobuf_BUILD_TESTS=OFF \
- -Dprotobuf_WITH_ZLIB_DEFAULT=ON \
+ -DZLIB_LIBRARY="${TP_LIB_DIR}/libz.a" \
-Dprotobuf_ABSL_PROVIDER=package \
-DCMAKE_INSTALL_PREFIX="${TP_INSTALL_DIR}" ../..
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 10/19: [fix](regression test) fix test_dynamic_partition_with_alter #31154
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit ccac4208e1fae21743765ccd39ef49c0fc42f18c
Author: yujun <yu...@gmail.com>
AuthorDate: Tue Feb 20 22:22:21 2024 +0800
[fix](regression test) fix test_dynamic_partition_with_alter #31154
---
.../test_dynamic_partition_with_alter.groovy | 19 ++++++++++++-------
1 file changed, 12 insertions(+), 7 deletions(-)
diff --git a/regression-test/suites/partition_p0/dynamic_partition/test_dynamic_partition_with_alter.groovy b/regression-test/suites/partition_p0/dynamic_partition/test_dynamic_partition_with_alter.groovy
index a844ad7549d..d488a53812f 100644
--- a/regression-test/suites/partition_p0/dynamic_partition/test_dynamic_partition_with_alter.groovy
+++ b/regression-test/suites/partition_p0/dynamic_partition/test_dynamic_partition_with_alter.groovy
@@ -38,14 +38,19 @@ suite("test_dynamic_partition_with_alter") {
result = sql "show partitions from ${tbl}"
assertEquals(7, result.size())
- test {
- sql "alter table ${tbl} set ('dynamic_partition.start' = '-2147483648')"
- exception "Provide start or history_partition_num property"
- }
+ try {
+ test {
+ sql "alter table ${tbl} set ('dynamic_partition.start' = '-2147483648')"
+ exception "Provide start or history_partition_num property"
+ }
- test {
- sql "alter table ${tbl} set ('dynamic_partition.start' = '-2147483647')"
- exception "Too many dynamic partitions"
+ test {
+ sql "alter table ${tbl} set ('dynamic_partition.start' = '-2147483647')"
+ exception "Too many dynamic partitions"
+ }
+ } catch (Exception e) {
+ sql "drop table if exists ${tbl}"
+ throw e
}
// modify distributed column comment, then try to add too more dynamic partition
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 06/19: [fix](case) adjust case if running on muti BEs (#31160)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 27ae96c5d036bd2189ba50bb3ca5d894a467d0a7
Author: Dongyang Li <he...@qq.com>
AuthorDate: Tue Feb 20 22:00:32 2024 +0800
[fix](case) adjust case if running on muti BEs (#31160)
Co-authored-by: stephen <he...@qq.com>
---
.../suites/datatype_p0/decimalv3/test_decimal256_outfile_csv.groovy | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/regression-test/suites/datatype_p0/decimalv3/test_decimal256_outfile_csv.groovy b/regression-test/suites/datatype_p0/decimalv3/test_decimal256_outfile_csv.groovy
index 54bbb5ca3fa..24054e6408d 100644
--- a/regression-test/suites/datatype_p0/decimalv3/test_decimal256_outfile_csv.groovy
+++ b/regression-test/suites/datatype_p0/decimalv3/test_decimal256_outfile_csv.groovy
@@ -107,6 +107,11 @@ suite("test_decimal256_outfile_csv") {
def uuid = UUID.randomUUID().toString()
def outFilePath = """/tmp/test_decimal256_outfile_csv_${uuid}"""
+ List<List<Object>> backends = sql """ show backends """
+ assertTrue(backends.size() > 0)
+ if (backends.size() > 1) {
+ outFilePath = "/tmp"
+ }
try {
logger.info("outfile: " + outFilePath)
// check outfile
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 04/19: [fix](docs) add joinHint.md to sidebars (#31155)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 7c1bb4cb22dae17c6d73ef568a5220a3630c6b4f
Author: yagagagaga <zh...@gmail.com>
AuthorDate: Tue Feb 20 19:37:11 2024 +0800
[fix](docs) add joinHint.md to sidebars (#31155)
---
docs/en/docs/query-acceleration/hint/joinHint.md | 2 ++
docs/sidebars.json | 7 +++++++
docs/zh-CN/docs/query-acceleration/hint/joinHint.md | 3 +++
3 files changed, 12 insertions(+)
diff --git a/docs/en/docs/query-acceleration/hint/joinHint.md b/docs/en/docs/query-acceleration/hint/joinHint.md
index af120425cc8..adce5d9ddca 100644
--- a/docs/en/docs/query-acceleration/hint/joinHint.md
+++ b/docs/en/docs/query-acceleration/hint/joinHint.md
@@ -1,5 +1,7 @@
# join hint using document
+<version since="2.0.4"></version>
+
In the database, "Hint" is an instruction that instructs the query optimizer to execute a plan. By embedding hints in SQL statements, you can influence the optimizer's decision to select the desired execution path. Here is a background example using Hint:
Suppose you have a table that contains a large amount of data, and you know that in some specific circumstances, the join order of the tables in a query may affect query performance. Leading Hint allows you to specify the order of table joins that you want the optimizer to follow.
diff --git a/docs/sidebars.json b/docs/sidebars.json
index f02c6782d0d..9a16dab7590 100644
--- a/docs/sidebars.json
+++ b/docs/sidebars.json
@@ -216,6 +216,13 @@
"query-acceleration/async-materialized-view/async-materialized-view",
"query-acceleration/async-materialized-view/query-async-materialized-view"
]
+ },
+ {
+ "type": "category",
+ "label": "Hint",
+ "items": [
+ "query-acceleration/hint/joinHint"
+ ]
}
]
},
diff --git a/docs/zh-CN/docs/query-acceleration/hint/joinHint.md b/docs/zh-CN/docs/query-acceleration/hint/joinHint.md
index 269eb9a7ee6..926104f5939 100644
--- a/docs/zh-CN/docs/query-acceleration/hint/joinHint.md
+++ b/docs/zh-CN/docs/query-acceleration/hint/joinHint.md
@@ -1,4 +1,7 @@
# join hint 使用文档
+
+<version since="2.0.4"></version>
+
# 背景
在数据库中，"Hint" 是一种用于指导查询优化器执行计划的指令。通过在SQL语句中嵌入Hint，可以影响优化器的决策，以选中期望的执行路径。以下是一个使用Hint的背景示例：
假设有一个包含大量数据的表，而你知道在某些特定情况下，在一个查询中，表的连接顺序可能会影响查询性能。Leading Hint允许你指定希望优化器遵循的表连接的顺序。
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 01/19: [fix](catalog recycle bin) table partition meta is error if recover partition in some case (#31125)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 0ab81435656eb76bba109f3f43eac10ce85faebe
Author: meiyi <my...@gmail.com>
AuthorDate: Tue Feb 20 16:28:52 2024 +0800
[fix](catalog recycle bin) table partition meta is error if recover partition in some case (#31125)
---
.../Show-Statements/SHOW-CATALOG-RECYCLE-BIN.md | 2 +
.../Show-Statements/SHOW-CATALOG-RECYCLE-BIN.md | 2 +
.../apache/doris/catalog/CatalogRecycleBin.java | 11 ++--
.../suites/catalog_recycle_bin_p0/recover.groovy | 60 ++++++++++++++++++++++
4 files changed, 69 insertions(+), 6 deletions(-)
diff --git a/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-CATALOG-RECYCLE-BIN.md b/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-CATALOG-RECYCLE-BIN.md
index c0a7522791d..558443fd05c 100644
--- a/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-CATALOG-RECYCLE-BIN.md
+++ b/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-CATALOG-RECYCLE-BIN.md
@@ -50,6 +50,8 @@ The meaning of each column is as follows:
TableId： id of table
PartitionId： id of partition
DropTime： drop time of meta information
+ DataSize： the amount of data. If the type is database, this value includes the data size of the recycled tables and partitions in the database
+ RemoteDataSize： the amount of data on remote storage(hdfs or object storage). If the type is database, this value includes the remote data size of the recycled tables and partitions in the database
```
### Example
diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-CATALOG-RECYCLE-BIN.md b/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-CATALOG-RECYCLE-BIN.md
index 589da1ec6f1..6b54f4477ac 100644
--- a/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-CATALOG-RECYCLE-BIN.md
+++ b/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-CATALOG-RECYCLE-BIN.md
@@ -54,6 +54,8 @@ SHOW CATALOG RECYCLE BIN [ WHERE NAME [ = "name" | LIKE "name_matcher"] ]
TableId： table对应的id
PartitionId： partition对应的id
DropTime： 元数据放入回收站的时间
+ DataSize： 数据量. 如果元数据类型是database, 该值包含了database下在回收站中的所有table和partition的数据量
+ RemoteDataSize： remote storage(hdfs或对象存储)的数据量. 如果元数据类型是database, 该值包含了database下在回收站中的所有table和partition的remote storage数据量
```
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
index d803cde648a..177d4641fc0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/CatalogRecycleBin.java
@@ -799,10 +799,9 @@ public class CatalogRecycleBin extends MasterDaemon implements Writable {
}
PartitionInfo partitionInfo = table.getPartitionInfo();
- Range<PartitionKey> recoverRange = recoverPartitionInfo.getRange();
PartitionItem recoverItem = null;
if (partitionInfo.getType() == PartitionType.RANGE) {
- recoverItem = new RangePartitionItem(recoverRange);
+ recoverItem = new RangePartitionItem(recoverPartitionInfo.getRange());
} else if (partitionInfo.getType() == PartitionType.LIST) {
recoverItem = recoverPartitionInfo.getListPartitionItem();
}
@@ -811,18 +810,18 @@ public class CatalogRecycleBin extends MasterDaemon implements Writable {
throw new DdlException("Can not recover partition[" + partitionName + "]. Partition item conflict.");
}
- // recover partition
+ // check if partition name exists
Partition recoverPartition = recoverPartitionInfo.getPartition();
Preconditions.checkState(recoverPartition.getName().equalsIgnoreCase(partitionName));
if (!Strings.isNullOrEmpty(newPartitionName)) {
if (table.checkPartitionNameExist(newPartitionName)) {
throw new DdlException("Partition name[" + newPartitionName + "] is already used");
}
+ recoverPartition.setName(newPartitionName);
}
+
+ // recover partition
table.addPartition(recoverPartition);
- if (!Strings.isNullOrEmpty(newPartitionName)) {
- table.renamePartition(partitionName, newPartitionName);
- }
// recover partition info
long partitionId = recoverPartition.getId();
diff --git a/regression-test/suites/catalog_recycle_bin_p0/recover.groovy b/regression-test/suites/catalog_recycle_bin_p0/recover.groovy
new file mode 100644
index 00000000000..884f12d03bc
--- /dev/null
+++ b/regression-test/suites/catalog_recycle_bin_p0/recover.groovy
@@ -0,0 +1,60 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("recover") {
+ def table = "test_recover"
+
+ // create table and insert data
+ sql """ drop table if exists ${table} """
+ sql """
+ create table ${table} (
+ `id` int(11),
+ `name` varchar(128),
+ `da` date
+ )
+ engine=olap
+ duplicate key(id)
+ partition by range(da)(
+ PARTITION p3 VALUES LESS THAN ('2023-01-01'),
+ PARTITION p4 VALUES LESS THAN ('2024-01-01'),
+ PARTITION p5 VALUES LESS THAN ('2025-01-01')
+ )
+ distributed by hash(id) buckets 2
+ properties(
+ "replication_num"="1",
+ "light_schema_change"="true"
+ );
+ """
+
+ sql """ insert into ${table} values(1, 'a', '2022-01-02'); """
+ sql """ insert into ${table} values(2, 'a', '2023-01-02'); """
+ sql """ insert into ${table} values(3, 'a', '2024-01-02'); """
+
+ // drop partition
+ sql """ ALTER TABLE ${table} DROP PARTITION p3; """
+
+ // add partition with the same name as the dropped partition
+ sql """ alter table ${table} add PARTITION p3 VALUES LESS THAN("2026-01-01"); """
+ sql """ insert into ${table} values(4, 'a', '2025-01-02'); """
+ sql """ insert into ${table} PARTITION(p3) values (5, 'a', '2025-01-02'); """
+
+ // recover partition use new name
+ sql """ recover partition p3 as p6 from regression_test_catalog_recycle_bin_p0.${table}; """
+
+ // insert into partition p3
+ sql """ insert into ${table} PARTITION(p3) values (6, 'a', '2025-01-02'); """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 03/19: Fix hive p2 case (#31149)
Posted by yi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-2.1
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 87b5ed187e51fbacc053d6cdf610ed6a3295da69
Author: Jibing-Li <64...@users.noreply.github.com>
AuthorDate: Tue Feb 20 18:54:37 2024 +0800
Fix hive p2 case (#31149)
---
.../hive/test_hive_analyze_db.groovy | 226 ++++++++++-----------
1 file changed, 113 insertions(+), 113 deletions(-)
diff --git a/regression-test/suites/external_table_p2/hive/test_hive_analyze_db.groovy b/regression-test/suites/external_table_p2/hive/test_hive_analyze_db.groovy
index b0fffdef241..7d6373ffdc5 100644
--- a/regression-test/suites/external_table_p2/hive/test_hive_analyze_db.groovy
+++ b/regression-test/suites/external_table_p2/hive/test_hive_analyze_db.groovy
@@ -39,157 +39,157 @@ suite("test_hive_analyze_db", "p2,external,hive,external_remote,external_remote_
assertEquals(result.size(), 17)
assertEquals(result[0][0], "lo_orderpriority")
- assertEquals(result[0][1], "100.0")
- assertEquals(result[0][2], "5.0")
- assertEquals(result[0][3], "0.0")
- assertEquals(result[0][4], "880.0")
- assertEquals(result[0][5], "8.8")
- assertEquals(result[0][6], "'1-URGENT'")
- assertEquals(result[0][7], "'5-LOW'")
+ assertEquals(result[0][2], "100.0")
+ assertEquals(result[0][3], "5.0")
+ assertEquals(result[0][4], "0.0")
+ assertEquals(result[0][5], "880.0")
+ assertEquals(result[0][6], "8.8")
+ assertEquals(result[0][7], "'1-URGENT'")
+ assertEquals(result[0][8], "'5-LOW'")
assertEquals(result[1][0], "lo_custkey")
- assertEquals(result[1][1], "100.0")
- assertEquals(result[1][2], "26.0")
- assertEquals(result[1][3], "0.0")
- assertEquals(result[1][4], "400.0")
- assertEquals(result[1][5], "4.0")
- assertEquals(result[1][6], "67423")
- assertEquals(result[1][7], "2735521")
+ assertEquals(result[1][2], "100.0")
+ assertEquals(result[1][3], "26.0")
+ assertEquals(result[1][4], "0.0")
+ assertEquals(result[1][5], "400.0")
+ assertEquals(result[1][6], "4.0")
+ assertEquals(result[1][7], "67423")
+ assertEquals(result[1][8], "2735521")
assertEquals(result[2][0], "lo_partkey")
- assertEquals(result[2][1], "100.0")
assertEquals(result[2][2], "100.0")
- assertEquals(result[2][3], "0.0")
- assertEquals(result[2][4], "400.0")
- assertEquals(result[2][5], "4.0")
- assertEquals(result[2][6], "2250")
- assertEquals(result[2][7], "989601")
+ assertEquals(result[2][3], "100.0")
+ assertEquals(result[2][4], "0.0")
+ assertEquals(result[2][5], "400.0")
+ assertEquals(result[2][6], "4.0")
+ assertEquals(result[2][7], "2250")
+ assertEquals(result[2][8], "989601")
assertEquals(result[3][0], "lo_revenue")
- assertEquals(result[3][1], "100.0")
assertEquals(result[3][2], "100.0")
- assertEquals(result[3][3], "0.0")
- assertEquals(result[3][4], "400.0")
- assertEquals(result[3][5], "4.0")
- assertEquals(result[3][6], "101171")
- assertEquals(result[3][7], "8703450")
+ assertEquals(result[3][3], "100.0")
+ assertEquals(result[3][4], "0.0")
+ assertEquals(result[3][5], "400.0")
+ assertEquals(result[3][6], "4.0")
+ assertEquals(result[3][7], "101171")
+ assertEquals(result[3][8], "8703450")
assertEquals(result[4][0], "lo_commitdate")
- assertEquals(result[4][1], "100.0")
- assertEquals(result[4][2], "95.0")
- assertEquals(result[4][3], "0.0")
- assertEquals(result[4][4], "400.0")
- assertEquals(result[4][5], "4.0")
- assertEquals(result[4][6], "19920515")
- assertEquals(result[4][7], "19981016")
+ assertEquals(result[4][2], "100.0")
+ assertEquals(result[4][3], "95.0")
+ assertEquals(result[4][4], "0.0")
+ assertEquals(result[4][5], "400.0")
+ assertEquals(result[4][6], "4.0")
+ assertEquals(result[4][7], "19920515")
+ assertEquals(result[4][8], "19981016")
assertEquals(result[5][0], "lo_quantity")
- assertEquals(result[5][1], "100.0")
- assertEquals(result[5][2], "46.0")
- assertEquals(result[5][3], "0.0")
- assertEquals(result[5][4], "400.0")
- assertEquals(result[5][5], "4.0")
- assertEquals(result[5][6], "1")
- assertEquals(result[5][7], "50")
+ assertEquals(result[5][2], "100.0")
+ assertEquals(result[5][3], "46.0")
+ assertEquals(result[5][4], "0.0")
+ assertEquals(result[5][5], "400.0")
+ assertEquals(result[5][6], "4.0")
+ assertEquals(result[5][7], "1")
+ assertEquals(result[5][8], "50")
assertEquals(result[6][0], "lo_orderkey")
- assertEquals(result[6][1], "100.0")
- assertEquals(result[6][2], "26.0")
- assertEquals(result[6][3], "0.0")
- assertEquals(result[6][4], "400.0")
- assertEquals(result[6][5], "4.0")
- assertEquals(result[6][6], "1")
- assertEquals(result[6][7], "98")
+ assertEquals(result[6][2], "100.0")
+ assertEquals(result[6][3], "26.0")
+ assertEquals(result[6][4], "0.0")
+ assertEquals(result[6][5], "400.0")
+ assertEquals(result[6][6], "4.0")
+ assertEquals(result[6][7], "1")
+ assertEquals(result[6][8], "98")
assertEquals(result[7][0], "lo_suppkey")
- assertEquals(result[7][1], "100.0")
assertEquals(result[7][2], "100.0")
- assertEquals(result[7][3], "0.0")
- assertEquals(result[7][4], "400.0")
- assertEquals(result[7][5], "4.0")
- assertEquals(result[7][6], "4167")
- assertEquals(result[7][7], "195845")
+ assertEquals(result[7][3], "100.0")
+ assertEquals(result[7][4], "0.0")
+ assertEquals(result[7][5], "400.0")
+ assertEquals(result[7][6], "4.0")
+ assertEquals(result[7][7], "4167")
+ assertEquals(result[7][8], "195845")
assertEquals(result[8][0], "lo_supplycost")
- assertEquals(result[8][1], "100.0")
assertEquals(result[8][2], "100.0")
- assertEquals(result[8][3], "0.0")
- assertEquals(result[8][4], "400.0")
- assertEquals(result[8][5], "4.0")
- assertEquals(result[8][6], "58023")
- assertEquals(result[8][7], "121374")
+ assertEquals(result[8][3], "100.0")
+ assertEquals(result[8][4], "0.0")
+ assertEquals(result[8][5], "400.0")
+ assertEquals(result[8][6], "4.0")
+ assertEquals(result[8][7], "58023")
+ assertEquals(result[8][8], "121374")
assertEquals(result[9][0], "lo_shipmode")
- assertEquals(result[9][1], "100.0")
- assertEquals(result[9][2], "7.0")
- assertEquals(result[9][3], "0.0")
- assertEquals(result[9][4], "421.0")
- assertEquals(result[9][5], "4.21")
- assertEquals(result[9][6], "'AIR'")
- assertEquals(result[9][7], "'TRUCK'")
+ assertEquals(result[9][2], "100.0")
+ assertEquals(result[9][3], "7.0")
+ assertEquals(result[9][4], "0.0")
+ assertEquals(result[9][5], "421.0")
+ assertEquals(result[9][6], "4.21")
+ assertEquals(result[9][7], "'AIR'")
+ assertEquals(result[9][8], "'TRUCK'")
assertEquals(result[10][0], "lo_orderdate")
- assertEquals(result[10][1], "100.0")
- assertEquals(result[10][2], "26.0")
- assertEquals(result[10][3], "0.0")
- assertEquals(result[10][4], "400.0")
- assertEquals(result[10][5], "4.0")
- assertEquals(result[10][6], "19920221")
- assertEquals(result[10][7], "19980721")
+ assertEquals(result[10][2], "100.0")
+ assertEquals(result[10][3], "26.0")
+ assertEquals(result[10][4], "0.0")
+ assertEquals(result[10][5], "400.0")
+ assertEquals(result[10][6], "4.0")
+ assertEquals(result[10][7], "19920221")
+ assertEquals(result[10][8], "19980721")
assertEquals(result[11][0], "lo_linenumber")
- assertEquals(result[11][1], "100.0")
- assertEquals(result[11][2], "7.0")
- assertEquals(result[11][3], "0.0")
- assertEquals(result[11][4], "400.0")
- assertEquals(result[11][5], "4.0")
- assertEquals(result[11][6], "1")
- assertEquals(result[11][7], "7")
+ assertEquals(result[11][2], "100.0")
+ assertEquals(result[11][3], "7.0")
+ assertEquals(result[11][4], "0.0")
+ assertEquals(result[11][5], "400.0")
+ assertEquals(result[11][6], "4.0")
+ assertEquals(result[11][7], "1")
+ assertEquals(result[11][8], "7")
assertEquals(result[12][0], "lo_shippriority")
- assertEquals(result[12][1], "100.0")
- assertEquals(result[12][2], "1.0")
- assertEquals(result[12][3], "0.0")
- assertEquals(result[12][4], "400.0")
- assertEquals(result[12][5], "4.0")
- assertEquals(result[12][6], "0")
+ assertEquals(result[12][2], "100.0")
+ assertEquals(result[12][3], "1.0")
+ assertEquals(result[12][4], "0.0")
+ assertEquals(result[12][5], "400.0")
+ assertEquals(result[12][6], "4.0")
assertEquals(result[12][7], "0")
+ assertEquals(result[12][8], "0")
assertEquals(result[13][0], "lo_ordtotalprice")
- assertEquals(result[13][1], "100.0")
- assertEquals(result[13][2], "26.0")
- assertEquals(result[13][3], "0.0")
- assertEquals(result[13][4], "400.0")
- assertEquals(result[13][5], "4.0")
- assertEquals(result[13][6], "3428256")
- assertEquals(result[13][7], "36771805")
+ assertEquals(result[13][2], "100.0")
+ assertEquals(result[13][3], "26.0")
+ assertEquals(result[13][4], "0.0")
+ assertEquals(result[13][5], "400.0")
+ assertEquals(result[13][6], "4.0")
+ assertEquals(result[13][7], "3428256")
+ assertEquals(result[13][8], "36771805")
assertEquals(result[14][0], "lo_extendedprice")
- assertEquals(result[14][1], "100.0")
assertEquals(result[14][2], "100.0")
- assertEquals(result[14][3], "0.0")
- assertEquals(result[14][4], "400.0")
- assertEquals(result[14][5], "4.0")
- assertEquals(result[14][6], "104300")
- assertEquals(result[14][7], "9066094")
+ assertEquals(result[14][3], "100.0")
+ assertEquals(result[14][4], "0.0")
+ assertEquals(result[14][5], "400.0")
+ assertEquals(result[14][6], "4.0")
+ assertEquals(result[14][7], "104300")
+ assertEquals(result[14][8], "9066094")
assertEquals(result[15][0], "lo_tax")
- assertEquals(result[15][1], "100.0")
- assertEquals(result[15][2], "9.0")
- assertEquals(result[15][3], "0.0")
- assertEquals(result[15][4], "400.0")
- assertEquals(result[15][5], "4.0")
- assertEquals(result[15][6], "0")
- assertEquals(result[15][7], "8")
+ assertEquals(result[15][2], "100.0")
+ assertEquals(result[15][3], "9.0")
+ assertEquals(result[15][4], "0.0")
+ assertEquals(result[15][5], "400.0")
+ assertEquals(result[15][6], "4.0")
+ assertEquals(result[15][7], "0")
+ assertEquals(result[15][8], "8")
assertEquals(result[16][0], "lo_discount")
- assertEquals(result[16][1], "100.0")
- assertEquals(result[16][2], "11.0")
- assertEquals(result[16][3], "0.0")
- assertEquals(result[16][4], "400.0")
- assertEquals(result[16][5], "4.0")
- assertEquals(result[16][6], "0")
- assertEquals(result[16][7], "10")
+ assertEquals(result[16][2], "100.0")
+ assertEquals(result[16][3], "11.0")
+ assertEquals(result[16][4], "0.0")
+ assertEquals(result[16][5], "400.0")
+ assertEquals(result[16][6], "4.0")
+ assertEquals(result[16][7], "0")
+ assertEquals(result[16][8], "10")
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org