You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by ji...@apache.org on 2022/07/29 08:46:49 UTC
[doris] branch master updated: [doc]Stream load doc fix (#11328)
This is an automated email from the ASF dual-hosted git repository.
jiafengzheng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 303963cfdd [doc]Stream load doc fix (#11328)
303963cfdd is described below
commit 303963cfddee877e93839e598c6858d0fbcf2636
Author: jiafeng.zhang <zh...@gmail.com>
AuthorDate: Fri Jul 29 16:46:44 2022 +0800
[doc]Stream load doc fix (#11328)
Stream load doc fix
---
.../join-optimization/doris-join-optimization.md | 222 ++++++++++++++++++++
.../import/import-way/stream-load-manual.md | 9 +-
.../Show-Statements/SHOW-STREAM-LOAD.md | 15 +-
.../join-optimization/doris-join-optimization.md | 226 +++++++++++++++++++++
.../import/import-way/stream-load-manual.md | 8 +-
.../Show-Statements/SHOW-STREAM-LOAD.md | 15 +-
6 files changed, 479 insertions(+), 16 deletions(-)
diff --git a/docs/en/advanced/join-optimization/doris-join-optimization.md b/docs/en/advanced/join-optimization/doris-join-optimization.md
new file mode 100644
index 0000000000..da17f8e699
--- /dev/null
+++ b/docs/en/advanced/join-optimization/doris-join-optimization.md
@@ -0,0 +1,222 @@
+---
+{
+ "title": "Doris Join optimization principle",
+ "language": "en"
+}
+
+
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Doris Join optimization principle
+
+Doris supports two physical operators, one is **Hash Join**, and the other is **Nest Loop Join**.
+
+- Hash Join: Create a hash table on the right table based on the equivalent join column, and the left table uses the hash table to perform join calculations in a streaming manner. Its limitation is that it can only be applied to equivalent joins.
+- Nest Loop Join: With two for loops, it is very intuitive. Then it is applicable to unequal-valued joins, such as: greater than or less than or the need to find a Cartesian product. It is a general join operator, but has poor performance.
+
+As a distributed MPP database, data shuffle needs to be performed during the Join process. Data needs to be split and scheduled to ensure that the final Join result is correct. As a simple example, assume that the relationship S and R are joined, and N represents the number of nodes participating in the join calculation; T represents the number of tuples in the relationship.
+
+
+
+## Doris Shuffle way
+
+1. Doris supports 4 Shuffle methods
+
+ 1. BroadCast Join
+
+ It requires the full data of the right table to be sent to the left table, that is, each node participating in Join has the full data of the right table, that is, T(R).
+
+ Its applicable scenarios are more general, and it can support Hash Join and Nest loop Join at the same time, and its network overhead is N * T(R).
+
+ 
+
+ The data in the left table is not moved, and the data in the right table is sent to the scanning node of the data in the left table.
+
+2. Shuffle Join
+
+ When Hash Join is performed, the corresponding Hash value can be calculated through the Join column, and Hash bucketing can be performed.
+
+ Its network overhead is: T(R) + T(N), but it can only support Hash Join, because it also calculates buckets according to the conditions of Join.
+
+ 
+
+ The left and right table data are sent to different partition nodes according to the partition, and the calculated demerits are sent.
+
+3. Bucket Shuffle Join
+
+ Doris's table data itself is bucketed by Hash calculation, so you can use the properties of the bucketed columns of the table itself to shuffle the Join data. If two tables need to be joined, and the Join column is the bucket column of the left table, then the data in the left table can actually be calculated by sending the data into the buckets of the left table without moving the data in the right table.
+
+ Its network overhead is: T(R) is equivalent to only Shuffle the data in the right table.
+
+ 
+
+ The data in the left table does not move, and the data in the right table is sent to the node that scans the table in the left table according to the result of the partition calculation.
+
+4. Colocation
+
+ It is similar to Bucket Shuffle Join, which means that the data has been shuffled according to the preset Join column scenario when data is imported. Then the join calculation can be performed directly without considering the Shuffle problem of the data during the actual query.
+
+ 
+
+ The data has been pre-partitioned, and the Join calculation is performed directly locally
+
+### Comparison of four Shuffle methods
+
+| Shuffle Mode | Network Overhead | Physical Operators | Applicable Scenarios |
+| -------------- | ------------- | ------------ ---- | --------------------------------------------- --------------- |
+| BroadCast | N * T(R) | Hash Join / Nest Loop Join | Universal |
+| Shuffle | T(S) + T(R) | Hash Join | General |
+| Bucket Shuffle | T(R) | Hash Join | There are distributed columns in the left table in the join condition, and the left table is executed as a single partition |
+| Colocate | 0 | Hash Join | There are distributed columns in the left table in the join condition, and the left and right tables belong to the same Colocate Group |
+
+N : The number of Instances participating in the Join calculation
+
+T(relation) : Tuple number of relation
+
+The flexibility of the above four methods is from high to low, and its requirements for this data distribution are becoming more and more strict, but the performance of Join calculation is also getting better and better.
+
+## Runtime Filter Join optimization
+
+Doris will build a hash table in the right table when performing Hash Join calculation, and the left table will stream through the hash table of the right table to obtain the join result. The RuntimeFilter makes full use of the Hash table of the right table. When the right table generates a hash table, a filter condition based on the hash table data is generated at the same time, and then pushed down to the data scanning node of the left table. In this way, Doris can perform data filteri [...]
+
+If the left table is a large table and the right table is a small table, then using the filter conditions generated by the left table, most of the data to be filtered in the Join layer can be filtered in advance when the data is read, so that a large amount of data can be filtered. Improve the performance of join queries.
+
+Currently Doris supports three types of RuntimeFilter
+
+- One is IN-IN, which is well understood, and pushes a hashset down to the data scanning node.
+- The second is BloomFilter, which uses the data of the hash table to construct a BloomFilter, and then pushes the BloomFilter down to the scanning node that queries the data. .
+- The last one is MinMax, which is a Range range. After the Range range is determined by the data in the right table, it is pushed down to the data scanning node.
+
+There are two requirements for the applicable scenarios of Runtime Filter:
+
+- The first requirement is that the right table is large and the left table is small, because building a Runtime Filter needs to bear the computational cost, including some memory overhead.
+- The second requirement is that there are few results from the join of the left and right tables, indicating that this join can filter out most of the data in the left table.
+
+When the above two conditions are met, turning on the Runtime Filter can achieve better results
+
+When the Join column is the Key column of the left table, the RuntimeFilter will be pushed down to the storage engine. Doris itself supports delayed materialization,
+
+Delayed materialization is simply like this: if you need to scan three columns A, B, and C, there is a filter condition on column A: A is equal to 2, if you want to scan 100 rows, you can scan 100 rows of column A first, Then filter through the filter condition A = 2. After filtering the results, read columns B and C, which can greatly reduce the data read IO. Therefore, if the Runtime Filter is generated on the Key column, and the delayed materialization of Doris itself is used to furth [...]
+
+### Runtime Filter Type
+
+- Doris provides three different Runtime Filter types:
+ - The advantage of **IN** is that the effect filtering effect is obvious and fast. Its shortcomings are: First, it only applies to BroadCast. Second, when the right table exceeds a certain amount of data, it will fail. The current Doris configuration is 1024, that is, if the right table is larger than 1024, the Runtime Filter of IN will directly failed.
+ - The advantage of **MinMax** is that the overhead is relatively small. Its disadvantage is that it has a relatively good effect on numeric columns, but basically no effect on non-numeric columns.
+ - The feature of **Bloom Filter** is that it is universal, suitable for various types, and the effect is better. The disadvantage is that its configuration is more complicated and the calculation is high.
+
+## Join Reader
+
+Once the database involves multi-table Join, the order of Join has a great impact on the performance of the entire Join query. Assuming that there are three tables to join, refer to the following picture, the left is the a table and the b table to do the join first, the intermediate result has 2000 rows, and then the c table is joined.
+
+Next, look at the picture on the right and adjust the order of Join. Join the a table with the c table first, the intermediate result generated is only 100, and then finally join with the b table for calculation. The final join result is the same, but the intermediate result it generates has a 20x difference, which results in a big performance diff.
+
+
+
+- Doris currently supports the rule-based Join Reorder algorithm. Its logic is:
+ - Make joins with large tables and small tables as much as possible, and the intermediate results it generates are as small as possible.
+ - Put the conditional join table forward, that is to say, try to filter the conditional join table
+ - Hash Join has higher priority than Nest Loop Join, because Hash Join itself is much faster than Nest Loop Join.
+
+## Doris Join optimization method
+
+Doris Join tuning method:
+
+- Use the Profile provided by Doris itself to locate the bottleneck of the query. Profile records various information in Doris' entire query, which is first-hand information for performance tuning. .
+- Understand the Join mechanism of Doris, which is also the content shared with you in the second part. Only by knowing why and understanding its mechanism can we analyze why it is slow.
+- Use Session variables to change some behaviors of Join, so as to realize the tuning of Join.
+- Check the Query Plan to analyze whether this tuning is effective.
+
+The above 4 steps basically complete a standard Join tuning process, and then it is to actually query and verify it to see what the effect is.
+
+If the first 4 methods are connected in series, it still does not work. At this time, it may be necessary to rewrite the Join statement, or to adjust the data distribution. It is necessary to recheck whether the entire data distribution is reasonable, including querying the Join statement, and some manual adjustments may be required. Of course, this method has a relatively high mental cost, which means that further analysis is required only when the previous method does not work.
+
+## Optimization case practice
+
+### Case one
+
+A four-table join query, through Profile, found that the second join took a long time, taking 14 seconds.
+
+
+
+After further analysis of Profile, it is found that BuildRows, that is, the data volume of the right table is about 25 million. And ProbeRows (ProbeRows is the amount of data in the left table) is only more than 10,000. In this scenario, the right table is much larger than the left table, which is obviously an unreasonable situation. This obviously shows that there is some problem with the order of Join. At this time, try to change the Session variable and enable Join Reorder.
+
+```
+set enable_cost_based_join_reorder = true
+```
+
+This time, the time has been reduced from 14 seconds to 4 seconds, and the performance has been improved by more than 3 times.
+
+At this time, when checking the profile again, the order of the left and right tables has been adjusted correctly, that is, the right table is a large table, and the left table is a small table. The overhead of building a hash table based on a small table is very small. This is a typical scenario of using Join Reorder to improve Join performance.
+
+
+
+### Case 2
+
+There is a slow query. After viewing the Profile, the entire Join node takes about 44 seconds. Its right table has 10 million, the left table has 60 million, and the final returned result is only 60 million.
+
+
+
+It can be roughly estimated that the filtering rate is very high, so why does the Runtime Filter not take effect? Check it out through Query Plan and find that it only enables the Runtime Filter of IN.
+
+
+
+When the right table exceeds 1024 rows, IN will not take effect, so there is no filtering effect at all, so try to adjust the type of RuntimeFilter.
+
+This is changed to BloomFilter, and the 60 million pieces of data in the left table have filtered 59 million pieces. Basically, 99% of the data is filtered out, and this effect is very significant. The query has also dropped from the original 44 seconds to 13 seconds, and the performance has been improved by about three times.
+
+### Case 3
+
+The following is a relatively extreme case, which cannot be solved by tuning some environment variables, because it involves SQL Rewrite, so the original SQL is listed here.
+
+```sql
+select 100.00 * sum (case
+ when P_type like 'PROMOS'
+ then 1 extendedprice * (1 - 1 discount)
+ else 0
+ end ) / sum(1 extendedprice * (1 - 1 discount)) as promo revenue
+from lineitem, part
+where
+ 1_partkey = p_partkey
+ and 1_shipdate >= date '1997-06-01'
+ and 1 shipdate < date '1997-06-01' + interval '1' month
+```
+
+This Join query is very simple, a simple join of left and right tables. Of course, there are some filter conditions on it. When I opened the Profile, I found that the entire query Hash Join was executed for more than three minutes. It is a BroadCast Join, and its right table has 200 million entries, while the left table has only 700,000. In this case, it is unreasonable to choose Broadcast Join, which is equivalent to making a Hash Table of 200 million records, and then traversing the Ha [...]
+
+
+
+Why is there an unreasonable Join order? In fact, the left table is a large table with a level of 1 billion records. Two filter conditions are added to it. After adding these two filter conditions, there are 700,000 records of 1 billion records. But Doris currently does not have a good framework for collecting statistics, so it does not know what the filtering rate of this filter condition is. Therefore, when the join order is arranged, the wrong left and right table order of the join is [...]
+
+The following figure is an SQL statement after the rewrite is completed. A Join Hint is added after the Join, a square bracket is added after the Join, and then the required Join method is written. Here, Shuffle Join is selected. You can see in the actual query plan on the right that the data is indeed Partitioned. The original 3-minute time-consuming is only 7 seconds after the rewriting, and the performance is improved significantly.
+
+
+
+## Doris Join optimization suggestion
+
+Finally, we summarize four suggestions for optimization and tuning of Doris Join:
+
+- The first point: When doing Join, try to select columns of the same type or simple type. If the same type is used, reduce its data cast, and the simple type itself joins the calculation quickly.
+- The second point: try to choose the Key column for Join. The reason is also introduced in the Runtime Filter. The Key column can play a better effect on delayed materialization.
+- The third point: Join between large tables, try to make it Co-location, because the network overhead between large tables is very large, if you need to do Shuffle, the cost is very high.
+- Fourth point: Use Runtime Filter reasonably, which is very effective in scenarios with high join filtering rate. But it is not a panacea, but has certain side effects, so it needs to be switched according to the granularity of specific SQL.
+- Finally: When it comes to multi-table Join, it is necessary to judge the rationality of Join. Try to ensure that the left table is a large table and the right table is a small table, and then Hash Join will be better than Nest Loop Join. If necessary, you can use SQL Rewrite to adjust the order of Join using Hint.
diff --git a/docs/en/docs/data-operate/import/import-way/stream-load-manual.md b/docs/en/docs/data-operate/import/import-way/stream-load-manual.md
index 85b13fb12c..54ba913b11 100644
--- a/docs/en/docs/data-operate/import/import-way/stream-load-manual.md
+++ b/docs/en/docs/data-operate/import/import-way/stream-load-manual.md
@@ -59,7 +59,7 @@ The following figure shows the main flow of Stream load, omitting some import de
In Stream load, Doris selects a node as the Coordinator node. This node is responsible for receiving data and distributing data to other data nodes.
-Users submit import commands through HTTP protocol. If submitted to FE, FE forwards the request to a BE via the HTTP redirect instruction. Users can also submit import commands directly to a specified BE.Therefore, when submitting the stream load command to FE, you need to add "--location-trusted", while it is not necessary to submit it directly to BE.
+Users submit import commands through HTTP protocol. If submitted to FE, FE forwards the request to a BE via the HTTP redirect instruction. Users can also submit import commands directly to a specified BE.
The final result of the import is returned to the user by Coordinator BE.
@@ -298,6 +298,13 @@ The following main explanations are given for the Stream load import result para
Users can't cancel Stream load manually. Stream load will be cancelled automatically by the system after a timeout or import error.
+### View Stream Load
+
+Users can view completed stream load tasks through `show stream load`.
+
+By default, BE does not record Stream Load records. If you want to view records that need to be enabled on BE, the configuration parameter is: `enable_stream_load_record=true`. For details, please refer to [BE Configuration Items](https://doris.apache. org/zh-CN/docs/admin-manual/config/be-config)
+
+
## Relevant System Configuration
### FE configuration
diff --git a/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-STREAM-LOAD.md b/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-STREAM-LOAD.md
index 46d1a8b015..b7e912b136 100644
--- a/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-STREAM-LOAD.md
+++ b/docs/en/docs/sql-manual/sql-reference/Show-Statements/SHOW-STREAM-LOAD.md
@@ -50,13 +50,14 @@ SHOW STREAM LOAD
illustrate:
-1. If db_name is not specified, the current default db is used
-2. If LABEL LIKE is used, it will match the tasks whose label of the Stream Load task contains label_matcher
-3. If LABEL = is used, it will match the specified label exactly
-4. If STATUS is specified, matches STREAM LOAD status
-5. You can use ORDER BY to sort on any combination of columns
-6. If LIMIT is specified, limit matching records are displayed. Otherwise show all
-7. If OFFSET is specified, the query results are displayed starting at offset offset. By default the offset is 0.
+1. By default, BE does not record Stream Load records. If you want to view records that need to be enabled on BE, the configuration parameter is: `enable_stream_load_record=true`. For details, please refer to [BE Configuration Items](https://doris.apache. org/zh-CN/docs/admin-manual/config/be-config)
+2. If db_name is not specified, the current default db is used
+3. If LABEL LIKE is used, it will match the tasks whose label of the Stream Load task contains label_matcher
+4. If LABEL = is used, it will match the specified label exactly
+5. If STATUS is specified, matches STREAM LOAD status
+6. You can use ORDER BY to sort on any combination of columns
+7. If LIMIT is specified, limit matching records are displayed. Otherwise show all
+8. If OFFSET is specified, the query results are displayed starting at offset offset. By default the offset is 0.
### Example
diff --git a/docs/zh-CN/advanced/join-optimization/doris-join-optimization.md b/docs/zh-CN/advanced/join-optimization/doris-join-optimization.md
new file mode 100644
index 0000000000..40b71e6f47
--- /dev/null
+++ b/docs/zh-CN/advanced/join-optimization/doris-join-optimization.md
@@ -0,0 +1,226 @@
+---
+{
+ "title": "Doris Join 优化原理",
+ "language": "zh-CN"
+}
+
+
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Doris Join 优化原理
+
+Doris 支持两种物理算子，一类是 **Hash Join**，另一类是 **Nest Loop Join**。
+
+- Hash Join：在右表上根据等值 Join 列建立哈希表，左表流式的利用哈希表进行 Join 计算，它的限制是只能适用于等值 Join。
+- Nest Loop Join：通过两个 for 循环，很直观。然后它适用的场景就是不等值的 Join，例如：大于小于或者是需要求笛卡尔积的场景。它是一个通用的 Join 算子，但是性能表现差。
+
+作为分布式的 MPP 数据库， 在 Join 的过程中是需要进行数据的 Shuffle。数据需要进行拆分调度，才能保证最终的 Join 结果是正确的。举个简单的例子，假设关系S 和 R 进行Join，N 表示参与 Join 计算的节点的数量；T 则表示关系的 Tuple 数目。
+
+
+
+## Doris Shuffle 方式
+
+Doris 支持 4 种 Shuffle 方式
+
+1. BroadCast Join
+
+ 它要求把右表全量的数据都发送到左表上，即每一个参与 Join 的节点，它都拥有右表全量的数据，也就是 T(R)。
+
+ 它适用的场景是比较通用的，同时能够支持 Hash Join 和 Nest loop Join，它的网络开销 N * T(R)。
+
+ 
+
+ 左表数据不移动，右表数据发送到左表数据的扫描节点。
+
+2. Shuffle Join
+
+ 当进行 Hash Join 时候，可以通过 Join 列计算对应的 Hash 值，并进行 Hash 分桶。
+
+ 它的网络开销则是：T（R） + T（N），但它只能支持 Hash Join，因为它是根据 Join 的条件也去做计算分桶的。
+
+ 
+
+ 左右表数据根据分区，计算的记过发送到不同的分区节点上。
+
+3. Bucket Shuffle Join
+
+ Doris 的表数据本身是通过 Hash 计算分桶的，所以就可以利用表本身的分桶列的性质来进行 Join 数据的 Shuffle。假如两张表需要做 Join，并且 Join 列是左表的分桶列，那么左表的数据其实可以不用去移动右表通过左表的数据分桶发送数据就可以完成 Join 的计算。
+
+ 它的网络开销则是：T（R）相当于只 Shuffle 右表的数据就可以了。
+
+ 
+
+ 左表数据不移动，右表数据根据分区计算的结果发送到左表扫表的节点
+
+4. Colocation
+
+ 它与 Bucket Shuffle Join 相似，相当于在数据导入的时候，根据预设的 Join 列的场景已经做好了数据的 Shuffle。那么实际查询的时候就可以直接进行 Join 计算而不需要考虑数据的 Shuffle 问题了。
+
+ 
+
+ 数据已经预先分区，直接在本地进行 Join 计算
+
+### 四种 Shuffle 方式对比
+
+| Shuffle方式 | 网络开销 | 物理算子 | 适用场景 |
+| -------------- | ----------- | -------------------------- | ------------------------------------------------------------ |
+| BroadCast | N * T(R) | Hash Join / Nest Loop Join | 通用 |
+| Shuffle | T(S) + T(R) | Hash Join | 通用 |
+| Bucket Shuffle | T(R) | Hash Join | Join条件中存在左表的分布式列，且左表执行时为单分区 |
+| Colocate | 0 | Hash Join | Join条件中存在左表的分布式列，切左右表同属于一个Colocate Group |
+
+N ： 参与 Join 计算的 Instance 个数
+
+T(关系) : 关系的 Tuple 数目
+
+上面这 4 种方式灵活度是从高到低的，它对这个数据分布的要求是越来越严格，但 Join 计算的性能也是越来越好的。
+
+## Runtime Filter Join 优化
+
+Doris 在进行 Hash Join 计算时会在右表构建一个哈希表，左表流式的通过右表的哈希表从而得出 Join 结果。而 RuntimeFilter 就是充分利用了右表的 Hash 表，在右表生成哈希表的时，同时生成一个基于哈希表数据的一个过滤条件，然后下推到左表的数据扫描节点。通过这样的方式，Doris 可以在运行时进行数据过滤。
+
+假如左表是一张大表，右表是一张小表，那么利用左表生成的过滤条件就可以把绝大多数在 Join 层要过滤的数据在数据读取时就提前过滤，这样就能大幅度的提升 Join 查询的性能。
+
+当前 Doris 支持三种类型 RuntimeFilter
+
+- 一种是 IN— IN，很好理解，将一个 hashset 下推到数据扫描节点。
+- 第二种就是 BloomFilter，就是利用哈希表的数据构造一个 BloomFilter，然后把这个 BloomFilter 下推到查询数据的扫描节点。。
+- 最后一种就是 MinMax，就是个 Range 范围，通过右表数据确定 Range 范围之后，下推给数据扫描节点。
+
+Runtime Filter 适用的场景有两个要求：
+
+- 第一个要求就是右表大左表小，因为构建 Runtime Filter是需要承担计算成本的，包括一些内存的开销。
+- 第二个要求就是左右表 Join 出来的结果很少，说明这个 Join 可以过滤掉左表的绝大部分数据。
+
+当符合上面两个条件的情况下，开启 Runtime Filter 就能收获比较好的效果
+
+当 Join 列为左表的 Key 列时，RuntimeFilter 会下推到存储引擎。Doris 本身支持延迟物化，
+
+延迟物化简单来说是这样的：假如需要扫描 A、B、C 三列，在 A 列上有一个过滤条件: A 等于 2，要扫描 100 行的话，可以先把 A 列的 100 行扫描出来，再通过 A = 2 这个过滤条件过滤。之后通过过滤完成后的结果，再去读取 B、C 列，这样就能极大的降低数据的读取 IO。所以说 Runtime Filter 如果在 Key 列上生成，同时利用 Doris 本身的延迟物化来进一步提升查询的性能。
+
+### Runtime Filter 类型
+
+Doris 提供了三种不同的 Runtime Filter 类型：
+
+- **IN** 的优点就是效果过滤效果明显，且快速。它的缺点首先第一个它只适用于 BroadCast，第二，它右表超过一定数据量的时候就失效了，当前 Doris 目前配置的是1024，即右表如果大于 1024，IN 的 Runtime Filter 就直接失效了。
+- **MinMax** 的优点是开销比较小。它的缺点就是对数值列还有比较好的效果，但对于非数值列，基本上就没什么效果。
+- **Bloom Filter** 的特点就是通用，适用于各种类型、效果也比较好。缺点就是它的配置比较复杂并且计算较高。
+
+
+
+## Join Reader
+
+数据库一旦涉及到多表 Join，Join 的顺序对整个 Join 查询的性能是影响很大的。假设有三张表 Join，参考下面这张图，左边是 a 表跟 b 张表先做 Join，中间结果的有 2000 行，然后与 c 表再进行 Join 计算。
+
+接下来看右图，把 Join 的顺序调整了一下。把 a 表先与 c 表 Join，生成的中间结果只有 100，然后最终再与 b 表 Join 计算。最终的 Join 结果是一样的，但是它生成的中间结果有 20 倍的差距，这就会产生一个很大的性能 Diff 了。
+
+
+
+Doris 目前支持基于规则的 Join Reorder 算法。它的逻辑是：
+
+- 让大表、跟小表尽量做 Join，它生成的中间结果是尽可能小的。
+- 把有条件的 Join 表往前放，也就是说尽量让有条件的 Join 表进行过滤
+- Hash Join 的优先级高于 Nest Loop Join，因为 Hash join 本身是比 Nest Loop Join 快很多的。
+
+## Doris Join 调优方法
+
+Doris Join 调优的方法：
+
+- 利用 Doris 本身提供的 Profile，去定位查询的瓶颈。Profile 会记录 Doris 整个查询当中各种信息，这是进行性能调优的一手资料。。
+- 了解 Doris 的 Join 机制，这也是第二部分跟大家分享的内容。知其然知其所以然、了解它的机制，才能分析它为什么比较慢。
+- 利用 Session 变量去改变 Join 的一些行为，从而实现 Join 的调优。
+- 查看 Query Plan 去分析这个调优是否生效。
+
+上面的 4 步基本上完成了一个标准的 Join 调优流程，接着就是实际去查询验证它，看看效果到底怎么样。
+
+如果前面 4 种方式串联起来之后，还是不奏效。这时候可能就需要去做 Join 语句的改写，或者是数据分布的调整、需要重新去 Recheck 整个数据分布是否合理，包括查询 Join 语句，可能需要做一些手动的调整。当然这种方式是心智成本是比较高的，也就是说要在尝试前面方式不奏效的情况下，才需要去做进一步的分析。
+
+## 调优案例实战
+
+### 案例一
+
+一个四张表 Join 的查询，通过 Profile 的时候发现第二个 Join 耗时很高，耗时 14 秒。
+
+
+
+进一步分析 Profile 之后，发现 BuildRows，就是右表的数据量是大概 2500 万。而 ProbeRows （ ProbeRows 是左表的数据量）只有 1 万多。这种场景下右表是远远大于左表，这显然是个不合理的情况。这显然说明 Join 的顺序出现了一些问题。这时候尝试改变 Session 变量，开启 Join Reorder。
+
+```
+set enable_cost_based_join_reorder = true
+```
+
+这次耗时从 14 秒降到了 4 秒，性能提升了 3 倍多。
+
+此时再 Check Profile 的时候，左右表的顺序已经调整正确，即右表是大表，左表是小表。基于小表去构建哈希表，开销是很小的，这就是典型的一个利用 Join Reorder 去提升 Join 性能的一个场景
+
+
+
+### 案例二
+
+存在一个慢查询，查看 Profile 之后，整个 Join 节点耗时大概44秒。它的右表有 1000 万，左表有 6000 万，最终返回的结果也只有 6000 万。
+
+
+
+这里可以大致的估算出过滤率是很高的，那为什么 Runtime Filter 没有生效呢？通过 Query Plan 去查看它，发现它只开启了 IN 的 Runtime Filter。
+
+
+
+当右表超过1024行的话， IN 是不生效的，所以根本起不到什么过滤的效果，所以尝试调整 RuntimeFilter 的类型。
+
+这里改为了 BloomFilter，左表的 6000 万条数据过滤了 5900 万条。基本上 99% 的数据都被过滤掉了，这个效果是很显著的。查询也从原来的 44 秒降到了 13 秒，性能提升了大概也是三倍多。
+
+### 案例三
+
+下面是一个比较极端的 Case，通过一些环境变量调优也没有办法解决，因为它涉及到 SQL Rewrite，所以这里列出来了原始的 SQL 。
+
+```sql
+select 100.00 * sum (case
+ when P_type like 'PROMOS'
+ then 1 extendedprice * (1 - 1 discount)
+ else 0
+ end ) / sum(1 extendedprice * (1 - 1 discount)) as promo revenue
+from lineitem, part
+where
+ 1_partkey = p_partkey
+ and 1_shipdate >= date '1997-06-01'
+ and 1 shipdate < date '1997-06-01' + interval '1' month
+```
+
+这个 Join 查询是很简单的，单纯的一个左右表的 Join 。当然它上面有一些过滤条件，打开 Profile 的时候，发现整个查询 Hash Join 执行了三分多钟，它是一个 BroadCast 的 Join，它的右表有 2 亿条，左表只有 70 万。在这种情况下选择了 Broadcast Join 是不合理的，这相当于要把 2 亿条做一个 Hash Table，然后用 70 万条遍历两亿条的 Hash Table ，这显然是不合理的。
+
+
+
+为什么会产生不合理的 Join 顺序呢？其实这个左表是一个 10 亿条级别的大表，它上面加了两个过滤条件，加完这两个过滤条件之后， 10 亿条的数据就剩 70 万条了。但 Doris 目前没有一个好的统计信息收集的框架，所以它不知道这个过滤条件的过滤率到底怎么样。所以这个 Join 顺序安排的时候，就选择了错误的 Join 的左右表顺序，导致它的性能是极其低下的。
+
+下图是改写完成之后的一个 SQL 语句，在 Join 后面添加了一个Join Hint，在Join 后面加一个方括号，然后把需要的 Join 方式写入。这里选择了 Shuffle Join，可以看到右边它实际查询计划里面看到这个数据确实是做了 Partition ，原先 3 分钟的耗时通过这样的改写完之后只剩下 7 秒，性能提升明显
+
+
+
+## Doris Join 调优建议
+
+最后我们总结 Doris Join 优化调优的四点建议：
+
+- 第一点：在做 Join 的时候，要尽量选择同类型或者简单类型的列，同类型的话就减少它的数据 Cast，简单类型本身 Join 计算就很快。
+- 第二点：尽量选择 Key 列进行 Join， 原因前面在 Runtime Filter 的时候也介绍了，Key 列在延迟物化上能起到一个比较好的效果。
+- 第三点：大表之间的 Join ，尽量让它 Co-location ，因为大表之间的网络开销是很大的，如果需要去做 Shuffle 的话，代价是很高的。
+- 第四点：合理的使用 Runtime Filter，它在 Join 过滤率高的场景下效果是非常显著的。但是它并不是万灵药，而是有一定副作用的，所以需要根据具体的 SQL 的粒度做开关。
+- 最后：要涉及到多表 Join 的时候，需要去判断 Join 的合理性。尽量保证左表为大表，右表为小表，然后 Hash Join 会优于 Nest Loop Join。必要的时可以通过 SQL Rewrite，利用 Hint 去调整 Join 的顺序。
diff --git a/docs/zh-CN/docs/data-operate/import/import-way/stream-load-manual.md b/docs/zh-CN/docs/data-operate/import/import-way/stream-load-manual.md
index 239cf721a9..714cc0a989 100644
--- a/docs/zh-CN/docs/data-operate/import/import-way/stream-load-manual.md
+++ b/docs/zh-CN/docs/data-operate/import/import-way/stream-load-manual.md
@@ -59,7 +59,7 @@ Stream load 主要适用于导入本地文件，或通过程序导入数据流
Stream load 中，Doris 会选定一个节点作为 Coordinator 节点。该节点负责接数据并分发数据到其他数据节点。
-用户通过 HTTP 协议提交导入命令。如果提交到 FE，则 FE 会通过 HTTP redirect 指令将请求转发给某一个 BE。用户也可以直接提交导入命令给某一指定 BE。所以Stream load命令提交到FE则需要加"--location-trusted"，而直接提交到BE则不需要。
+用户通过 HTTP 协议提交导入命令。如果提交到 FE，则 FE 会通过 HTTP redirect 指令将请求转发给某一个 BE。用户也可以直接提交导入命令给某一指定 BE。
导入的最终结果由 Coordinator BE 返回给用户。
@@ -310,6 +310,12 @@ Stream Load 由于使用的是 HTTP 协议，所以所有导入任务有关的
用户无法手动取消 Stream Load，Stream Load 在超时或者导入错误后会被系统自动取消。
+### 查看 Stream Load
+
+用户可以通过 `show stream load` 来查看已经完成的 stream load 任务。
+
+默认 BE 是不记录 Stream Load 的记录，如果你要查看需要在 BE 上启用记录，配置参数是：`enable_stream_load_record=true` ，具体怎么配置请参照 [BE 配置项](https://doris.apache.org/zh-CN/docs/admin-manual/config/be-config)
+
## 相关系统配置
### FE配置
diff --git a/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-STREAM-LOAD.md b/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-STREAM-LOAD.md
index fd4fbff163..a34cb8dc91 100644
--- a/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-STREAM-LOAD.md
+++ b/docs/zh-CN/docs/sql-manual/sql-reference/Show-Statements/SHOW-STREAM-LOAD.md
@@ -50,13 +50,14 @@ SHOW STREAM LOAD
说明：
-1. 如果不指定 db_name，使用当前默认db
-2. 如果使用 LABEL LIKE，则会匹配Stream Load任务的 label 包含 label_matcher 的任务
-3. 如果使用 LABEL = ，则精确匹配指定的 label
-4. 如果指定了 STATUS，则匹配 STREAM LOAD 状态
-5. 可以使用 ORDER BY 对任意列组合进行排序
-6. 如果指定了 LIMIT，则显示 limit 条匹配记录。否则全部显示
-7. 如果指定了 OFFSET，则从偏移量offset开始显示查询结果。默认情况下偏移量为0。
+1.默认 BE 是不记录 Stream Load 的记录，如果你要查看需要在 BE 上启用记录，配置参数是：`enable_stream_load_record=true` ，具体怎么配置请参照 [BE 配置项](https://doris.apache.org/zh-CN/docs/admin-manual/config/be-config)
+2. 如果不指定 db_name，使用当前默认db
+3. 如果使用 LABEL LIKE，则会匹配Stream Load任务的 label 包含 label_matcher 的任务
+4. 如果使用 LABEL = ，则精确匹配指定的 label
+5. 如果指定了 STATUS，则匹配 STREAM LOAD 状态
+6. 可以使用 ORDER BY 对任意列组合进行排序
+7. 如果指定了 LIMIT，则显示 limit 条匹配记录。否则全部显示
+8. 如果指定了 OFFSET，则从偏移量offset开始显示查询结果。默认情况下偏移量为0。
### Example
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org