You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2023/01/19 07:16:21 UTC
[doris] branch master updated: [ComputeNode](Doc)Add document for compute node (#15743)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new f9406234c6 [ComputeNode](Doc)Add document for compute node (#15743)
f9406234c6 is described below
commit f9406234c6186a7af908fb986ab6f0aaecff84c5
Author: huangzhaowei <ca...@gmail.com>
AuthorDate: Thu Jan 19 15:16:14 2023 +0800
[ComputeNode](Doc)Add document for compute node (#15743)
Issue Number: close #13144
---
docs/en/docs/advanced/compute_node.md | 115 +++++++++++++++++++++++++++++++
docs/sidebars.json | 3 +-
docs/zh-CN/docs/advanced/compute_node.md | 104 ++++++++++++++++++++++++++++
3 files changed, 221 insertions(+), 1 deletion(-)
diff --git a/docs/en/docs/advanced/compute_node.md b/docs/en/docs/advanced/compute_node.md
new file mode 100644
index 0000000000..5be6e131fe
--- /dev/null
+++ b/docs/en/docs/advanced/compute_node.md
@@ -0,0 +1,115 @@
+---
+{
+ "title": "Compute Node",
+ "language": "en"
+}
+---
+
+ <! --
+ Licensed to the Apache Software Foundation (ASF) under one
+ Or more contributor license agreements. See the NOTICE file
+ Distributed with this work for additional information
+ Regarding copyright ownership. The ASF licenses this file
+ To you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ With the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ Software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either expressed or implied. See the License for the
+ Specific language governing permissions and limitations
+ Under the License.
+ -- >
+
+
+# Compute node
+
+<version since="1.2.1">
+</version>
+
+## Scenario
+
+At present, Doris is a typical Share-Nothing architecture, which achieves very high performance by binding data and computing resources in the same node.
+With the continuous improvement of the performance for the Doris computing engine, more and more users have begun to use Doris to directly query data on data lake.
+This is a Share-Disk scenario that data is often stored on the remote HDFS/S3, and calculated in Doris.
+Doris will get the data through the network, and then completes the computation in memory.
+For these two mixed loads in one cluster, current Doris architecture will appear some disadvantages:
+1. Poor resource isolation, the response requirements of these two loads are different, and the hybrid deployment will have mutual effects.
+2. Poor disk usage, the data lake query only needs the computing resources, while doris binding the storage and computing and we have to expand them together, and cause a low utilization rate for disk.
+3. Poor expansion efficiency, when the cluster is expanded, Doris will start the migration of Tablet data, and this process will take a lot of time. And the data lake query load has obvious peaks and valleys, it need hourly flexibility.
+
+## solution
+Implement a BE node role specially used for federated computing named `Compute node`.
+`Compute node` is used to handle remote federated queries such as this query of data lake.
+The original BE node type is called `hybrid node`, and this type of node can not only execute SQL query, but also handle tablet data storage.
+And the `Compute node` only can execute SQL query, it have no data on node.
+
+With the computing node, the cluster deployment topology will also change:
+- the `hybrid node` is used for the data calculation of the OLAP type table, the node is expanded according to the storage demand
+- the `computing node` is used for the external computing, and this node is expanded according to the query load.
+
+ Since the compute node has no storage, the compute node can be deployed on an HDD disk machine with other workload or on a container.
+
+
+## Usage of ComputeNode
+
+### Configure
+Add configuration items to BE's configuration file `be.conf`:
+```
+ be_node_role = computation
+```
+
+This defualt value of this is `mix`, and this is original BE node type. After setting to `computation`, the node is a computing node.
+
+You can see the value of the'NodeRole 'field through the `show backend\G` command. If it is'mix ', it is a mixed node, and if it is'computation', it is a computing node
+
+```sql
+*************************** 1. row ***************************
+ BackendId: 10010
+ Cluster: default_cluster
+ IP: 10.248.181.219
+ HeartbeatPort: 9050
+ BePort: 9060
+ HttpPort: 8040
+ BrpcPort: 8060
+ LastStartTime: 2022-11-30 23:01:40
+ LastHeartbeat: 2022-12-05 15:01:18
+ Alive: true
+ SystemDecommissioned: false
+ ClusterDecommissioned: false
+ TabletNum: 753
+ DataUsedCapacity: 1.955 GB
+ AvailCapacity: 202.987 GB
+ TotalCapacity: 491.153 GB
+ UsedPct: 58.67 %
+ MaxDiskUsedPct: 58.67 %
+ RemoteUsedCapacity: 0.000
+ Tag: {"location" : "default"}
+ ErrMsg:
+ Version: doris-0.0.0-trunk-80baca264
+ Status: {"lastSuccessReportTabletsTime":"2022-12-05 15:00:38","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false}
+HeartbeatFailureCounter: 0
+ NodeRole: computation
+```
+
+### Usage
+When using the [MultiCatalog](https://doris.apache.org/docs/dev/ecosystem/external-table/multi-catalog/) , the query will be preferentially scheduled to the compute node.
+In order to balance task scheduling, FE has a `backend_num_for_federation` configuration item, which defaults to 3.
+When executing a federated query, the optimizer will select `backend_num_for_federation` as an alternative to the scheduler,
+and the scheduler will decide which node to execute on to prevent the task from being skewed.
+If the number of compute nodes is less than `backend_num_for_federation`, the mixed nodes will be randomly selected to meet the number.
+If the compute node is greater than `backend_num_for_federation`, the federated query task will only be scheduled on the compute node.
+
+### some restrictions
+- The compute node currently only supports the query for Hive MetaStore of MultiCatalog and the others are still on the hybrid node.
+- Compute nodes are controlled by configuration items, so do not configure mixed type nodes, modify the configuration to compute nodes.
+
+## Unfinished business
+- Computational spillover: Doris inner table query, when the cluster load is high, the upper layer (outside TableScan) operator can be scheduled to the compute node.
+- Graceful offline:
+ - When the compute node goes offline, the new task of the task is automatically scheduled to online nodes
+ - the node go offline after all the old tasks on the node are completed
+ - when the old task cannot be completed on time, the task can kill by itself
diff --git a/docs/sidebars.json b/docs/sidebars.json
index fed0b1f368..e77602eb26 100644
--- a/docs/sidebars.json
+++ b/docs/sidebars.json
@@ -174,7 +174,8 @@
"advanced/variables",
"advanced/time-zone",
"advanced/small-file-mgr",
- "advanced/cold_hot_separation"
+ "advanced/cold_hot_separation",
+ "advanced/compute_node"
]
},
{
diff --git a/docs/zh-CN/docs/advanced/compute_node.md b/docs/zh-CN/docs/advanced/compute_node.md
new file mode 100644
index 0000000000..9fe03e86d3
--- /dev/null
+++ b/docs/zh-CN/docs/advanced/compute_node.md
@@ -0,0 +1,104 @@
+---
+{
+ "title": "计算节点",
+ "language": "zh-CN"
+}
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# 计算节点
+
+<version since="1.2.1">
+</version>
+
+## 需求场景
+目前Doris是一个典型Share-Nothing的架构, 通过绑定数据和计算资源在同一个节点获得非常好的性能表现.
+但随着Doris计算引擎性能持续提高, 越来越多的用户也开始选择使用Doris直接查询数据湖数据.
+这类场景是一种Share-Disk场景, 数据往往存储在远端的HDFS/S3上, 计算在Doris中, Doris通过网络获取数据, 然后在内存完成计算.
+而如果这两个负载都混合在同一个集群时, 对于目前Doris的架构就会出现以下不足:
+1. 资源隔离差, 两个负载对集群的响应要求不一, 混合部署会有相互的影响.
+2. 集群扩容时, 数据湖查询只需要扩容计算资源, 而目前只能存储计算一起扩容, 导致磁盘使用率变低.
+3. 扩容效率差, 扩容后会启动Tablet数据的迁移, 整体过程比较漫长. 而数据湖查询有着明显的高峰低谷, 需要小时级弹性能力.
+
+## 解决方案
+实现一种专门用于联邦计算的BE节点角色: `计算节点`, 计算节点专门处理数据湖这类远程的联邦查询.
+原来的BE节点类型称为`混合节点`, 这类节点既能做SQL查询, 又有Tablet数据存储管理.
+而`计算节点`只能做SQL查询, 它不会保存任何数据.
+
+有了计算节点后, 集群部署拓扑也会发生变: 混合节点用于OLAP类型表的数据计算, 这个节点根据存储的需求而扩容, 而计算节点用于联邦查询, 该节点类型随着计算负载而扩容.
+
+此外, 计算节点由于没有存储, 因此在部署时, 计算节点可以混部在HDD磁盘机器或者部署在容器之中.
+
+## Compute Node的使用
+
+### 配置
+在BE的配置文件be.conf中添加配置项:
+```
+be_node_role=computation
+```
+
+该配置项默认为`mix`, 即原来的BE节点类型, 设置为`computation`后, 该节点为计算节点.
+
+可以通过`show backend\G`命令看到其中`NodeRole`字段的值, 如果是`mix`, 则为混合节点, 如果是`computation`, 则为计算节点
+
+```sql
+*************************** 1. row ***************************
+ BackendId: 10010
+ Cluster: default_cluster
+ IP: 10.248.181.219
+ HeartbeatPort: 9050
+ BePort: 9060
+ HttpPort: 8040
+ BrpcPort: 8060
+ LastStartTime: 2022-11-30 23:01:40
+ LastHeartbeat: 2022-12-05 15:01:18
+ Alive: true
+ SystemDecommissioned: false
+ ClusterDecommissioned: false
+ TabletNum: 753
+ DataUsedCapacity: 1.955 GB
+ AvailCapacity: 202.987 GB
+ TotalCapacity: 491.153 GB
+ UsedPct: 58.67 %
+ MaxDiskUsedPct: 58.67 %
+ RemoteUsedCapacity: 0.000
+ Tag: {"location" : "default"}
+ ErrMsg:
+ Version: doris-0.0.0-trunk-80baca264
+ Status: {"lastSuccessReportTabletsTime":"2022-12-05 15:00:38","lastStreamLoadTime":-1,"isQueryDisabled":false,"isLoadDisabled":false}
+HeartbeatFailureCounter: 0
+ NodeRole: computation
+```
+
+### 使用
+当查询时使用[MultiCatalog](https://doris.apache.org/zh-CN/docs/dev/ecosystem/external-table/multi-catalog)功能时, 查询会优先调度到计算节点, 为了均衡任务调度, FE有一个`backend_num_for_federation`配置项, 默认是3.
+当执行联邦查询时, 优化器会选取`backend_num_for_federation`给调度器备选, 由调取器决定具体在哪个节点执行, 防止查询任务倾斜.
+当计算节点个数小于`backend_num_for_federation`时, 会随机选择混合节点补齐个数;当计算节点大于`backend_num_for_federation`, 那么联邦查询任务只会在计算节点执行.
+
+
+### 一些限制
+- 计算节点目前只支持[MultiCatalog](https://doris.apache.org/zh-CN/docs/dev/ecosystem/external-table/multi-catalog)对应的Hive MetaStore表类型查询语法, 普通外表的计算依然在混合节点上.
+- 计算节点由配置项控制, 但不要将混合类型节点, 修改配置为计算节点.
+
+
+## 未尽事项
+- 计算外溢: Doris内表查询, 当集群负载高的时候, 上层(TableScan之外)算子调度到计算节点中.
+- 优雅下线: 当节点下线的时候, 任务新任务自动调度到其他节点; 等待老任务后全部完成后节点再下线; 老任务无法按时结束时, 能够让任务能够自己结束.
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org