You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by gi...@apache.org on 2022/01/26 01:43:29 UTC
[dolphinscheduler-website] branch asf-site updated: Automated deployment: e1bd392304930c867a1fc9144d4e80445541f6a5
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new b4945ad Automated deployment: e1bd392304930c867a1fc9144d4e80445541f6a5
b4945ad is described below
commit b4945ad5909f1a2bd558215b93a1abe9f9732881
Author: github-actions[bot] <gi...@users.noreply.github.com>
AuthorDate: Wed Jan 26 01:43:24 2022 +0000
Automated deployment: e1bd392304930c867a1fc9144d4e80445541f6a5
---
en-us/docs/dev/user_doc/guide/resource.html | 40 +++++++++++++++++++++++++
en-us/docs/dev/user_doc/guide/resource.json | 2 +-
img/task_group_conf.png | Bin 0 -> 40086 bytes
img/task_group_conf_eng.png | Bin 0 -> 41026 bytes
img/task_group_create.png | Bin 0 -> 13122 bytes
img/task_group_create_eng.png | Bin 0 -> 13513 bytes
img/task_group_manage.png | Bin 0 -> 32113 bytes
img/task_group_manage_eng.png | Bin 0 -> 33045 bytes
img/task_group_process.png | Bin 0 -> 131735 bytes
img/task_group_queue_list.png | Bin 0 -> 71483 bytes
img/task_group_queue_list_eng.png | Bin 0 -> 23574 bytes
img/task_group_use.png | Bin 0 -> 79495 bytes
img/task_group_use_eng.png | Bin 0 -> 66904 bytes
zh-cn/docs/dev/user_doc/guide/resource.html | 44 +++++++++++++++++++++++++++-
zh-cn/docs/dev/user_doc/guide/resource.json | 2 +-
15 files changed, 85 insertions(+), 3 deletions(-)
diff --git a/en-us/docs/dev/user_doc/guide/resource.html b/en-us/docs/dev/user_doc/guide/resource.html
index 103bfc3..ac47fd3 100644
--- a/en-us/docs/dev/user_doc/guide/resource.html
+++ b/en-us/docs/dev/user_doc/guide/resource.html
@@ -139,6 +139,46 @@ Operation function: rename, download, delete.</p>
<p align="center">
<img src="/img/udf_edit_en.png" width="80%" />
</p>
+<h2>Task group settings</h2>
+<p>The task group is mainly used to control the concurrency of task instances, and is designed to control the pressure of other resources (it can also control the pressure of the Hadoop cluster, the cluster will have queue control it). When creating a new task definition, you can configure the corresponding task group and configure the priority of the task running in the task group.</p>
+<h3>Task group configuration</h3>
+<h4>Create task group</h4>
+<p align="center">
+ <img src="/img/task_group_manage_eng.png" width="80%" />
+</p>
+<p>The user clicks [Resources] - [Task Group Management] - [Task Group option] - create task Group</p>
+<p align="center">
+<img src="/img/task_group_create_eng.png" width="80%" />
+</p>
+<p>You need to enter the information in the picture:</p>
+<p>[Task group name]: The name displayed when the task group is used</p>
+<p>[Project name]: The project that the task group functions, this item is optional, if not selected, all the projects in the whole system can use this task group.</p>
+<p>[Resource pool size]: The maximum number of concurrent task instances allowed</p>
+<h4>View task group queue</h4>
+<p align="center">
+ <img src="/img/task_group_conf_eng.png" width="80%" />
+</p>
+<p>Click the button to view task group usage information</p>
+<p align="center">
+ <img src="/img/task_group_queue_list_eng.png" width="80%" />
+</p>
+<h4>Use of task groups</h4>
+<p>Note: The use of task groups is applicable to tasks executed by workers, such as [switch] nodes, [condition] nodes, [sub_process] and other node types executed by the master are not controlled by the task group. Let's take the shell node as an example:</p>
+<p align="center">
+ <img src="/img/task_group_use_eng.png" width="80%" />
+</p>
+<p>Regarding the configuration of the task group, all you need to do is to configure the part in the red box:</p>
+<p>[Task group name] : The task group name displayed on the task group configuration page. Here you can only see the task group that the project has permission to (the project is selected when creating a task group), or the task group that acts globally (the new task group is created). when no item is selected)</p>
+<p>[Priority] : When there is a waiting resource, the task with high priority will be distributed to the worker by the master first. The larger the value of this part, the higher the priority.</p>
+<h3>Implementation logic of task group</h3>
+<h4>Get task group resources:</h4>
+<p>The master judges whether the task is configured with a task group when distributing the task. If the task is not configured, it is normally thrown to the worker to run; if a task group is configured, it checks whether the remaining size of the task group resource pool meets the current task operation before throwing it to the worker for execution. , if the resource pool -1 is satisfied, continue to run; if not, exit the task distribution and wait for other tasks to wake up.</p>
+<h4>Release and wake up:</h4>
+<p>When the task that has obtained the task group resource ends, the task group resource will be released. After the release, it will check whether there is a task waiting in the current task group. If there is, mark the task with the best priority to run, and create a new executable event. . The event stores the task id that is marked to obtain the resource, and then obtains the task group resource and then runs it.</p>
+<h4>Task Group Flowchart</h4>
+<p align="center">
+ <img src="/img/task_group_process.png" width="80%" />
+</p>
</div></section><footer class="footer-container"><div class="footer-body"><div><h3>About us</h3><h4>Do you need feedback? Please contact us through the following ways.</h4></div><div class="contact-container"><ul><li><a href="/en-us/community/development/subscribe.html"><img class="img-base" src="/img/emailgray.png"/><img class="img-change" src="/img/emailblue.png"/><p>Email List</p></a></li><li><a href="https://twitter.com/dolphinschedule"><img class="img-base" src="/img/twittergray.png [...]
<script src="//cdn.jsdelivr.net/npm/react@15.6.2/dist/react-with-addons.min.js"></script>
<script src="//cdn.jsdelivr.net/npm/react-dom@15.6.2/dist/react-dom.min.js"></script>
diff --git a/en-us/docs/dev/user_doc/guide/resource.json b/en-us/docs/dev/user_doc/guide/resource.json
index 33f0d1f..b774d80 100644
--- a/en-us/docs/dev/user_doc/guide/resource.json
+++ b/en-us/docs/dev/user_doc/guide/resource.json
@@ -1,6 +1,6 @@
{
"filename": "resource.md",
- "__html": "<h1>Resource Center</h1>\n<p>If you want to use the resource upload function, you can select the local file directory for a single machine(this operation does not need to deploy Hadoop). Or you can also upload to a Hadoop or MinIO cluster, at this time, you need to have Hadoop (2.6+) or MinIO and other related environments</p>\n<blockquote>\n<p><strong><em>Note:</em></strong></p>\n<ul>\n<li>If the resource upload function is used, the deployment user in <a href=\"installatio [...]
+ "__html": "<h1>Resource Center</h1>\n<p>If you want to use the resource upload function, you can select the local file directory for a single machine(this operation does not need to deploy Hadoop). Or you can also upload to a Hadoop or MinIO cluster, at this time, you need to have Hadoop (2.6+) or MinIO and other related environments</p>\n<blockquote>\n<p><strong><em>Note:</em></strong></p>\n<ul>\n<li>If the resource upload function is used, the deployment user in <a href=\"installatio [...]
"link": "/dist/en-us/docs/dev/user_doc/guide/resource.html",
"meta": {}
}
\ No newline at end of file
diff --git a/img/task_group_conf.png b/img/task_group_conf.png
new file mode 100644
index 0000000..571fcaf
Binary files /dev/null and b/img/task_group_conf.png differ
diff --git a/img/task_group_conf_eng.png b/img/task_group_conf_eng.png
new file mode 100644
index 0000000..3dcfe52
Binary files /dev/null and b/img/task_group_conf_eng.png differ
diff --git a/img/task_group_create.png b/img/task_group_create.png
new file mode 100644
index 0000000..ca71a15
Binary files /dev/null and b/img/task_group_create.png differ
diff --git a/img/task_group_create_eng.png b/img/task_group_create_eng.png
new file mode 100644
index 0000000..0a06693
Binary files /dev/null and b/img/task_group_create_eng.png differ
diff --git a/img/task_group_manage.png b/img/task_group_manage.png
new file mode 100644
index 0000000..1b12174
Binary files /dev/null and b/img/task_group_manage.png differ
diff --git a/img/task_group_manage_eng.png b/img/task_group_manage_eng.png
new file mode 100644
index 0000000..a089a4a
Binary files /dev/null and b/img/task_group_manage_eng.png differ
diff --git a/img/task_group_process.png b/img/task_group_process.png
new file mode 100644
index 0000000..492ad1a
Binary files /dev/null and b/img/task_group_process.png differ
diff --git a/img/task_group_queue_list.png b/img/task_group_queue_list.png
new file mode 100644
index 0000000..92e38ae
Binary files /dev/null and b/img/task_group_queue_list.png differ
diff --git a/img/task_group_queue_list_eng.png b/img/task_group_queue_list_eng.png
new file mode 100644
index 0000000..a90654a
Binary files /dev/null and b/img/task_group_queue_list_eng.png differ
diff --git a/img/task_group_use.png b/img/task_group_use.png
new file mode 100644
index 0000000..93ce0d6
Binary files /dev/null and b/img/task_group_use.png differ
diff --git a/img/task_group_use_eng.png b/img/task_group_use_eng.png
new file mode 100644
index 0000000..32f4bb0
Binary files /dev/null and b/img/task_group_use_eng.png differ
diff --git a/zh-cn/docs/dev/user_doc/guide/resource.html b/zh-cn/docs/dev/user_doc/guide/resource.html
index b4901bd..887e041 100644
--- a/zh-cn/docs/dev/user_doc/guide/resource.html
+++ b/zh-cn/docs/dev/user_doc/guide/resource.html
@@ -133,7 +133,49 @@
</ul>
<p align="center">
<img src="/img/udf_edit.png" width="80%" />
- </p></div></section><footer class="footer-container"><div class="footer-body"><div><h3>联系我们</h3><h4>有问题需要反馈？请通过以下方式联系我们。</h4></div><div class="contact-container"><ul><li><a href="/zh-cn/community/development/subscribe.html"><img class="img-base" src="/img/emailgray.png"/><img class="img-change" src="/img/emailblue.png"/><p>邮件列表</p></a></li><li><a href="https://twitter.com/dolphinschedule"><img class="img-base" src="/img/twittergray.png"/><img class="img-change" src="/img/twitterblue.png [...]
+ </p>
+<h2>任务组设置</h2>
+<p>任务组主要用于控制任务实例并发，旨在控制其他资源的压力（也可以控制Hadoop集群压力，不过集群会有队列管控）。您可在新建任务定义时，可配置对应的任务组，并配置任务在任务组内运行的优先级。</p>
+<h3>任务组配置</h3>
+<h4>新建任务组</h4>
+<p align="center">
+ <img src="/img/task_group_manage.png" width="80%" />
+</p>
+<p>用户点击【资源中心】-【任务组管理】-【任务组配置】-新建任务组</p>
+<p align="center">
+<img src="/img/task_group_create.png" width="80%" />
+</p>
+<p>您需要输入图片中信息，其中</p>
+<p>【任务组名称】：任务组在被使用时显示的名称</p>
+<p>【项目名称】：任务组作用的项目，该项为非必选项，如果不选择，则整个系统所有项目均可使用该任务组。</p>
+<p>【资源容量】：允许任务实例并发的最大数量</p>
+<h4>查看任务组队列</h4>
+<p align="center">
+ <img src="/img/task_group_conf.png" width="80%" />
+</p>
+<p>点击按钮查看任务组使用信息</p>
+<p align="center">
+ <img src="/img/task_group_queue_list.png" width="80%" />
+</p>
+<h4>任务组的使用</h4>
+<p>注：任务组的使用适用于由worker执行的任务，例如【switch】节点、【condition】节点、【sub_process】等由master负责执行的节点类型不受任务组控制。</p>
+<p>我们以shell节点为例：</p>
+<p align="center">
+ <img src="/img/task_group_use.png" width="80%" />
+</p>
+<p>关于任务组的配置，您需要做的只需要配置红色框内的部分，其中：</p>
+<p>【任务组名称】：任务组配置页面显示的任务组名称，这里只能看到该项目有权限的任务组（新建任务组时选择了该项目），或作用在全局的任务组（新建任务组时没有选择项目）</p>
+<p>【组内优先级】：在出现等待资源时，优先级高的任务会最先被master分发给worker执行，该部分数值越大，优先级越高。</p>
+<h3>任务组的实现逻辑</h3>
+<h4>获取任务组资源：</h4>
+<p>Master在分发任务时判断该任务是否配置了任务组，如果任务没有配置，则正常抛给worker运行；如果配置了任务组，在抛给worker执行之前检查任务组资源池剩余大小是否满足当前任务运行，如果满足资源池-1，继续运行；如果不满足则退出任务分发，等待其他任务结束唤醒。</p>
+<h4>释放与唤醒：</h4>
+<p>当获取到任务组资源的任务结束运行后，会释放任务组资源，释放后会检查当前任务组是否有任务等待，如果有则标记优先级最好的任务可以运行，并新建一个可以执行的event。该event中存储着被标记可以获取资源的任务id，随后在获取任务组资源然后运行。</p>
+<h4>任务组流程图</h4>
+<p align="center">
+ <img src="/img/task_group_process.png" width="80%" />
+</p>
+</div></section><footer class="footer-container"><div class="footer-body"><div><h3>联系我们</h3><h4>有问题需要反馈？请通过以下方式联系我们。</h4></div><div class="contact-container"><ul><li><a href="/zh-cn/community/development/subscribe.html"><img class="img-base" src="/img/emailgray.png"/><img class="img-change" src="/img/emailblue.png"/><p>邮件列表</p></a></li><li><a href="https://twitter.com/dolphinschedule"><img class="img-base" src="/img/twittergray.png"/><img class="img-change" src="/img/twitterblue.png"/><p [...]
<script src="//cdn.jsdelivr.net/npm/react@15.6.2/dist/react-with-addons.min.js"></script>
<script src="//cdn.jsdelivr.net/npm/react-dom@15.6.2/dist/react-dom.min.js"></script>
<script>window.rootPath = '';</script>
diff --git a/zh-cn/docs/dev/user_doc/guide/resource.json b/zh-cn/docs/dev/user_doc/guide/resource.json
index 6b45ce4..bc1e6dd 100644
--- a/zh-cn/docs/dev/user_doc/guide/resource.json
+++ b/zh-cn/docs/dev/user_doc/guide/resource.json
@@ -1,6 +1,6 @@
{
"filename": "resource.md",
- "__html": "<h1>资源中心</h1>\n<p>如果需要用到资源上传功能，针对单机可以选择本地文件目录作为上传文件夹(此操作不需要部署 Hadoop)。当然也可以选择上传到 Hadoop or MinIO 集群上，此时则需要有Hadoop (2.6+) 或者 MinIO 等相关环境</p>\n<blockquote>\n<p><strong><em>注意:</em></strong></p>\n<ul>\n<li>如果用到资源上传的功能，那么 <a href=\"installation/standalone.md\">安装部署</a>中，部署用户需要有这部分的操作权限</li>\n<li>如果 Hadoop 集群的 NameNode 配置了 HA 的话，需要开启 HDFS 类型的资源上传，同时需要将 Hadoop 集群下的 <code>core-site.xml</code> 和 <code>hdfs-site.xml</code> 复制到 <code>/opt/dolphinscheduler/conf</code>，非 NameNode HA 跳过次 [...]
+ "__html": "<h1>资源中心</h1>\n<p>如果需要用到资源上传功能，针对单机可以选择本地文件目录作为上传文件夹(此操作不需要部署 Hadoop)。当然也可以选择上传到 Hadoop or MinIO 集群上，此时则需要有Hadoop (2.6+) 或者 MinIO 等相关环境</p>\n<blockquote>\n<p><strong><em>注意:</em></strong></p>\n<ul>\n<li>如果用到资源上传的功能，那么 <a href=\"installation/standalone.md\">安装部署</a>中，部署用户需要有这部分的操作权限</li>\n<li>如果 Hadoop 集群的 NameNode 配置了 HA 的话，需要开启 HDFS 类型的资源上传，同时需要将 Hadoop 集群下的 <code>core-site.xml</code> 和 <code>hdfs-site.xml</code> 复制到 <code>/opt/dolphinscheduler/conf</code>，非 NameNode HA 跳过次 [...]
"link": "/dist/zh-cn/docs/dev/user_doc/guide/resource.html",
"meta": {}
}
\ No newline at end of file