You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by zh...@apache.org on 2022/03/04 15:56:56 UTC
[dolphinscheduler-website] branch master updated: Fix headlines (#711)
This is an automated email from the ASF dual-hosted git repository.
zhongjiajie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler-website.git
The following commit(s) were added to refs/heads/master by this push:
new 668390c Fix headlines (#711)
668390c is described below
commit 668390cd0ff1f6379d40bff0267a9f53106c83f1
Author: Tq <ti...@gmail.com>
AuthorDate: Fri Mar 4 23:56:50 2022 +0800
Fix headlines (#711)
* fix-headlines,according to rules below:
1. Use Headline-style capitalization in all headlines.
2. Use document name as lvl.1(#) heading.
3. Use ascend count of # to the sub-headings.
4. Use a blank line under and below each headings.
5. Delete number counters in the sub-headings.(like '1.1.x')
* fix according to the review:
1. replace &->and.
2. docker-compose->Docker Compose.(except code)
3. zookeeper->ZooKeeper.(except code)
4. /->or.
* fix according to the review:delete number counters
* little fix to some words
---
.../About_DolphinScheduler.md | 15 +++--
docs/en-us/2.0.3/user_doc/architecture/cache.md | 10 +--
.../2.0.3/user_doc/architecture/configuration.md | 69 ++++++++++++--------
docs/en-us/2.0.3/user_doc/architecture/design.md | 75 +++++++++++-----------
.../2.0.3/user_doc/architecture/designplus.md | 10 +--
.../2.0.3/user_doc/architecture/load-balance.md | 22 +++----
docs/en-us/2.0.3/user_doc/architecture/metadata.md | 44 +++++++++----
.../2.0.3/user_doc/architecture/task-structure.md | 61 +++++++-----------
.../guide/alert/alert_plugin_user_guide.md | 4 +-
.../user_doc/guide/alert/enterprise-wechat.md | 2 +
docs/en-us/2.0.3/user_doc/guide/datasource/hive.md | 2 +-
.../user_doc/guide/datasource/introduction.md | 1 -
.../en-us/2.0.3/user_doc/guide/datasource/mysql.md | 1 -
.../2.0.3/user_doc/guide/datasource/postgresql.md | 2 +-
.../2.0.3/user_doc/guide/expansion-reduction.md | 18 +++---
docs/en-us/2.0.3/user_doc/guide/flink-call.md | 22 +++----
.../2.0.3/user_doc/guide/installation/cluster.md | 12 ++--
.../2.0.3/user_doc/guide/installation/docker.md | 72 ++++++++++-----------
.../2.0.3/user_doc/guide/installation/hardware.md | 10 +--
.../user_doc/guide/installation/kubernetes.md | 38 +++++------
.../user_doc/guide/installation/pseudo-cluster.md | 20 +++---
.../user_doc/guide/installation/standalone.md | 6 +-
docs/en-us/2.0.3/user_doc/guide/monitor.md | 17 +++--
.../guide/observability/skywalking-agent.md | 12 ++--
docs/en-us/2.0.3/user_doc/guide/open-api.md | 8 +--
.../2.0.3/user_doc/guide/parameter/context.md | 4 +-
.../2.0.3/user_doc/guide/project/project-list.md | 4 +-
.../2.0.3/user_doc/guide/project/task-instance.md | 3 +-
.../user_doc/guide/project/workflow-definition.md | 12 ++--
.../user_doc/guide/project/workflow-instance.md | 12 ++--
docs/en-us/2.0.3/user_doc/guide/resource.md | 10 +--
docs/en-us/2.0.3/user_doc/guide/security.md | 13 ++--
docs/en-us/2.0.3/user_doc/guide/task/conditions.md | 2 +-
docs/en-us/2.0.3/user_doc/guide/task/datax.md | 3 +-
docs/en-us/2.0.3/user_doc/guide/task/dependent.md | 2 +-
docs/en-us/2.0.3/user_doc/guide/task/flink.md | 8 +--
docs/en-us/2.0.3/user_doc/guide/task/http.md | 1 -
docs/en-us/2.0.3/user_doc/guide/task/map-reduce.md | 7 +-
docs/en-us/2.0.3/user_doc/guide/task/spark.md | 8 +--
docs/en-us/2.0.3/user_doc/guide/task/sql.md | 6 +-
docs/en-us/2.0.3/user_doc/guide/upgrade.md | 17 +++--
41 files changed, 342 insertions(+), 323 deletions(-)
diff --git a/docs/en-us/2.0.3/user_doc/About_DolphinScheduler/About_DolphinScheduler.md b/docs/en-us/2.0.3/user_doc/About_DolphinScheduler/About_DolphinScheduler.md
index 5f1cb64..aafcca1 100644
--- a/docs/en-us/2.0.3/user_doc/About_DolphinScheduler/About_DolphinScheduler.md
+++ b/docs/en-us/2.0.3/user_doc/About_DolphinScheduler/About_DolphinScheduler.md
@@ -2,11 +2,18 @@
Apache DolphinScheduler is a cloud-native visual Big Data workflow scheduler system, committed to “solving complex big-data task dependencies and triggering relationships in data OPS orchestration so that various types of big data tasks can be used out of the box”.
-# High Reliability
+## High Reliability
+
- Decentralized multi-master and multi-worker, HA is supported by itself, overload processing
-# User-Friendly
+
+## User-Friendly
+
- All process definition operations are visualized, Visualization process defines key information at a glance, One-click deployment
-# Rich Scenarios
+
+## Rich Scenarios
+
- Support multi-tenant. Support many task types e.g., spark,flink,hive, mr, shell, python, sub_process
-# High Expansibility
+
+## High Expansibility
+
- Support custom task types, Distributed scheduling, and the overall scheduling capability will increase linearly with the scale of the cluster
diff --git a/docs/en-us/2.0.3/user_doc/architecture/cache.md b/docs/en-us/2.0.3/user_doc/architecture/cache.md
index 6a7359d..141a9c7 100644
--- a/docs/en-us/2.0.3/user_doc/architecture/cache.md
+++ b/docs/en-us/2.0.3/user_doc/architecture/cache.md
@@ -1,12 +1,12 @@
-### Cache
+# Cache
-#### Purpose
+## Purpose
Due to the master-server scheduling process, there will be a large number of database read operations, such as `tenant`, `user`, `processDefinition`, etc. On the one hand, it will put a lot of pressure on the DB, and on the other hand, it will slow down the entire core scheduling process.
Considering that this part of the business data is a scenario where more reads and less writes are performed, a cache module is introduced to reduce the DB read pressure and speed up the core scheduling process;
-#### Cache settings
+## Cache Settings
```yaml
spring:
@@ -27,11 +27,11 @@ The cache-module use [spring-cache](https://spring.io/guides/gs/caching/), so yo
With the config of [caffeine](https://github.com/ben-manes/caffeine), you can set the cache size, expire time, etc.
-#### Cache Read
+## Cache Read
The cache adopts the annotation `@Cacheable` of spring-cache and is configured in the mapper layer. For example: `TenantMapper`.
-#### Cache Evict
+## Cache Evict
The business data update comes from the api-server, and the cache end is in the master-server. So it is necessary to monitor the data update of the api-server (aspect intercept `@CacheEvict`), and the master-server will be notified when the cache eviction is required.
diff --git a/docs/en-us/2.0.3/user_doc/architecture/configuration.md b/docs/en-us/2.0.3/user_doc/architecture/configuration.md
index b40201d..b4ba603 100644
--- a/docs/en-us/2.0.3/user_doc/architecture/configuration.md
+++ b/docs/en-us/2.0.3/user_doc/architecture/configuration.md
@@ -1,14 +1,17 @@
<!-- markdown-link-check-disable -->
+# Configuration
+
+## Preface
-# Preface
This document explains the DolphinScheduler application configurations according to DolphinScheduler-1.3.x versions.
-# Directory Structure
+## Directory Structure
+
Currently, all the configuration files are under [conf ] directory. Please check the following simplified DolphinScheduler installation directories to have a direct view about the position [conf] directory in and configuration files inside. This document only describes DolphinScheduler configurations and other modules are not going into.
[Note: the DolphinScheduler (hereinafter called the ‘DS’) .]
-```
+```
├─bin DS application commands directory
│ ├─dolphinscheduler-daemon.sh startup/shutdown DS application
│ ├─start-all.sh A startup all DS services with configurations
@@ -43,16 +46,13 @@ Currently, all the configuration files are under [conf ] directory. Please check
│ ├─upgrade-dolphinscheduler.sh DS database upgrade script
│ ├─monitor-server.sh DS monitor-server start script
│ ├─scp-hosts.sh transfer installation files script
-│ ├─remove-zk-node.sh cleanup zookeeper caches script
+│ ├─remove-zk-node.sh cleanup ZooKeeper caches script
├─ui front-end web resources directory
├─lib DS .jar dependencies directory
├─install.sh auto-setup DS services script
-
-
```
-
-# Configurations in Details
+## Configurations in Details
serial number| service classification| config file|
|--|--|--|
@@ -70,8 +70,9 @@ serial number| service classification| config file|
12|services log config files|API-service log config : logback-api.xml <br /> master-service log config : logback-master.xml <br /> worker-service log config : logback-worker.xml <br /> alert-service log config : logback-alert.xml
-## 1.dolphinscheduler-daemon.sh [startup/shutdown DS application]
-dolphinscheduler-daemon.sh is responsible for DS startup & shutdown.
+### dolphinscheduler-daemon.sh [startup/shutdown DS application]
+
+dolphinscheduler-daemon.sh is responsible for DS startup and shutdown.
Essentially, start-all.sh/stop-all.sh startup/shutdown the cluster via dolphinscheduler-daemon.sh.
Currently, DS just makes a basic config, please config further JVＭ options based on your practical situation of resources.
@@ -92,7 +93,8 @@ export DOLPHINSCHEDULER_OPTS="
> "-XX:DisableExplicitGC" is not recommended due to may lead to memory link (DS dependent on Netty to communicate).
-## 2.datasource.properties [datasource config properties]
+### datasource.properties [datasource config properties]
+
DS uses Druid to manage database connections and default simplified configs are:
|Parameters | Default value| Description|
|--|--|--|
@@ -118,12 +120,13 @@ spring.datasource.poolPreparedStatements|true| Open PSCache
spring.datasource.maxPoolPreparedStatementPerConnectionSize|20| specify the size of PSCache on each connection
-## 3.registry.properties [registry config properties, default is zookeeper]
+### registry.properties [registry config properties, default is zookeeper]
+
|Parameters | Default value| Description|
|--|--|--|
registry.plugin.name|zookeeper| plugin name
-registry.servers|localhost:2181| zookeeper cluster connection info
-registry.namespace|dolphinscheduler| DS is stored under zookeeper root directory(Start without /)
+registry.servers|localhost:2181| ZooKeeper cluster connection info
+registry.namespace|dolphinscheduler| DS is stored under ZooKeeper root directory(Start without /)
registry.base.sleep.time.ms|60| time to wait between subsequent retries
registry.max.sleep.ms|300| maximum time to wait between subsequent retries
registry.max.retries|5| maximum retry times
@@ -131,7 +134,8 @@ registry.session.timeout.ms|30000| session timeout
registry.connection.timeout.ms|7500| connection timeout
-## 4.common.properties [hadoop、s3、yarn config properties]
+### common.properties [hadoop、s3、yarn config properties]
+
Currently, common.properties mainly configures hadoop/s3a related configurations.
|Parameters | Default value| Description|
|--|--|--|
@@ -155,7 +159,8 @@ dolphinscheduler.env.path|env/dolphinscheduler_env.sh|load environment variables
development.state|false| specify whether in development state
-## 5.application-api.properties [API-service log config]
+### application-api.properties [API-service log config]
+
|Parameters | Default value| Description|
|--|--|--|
server.port|12345|api service communication port
@@ -170,7 +175,8 @@ spring.messages.basename|i18n/messages| i18n config
security.authentication.type|PASSWORD| authentication type
-## 6.master.properties [master-service log config]
+### master.properties [master-service log config]
+
|Parameters | Default value| Description|
|--|--|--|
master.listen.port|5678|master listen port
@@ -185,7 +191,8 @@ master.max.cpuload.avg|-1|master max CPU load avg, only higher than the system C
master.reserved.memory|0.3|master reserved memory, only lower than system available memory, master server can schedule. default value 0.3, the unit is G
-## 7.worker.properties [worker-service log config]
+### worker.properties [worker-service log config]
+
|Parameters | Default value| Description|
|--|--|--|
worker.listen.port|1234|worker listen port
@@ -196,7 +203,8 @@ worker.reserved.memory|0.3|worker reserved memory, only lower than system availa
worker.groups|default|worker groups separated by comma, like 'worker.groups=default,test' <br> worker will join corresponding group according to this config when startup
-## 8.alert.properties [alert-service log config]
+### alert.properties [alert-service log config]
+
|Parameters | Default value| Description|
|--|--|--|
alert.type|EMAIL|alter type|
@@ -223,7 +231,8 @@ enterprise.wechat.team.send.msg||group message format
plugin.dir|/Users/xx/your/path/to/plugin/dir|plugin directory
-## 9.quartz.properties [quartz config properties]
+### quartz.properties [quartz config properties]
+
This part describes quartz configs and please configure them based on your practical situation and resources.
|Parameters | Default value| Description|
|--|--|--|
@@ -247,18 +256,20 @@ org.quartz.jobStore.dataSource | myDs
org.quartz.dataSource.myDs.connectionProvider.class | org.apache.dolphinscheduler.service.quartz.DruidConnectionProvider
-## 10.install_config.conf [DS environment variables configuration script[install/start DS]]
+### install_config.conf [DS environment variables configuration script[install/start DS]]
+
install_config.conf is a bit complicated and is mainly used in the following two places.
-* 1.DS cluster auto installation
+* DS Cluster Auto Installation
> System will load configs in the install_config.conf and auto-configure files below, based on the file content when executing 'install.sh'.
> Files such as dolphinscheduler-daemon.sh、datasource.properties、registry.properties、common.properties、application-api.properties、master.properties、worker.properties、alert.properties、quartz.properties and etc.
-* 2.Startup/shutdown DS cluster
+* Startup/Shutdown DS Cluster
> The system will load masters, workers, alertServer, apiServers and other parameters inside the file to startup/shutdown DS cluster.
-File content as follows:
+#### File Content as Follows:
+
```bash
# Note: please escape the character if the file contains special characters such as `.*[]^${}\+?|()@#&`.
@@ -267,7 +278,7 @@ File content as follows:
# Database type (DS currently only supports PostgreSQL and MySQL)
dbtype="mysql"
-# Database url & port
+# Database url and port
dbhost="192.168.xx.xx:3306"
# Database name
@@ -280,7 +291,7 @@ username="xx"
# Database password
password="xx"
-# Zookeeper url
+# ZooKeeper url
zkQuorum="192.168.xx.xx:2181,192.168.xx.xx:2181,192.168.xx.xx:2181"
# DS installation path, such as '/data1_1T/dolphinscheduler'
@@ -382,7 +393,8 @@ alertServer="ds3"
apiServers="ds1"
```
-## 11.dolphinscheduler_env.sh [load environment variables configs]
+### 11.dolphinscheduler_env.sh [load environment variables configs]
+
When using shell to commit tasks, DS will load environment variables inside dolphinscheduler_env.sh into the host.
Types of tasks involved are: Shell task、Python task、Spark task、Flink task、Datax task and etc.
```bash
@@ -400,7 +412,8 @@ export PATH=$HADOOP_HOME/bin:$SPARK_HOME1/bin:$SPARK_HOME2/bin:$PYTHON_HOME:$JAV
```
-## 12. Services logback configs
+### 12. Services logback configs
+
Services name| logback config name |
--|--|
API-service logback config |logback-api.xml|
diff --git a/docs/en-us/2.0.3/user_doc/architecture/design.md b/docs/en-us/2.0.3/user_doc/architecture/design.md
index de3a41a..c3f4d92 100644
--- a/docs/en-us/2.0.3/user_doc/architecture/design.md
+++ b/docs/en-us/2.0.3/user_doc/architecture/design.md
@@ -1,11 +1,11 @@
-## System Architecture Design
+# System Architecture Design
Before explaining the architecture of the scheduling system, let's first understand the commonly used terms of the
scheduling system
-### 1.System Structure
+## System Structure
-#### 1.1 System architecture diagram
+### System Architecture Diagram
<p align="center">
<img src="/img/architecture-1.3.0.jpg" alt="System architecture diagram" width="70%" />
@@ -14,7 +14,7 @@ scheduling system
</p>
</p>
-#### 1.2 Start process activity diagram
+### Start Process Activity Diagram
<p align="center">
<img src="/img/master-process-2.0-en.png" alt="Start process activity diagram" width="70%" />
@@ -23,17 +23,17 @@ scheduling system
</p>
</p>
-#### 1.3 Architecture description
+### Architecture Description
* **MasterServer**
MasterServer adopts a distributed and centerless design concept. MasterServer is mainly responsible for DAG task
segmentation, task submission monitoring, and monitoring the health status of other MasterServer and WorkerServer at
- the same time. When the MasterServer service starts, register a temporary node with Zookeeper, and perform fault
- tolerance by monitoring changes in the temporary node of Zookeeper. MasterServer provides monitoring services based on
+ the same time. When the MasterServer service starts, register a temporary node with ZooKeeper, and perform fault
+ tolerance by monitoring changes in the temporary node of ZooKeeper. MasterServer provides monitoring services based on
netty.
- ##### The service mainly includes:
+ #### The Service Mainly Includes:
- **MasterSchedulerService** is a scanning thread that scans the **command** table in the database regularly,
generates workflow instances, and performs different business operations according to different **command types**
@@ -48,9 +48,9 @@ scheduling system
* **WorkerServer**
WorkerServer also adopts a distributed centerless design concept, supports custom task plug-ins, and is mainly responsible for task execution and log services.
- When the WorkerServer service starts, it registers a temporary node with Zookeeper and maintains a heartbeat.
+ When the WorkerServer service starts, it registers a temporary node with ZooKeeper and maintains a heartbeat.
-##### The service mainly includes
+ #### The Service Mainly Includes
- **WorkerManagerThread** mainly receives tasks sent by the master through netty, and calls **TaskExecuteThread** corresponding executors according to different task types.
@@ -60,7 +60,7 @@ scheduling system
* **Registry**
- The registry is implemented as a plug-in, and Zookeeper is supported by default. The MasterServer and WorkerServer
+ The registry is implemented as a plug-in, and ZooKeeper is supported by default. The MasterServer and WorkerServer
nodes in the system use the registry for cluster management and fault tolerance. In addition, the system also performs
event monitoring and distributed locks based on the registry.
@@ -80,11 +80,11 @@ scheduling system
The front-end page of the system provides various visual operation interfaces of the system,See more
at [Introduction to Functions](../guide/homepage.md) section。
-#### 1.4 Architecture design ideas
+### Architecture Design Ideas
-##### One、Decentralization VS centralization
+#### Decentralization VS Centralization
-###### Centralized thinking
+##### Centralized Thinking
The centralized design concept is relatively simple. The nodes in the distributed cluster are divided into roles
according to roles, which are roughly divided into two roles:
@@ -93,24 +93,24 @@ according to roles, which are roughly divided into two roles:
<img src="https://analysys.github.io/easyscheduler_docs_cn/images/master_slave.png" alt="master-slave character" width="50%" />
</p>
-- The role of the master is mainly responsible for task distribution and monitoring the health status of the slave, and
+ - The role of the master is mainly responsible for task distribution and monitoring the health status of the slave, and
can dynamically balance the task to the slave, so that the slave node will not be in a "busy dead" or "idle dead"
state.
-- The role of Worker is mainly responsible for task execution and maintenance and Master's heartbeat, so that Master can
+ - The role of Worker is mainly responsible for task execution and maintenance and Master's heartbeat, so that Master can
assign tasks to Slave.
Problems in centralized thought design:
-- Once there is a problem with the Master, the dragons are headless and the entire cluster will collapse. In order to
+ - Once there is a problem with the Master, the dragons are headless and the entire cluster will collapse. In order to
solve this problem, most of the Master/Slave architecture models adopt the design scheme of active and standby Master,
which can be hot standby or cold standby, or automatic switching or manual switching, and more and more new systems
are beginning to have The ability to automatically elect and switch Master to improve the availability of the system.
-- Another problem is that if the Scheduler is on the Master, although it can support different tasks in a DAG running on
+ - Another problem is that if the Scheduler is on the Master, although it can support different tasks in a DAG running on
different machines, it will cause the Master to be overloaded. If the Scheduler is on the slave, all tasks in a DAG
can only submit jobs on a certain machine. When there are more parallel tasks, the pressure on the slave may be
greater.
-###### Decentralized
+##### Decentralized
<p align="center">
<img src="https://analysys.github.io/easyscheduler_docs_cn/images/decentralization.png" alt="Decentralization" width="50%" />
@@ -129,31 +129,30 @@ Problems in centralized thought design:
managers" To preside over the work. The most typical case is Etcd implemented by ZooKeeper and Go language.
-- The decentralization of DolphinScheduler is that the Master/Worker is registered in Zookeeper to realize the
+- The decentralization of DolphinScheduler is that the Master/Worker is registered in ZooKeeper to realize the
non-centralization of the Master cluster and the Worker cluster. The sharding mechanism is used to fairly distribute
the workflow for execution on the master, and tasks are sent to the workers for execution through different sending
strategies. Specific task
-##### Second, the master execution process
+#### The Master Execution Process
1. DolphinScheduler uses the sharding algorithm to modulate the command and assigns it according to the sort id of the
master. The master converts the received command into a workflow instance, and uses the thread pool to process the
workflow instance
2. DolphinScheduler's process of workflow:
-
-- Start the workflow through UI or API calls, and persist a command to the database
-- The Master scans the Command table through the sharding algorithm, generates a workflow instance ProcessInstance, and
+ - Start the workflow through UI or API calls, and persist a command to the database
+ - The Master scans the Command table through the sharding algorithm, generates a workflow instance ProcessInstance, and
deletes the Command data at the same time
-- The Master uses the thread pool to run WorkflowExecuteThread to execute the process of the workflow instance,
+ - The Master uses the thread pool to run WorkflowExecuteThread to execute the process of the workflow instance,
including building DAG, creating task instance TaskInstance, and sending TaskInstance to worker through netty
-- After the worker receives the task, it modifies the task status and returns the execution information to the Master
-- The Master receives the task information, persists it to the database, and stores the state change event in the
+ - After the worker receives the task, it modifies the task status and returns the execution information to the Master
+ - The Master receives the task information, persists it to the database, and stores the state change event in the
EventExecuteService event queue
-- EventExecuteService calls WorkflowExecuteThread according to the event queue to submit subsequent tasks and modify
+ - EventExecuteService calls WorkflowExecuteThread according to the event queue to submit subsequent tasks and modify
workflow status
-##### Three、Insufficient thread loop waiting problem
+#### Insufficient Thread Loop Waiting Problem
- If there is no sub-process in a DAG, if the number of data in the Command is greater than the threshold set by the
thread pool, the process directly waits or fails.
@@ -179,12 +178,12 @@ note: The Master Scheduler thread is executed by FIFO when acquiring the Command
So we chose the third way to solve the problem of insufficient threads.
-##### Four、Fault-tolerant design
+#### Fault-Tolerant Design
Fault tolerance is divided into service downtime fault tolerance and task retry, and service downtime fault tolerance is
divided into master fault tolerance and worker fault tolerance.
-###### 1. Downtime fault tolerance
+##### Downtime Fault Tolerance
The service fault-tolerance design relies on ZooKeeper's Watcher mechanism, and the implementation principle is shown in the figure:
@@ -221,7 +220,7 @@ Fault-tolerant post-processing: Once the Master Scheduler thread finds that the
Note: Due to "network jitter", the node may lose its heartbeat with ZooKeeper in a short period of time, and the node's remove event may occur. For this situation, we use the simplest way, that is, once the node and ZooKeeper timeout connection occurs, then directly stop the Master or Worker service.
-###### 2.Task failed and try again
+##### Task Failed and Try Again
Here we must first distinguish the concepts of task failure retry, process failure recovery, and process failure rerun:
@@ -246,7 +245,7 @@ supported. But the tasks in the logical node support retry.
If there is a task failure in the workflow that reaches the maximum number of retries, the workflow will fail to stop,
and the failed workflow can be manually rerun or process recovery operation
-##### Five、Task priority design
+#### Task Priority Design
In the early scheduling design, if there is no priority design and the fair scheduling design is used, the task
submitted first may be completed at the same time as the task submitted later, and the process or task priority cannot
@@ -272,7 +271,7 @@ be set, so We have redesigned this, and our current design is as follows:
<img src="https://analysys.github.io/easyscheduler_docs_cn/images/task_priority.png" alt="Task priority configuration" width="35%" />
</p>
-##### Six、Logback and netty implement log access
+#### Logback and Netty Implement Log Access
- Since Web (UI) and Worker are not necessarily on the same machine, viewing the log cannot be like querying a local
file. There are two options:
@@ -290,7 +289,7 @@ be set, so We have redesigned this, and our current design is as follows:
file.
- FileAppender is mainly implemented as follows：
- ```java
+```java
/**
* task log appender
*/
@@ -314,7 +313,7 @@ be set, so We have redesigned this, and our current design is as follows:
super.subAppend(event);
}
}
-
+```
Generate logs in the form of /process definition id/process instance id/task instance id.log
@@ -322,7 +321,7 @@ Generate logs in the form of /process definition id/process instance id/task ins
- TaskLogFilter is implemented as follows：
- ```java
+```java
/**
* task log filter
*/
@@ -336,4 +335,4 @@ public class TaskLogFilter extends Filter<ILoggingEvent> {
return FilterReply.DENY;
}
}
-
+```
diff --git a/docs/en-us/2.0.3/user_doc/architecture/designplus.md b/docs/en-us/2.0.3/user_doc/architecture/designplus.md
index 541d572..b8588d7 100644
--- a/docs/en-us/2.0.3/user_doc/architecture/designplus.md
+++ b/docs/en-us/2.0.3/user_doc/architecture/designplus.md
@@ -1,9 +1,9 @@
-## System Architecture Design
+# System Architecture Design
Before explaining the architecture of the scheduling system, let's first understand the commonly used terms of the
scheduling system
-### 1.Glossary
+## Glossary
**DAG：** The full name is Directed Acyclic Graph, referred to as DAG. Task tasks in the workflow are assembled in the
form of a directed acyclic graph, and topological traversal is performed from nodes with zero degrees of entry until
@@ -52,7 +52,7 @@ process fails and ends
**Complement**: Supplement historical data，Supports **interval parallel and serial** two complement methods
-### 2.Module introduction
+## Module Introduction
- dolphinscheduler-alert alarm module, providing AlertServer service.
@@ -66,12 +66,12 @@ process fails and ends
- dolphinscheduler-server MasterServer and WorkerServer services
-- dolphinscheduler-service service module, including Quartz, Zookeeper, log client access service, easy to call server
+- dolphinscheduler-service service module, including Quartz, ZooKeeper, log client access service, easy to call server
module and api module
- dolphinscheduler-ui front-end module
-### Sum up
+## Sum Up
From the perspective of scheduling, this article preliminarily introduces the architecture principles and implementation
ideas of the big data distributed workflow scheduling system-DolphinScheduler. To be continued
diff --git a/docs/en-us/2.0.3/user_doc/architecture/load-balance.md b/docs/en-us/2.0.3/user_doc/architecture/load-balance.md
index 33a8330..e21abba 100644
--- a/docs/en-us/2.0.3/user_doc/architecture/load-balance.md
+++ b/docs/en-us/2.0.3/user_doc/architecture/load-balance.md
@@ -1,10 +1,8 @@
-### Load Balance
+# Load Balance
Load balancing refers to the reasonable allocation of server pressure through routing algorithms (usually in cluster environments) to achieve the maximum optimization of server performance.
-
-
-### DolphinScheduler-Worker load balancing algorithms
+## DolphinScheduler-Worker Load Balancing Algorithms
DolphinScheduler-Master allocates tasks to workers, and by default provides three algorithms:
@@ -18,35 +16,35 @@ The default configuration is the linear load.
As the routing is done on the client side, the master service, you can change master.host.selector in master.properties to configure the algorithm what you want.
-eg: master.host.selector = random (case-insensitive)
+e.g. master.host.selector = random (case-insensitive)
-### Worker load balancing configuration
+## Worker Load Balancing Configuration
The configuration file is worker.properties
-#### weight
+### Weight
All of the above load algorithms are weighted based on weights, which affect the outcome of the triage. You can set different weights for different machines by modifying the worker.weight value.
-#### Preheating
+### Preheating
With JIT optimisation in mind, we will let the worker run at low power for a period of time after startup so that it can gradually reach its optimal state, a process we call preheating. If you are interested, you can read some articles about JIT.
So the worker will gradually reach its maximum weight over time after it starts (by default ten minutes, we don't provide a configuration item, you can change it and submit a PR if needed).
-### Load balancing algorithm breakdown
+## Load Balancing Algorithm Breakdown
-#### Random (weighted)
+### Random (Weighted)
This algorithm is relatively simple, one of the matched workers is selected at random (the weighting affects his weighting).
-#### Smoothed polling (weighted)
+### Smoothed Polling (Weighted)
An obvious drawback of the weighted polling algorithm. Namely, under certain specific weights, weighted polling scheduling generates an uneven sequence of instances, and this unsmoothed load may cause some instances to experience transient high loads, leading to a risk of system downtime. To address this scheduling flaw, we provide a smooth weighted polling algorithm.
Each worker is given two weights, weight (which remains constant after warm-up is complete) and current_weight (which changes dynamically), for each route. The current_weight + weight is iterated over all the workers, and the weight of all the workers is added up and counted as total_weight, then the worker with the largest current_weight is selected as the worker for this task. current_weight-total_weight.
-#### Linear weighting (default algorithm)
+### Linear Weighting (Default Algorithm)
The algorithm reports its own load information to the registry at regular intervals. We base our judgement on two main pieces of information
diff --git a/docs/en-us/2.0.3/user_doc/architecture/metadata.md b/docs/en-us/2.0.3/user_doc/architecture/metadata.md
index 9b66e1e..faf21ab 100644
--- a/docs/en-us/2.0.3/user_doc/architecture/metadata.md
+++ b/docs/en-us/2.0.3/user_doc/architecture/metadata.md
@@ -1,7 +1,9 @@
# Dolphin Scheduler 2.0.3 MetaData
<a name="V5KOl"></a>
-### Dolphin Scheduler 2.0 DB Table Overview
+
+## Dolphin Scheduler 2.0 DB Table Overview
+
| Table Name | Comment |
| :---: | :---: |
| t_ds_access_token | token for access ds backend |
@@ -29,20 +31,25 @@
| t_ds_user | user detail |
| t_ds_version | ds version |
-
---
<a name="XCLy1"></a>
-### E-R Diagram
+
+## E-R Diagram
+
<a name="5hWWZ"></a>
-#### User Queue DataSource
+
+### User Queue DataSource
+
- Multiple users can belong to one tenant
- The queue field in the t_ds_user table stores the queue_name information in the t_ds_queue table, but t_ds_tenant stores queue information using queue_id. During the execution of the process definition, the user queue has the highest priority. If the user queue is empty, the tenant queue is used.
- The user_id field in the t_ds_datasource table indicates the user who created the data source. The user_id in t_ds_relation_datasource_user indicates the user who has permission to the data source.
<a name="7euSN"></a>
-#### Project Resource Alert
+
+### Project Resource Alert
+
- User can have multiple projects, User project authorization completes the relationship binding using project_id and user_id in t_ds_relation_project_user table
@@ -50,7 +57,9 @@
- The user_id in the t_ds_resources table represents the user who created the resource, and the user_id in t_ds_relation_resources_user represents the user who has permissions to the resource
- The user_id in the t_ds_udfs table represents the user who created the UDF, and the user_id in the t_ds_relation_udfs_user table represents a user who has permission to the UDF
<a name="JEw4v"></a>
-#### Command Process Task
+
+### Command Process Task
+
<br />
- A project has multiple process definitions, a process definition can generate multiple process instances, and a process instance can generate multiple task instances
@@ -61,9 +70,13 @@
---
<a name="yd79T"></a>
-### Core Table Schema
+
+## Core Table Schema
+
<a name="6bVhH"></a>
-#### t_ds_process_definition
+
+### t_ds_process_definition
+
| Field | Type | Comment |
| --- | --- | --- |
| id | int | primary key |
@@ -86,7 +99,9 @@
| update_time | datetime | update time |
<a name="t5uxM"></a>
-#### t_ds_process_instance
+
+### t_ds_process_instance
+
| Field | Type | Comment |
| --- | --- | --- |
| id | int | primary key |
@@ -123,7 +138,9 @@
| tenant_id | int | tenant id |
<a name="tHZsY"></a>
-#### t_ds_task_instance
+
+### t_ds_task_instance
+
| Field | Type | Comment |
| --- | --- | --- |
| id | int | primary key |
@@ -150,7 +167,9 @@
| worker_group_id | int | worker group id |
<a name="gLGtm"></a>
-#### t_ds_command
+
+### t_ds_command
+
| Field | Type | Comment |
| --- | --- | --- |
| id | int | primary key |
@@ -168,6 +187,3 @@
| update_time | datetime | update time |
| process_instance_priority | int | process instance priority: 0 Highest,1 High,2 Medium,3 Low,4 Lowest |
| worker_group_id | int | worker group id |
-
-
-
diff --git a/docs/en-us/2.0.3/user_doc/architecture/task-structure.md b/docs/en-us/2.0.3/user_doc/architecture/task-structure.md
index a62f58d..42a5819 100644
--- a/docs/en-us/2.0.3/user_doc/architecture/task-structure.md
+++ b/docs/en-us/2.0.3/user_doc/architecture/task-structure.md
@@ -1,5 +1,7 @@
+# Task Structure
+
+## Overall Tasks Storage Structure
-# Overall Tasks Storage Structure
All tasks created in DolphinScheduler are saved in the t_ds_process_definition table.
The following shows the 't_ds_process_definition' table structure:
@@ -55,9 +57,10 @@ Data example:
}
```
-# The Detailed Explanation of The Storage Structure of Each Task Type
+## The Detailed Explanation of The Storage Structure of Each Task Type
+
+### Shell Nodes
-## Shell Nodes
**The node data structure is as follows:**
No.|parameter name||type|description |notes
-------- | ---------| ---------| -------- | --------- | ---------
@@ -81,7 +84,6 @@ No.|parameter name||type|description |notes
18|workerGroup | |String |Worker group| |
19|preTasks | |Array|preposition tasks | |
-
**Node data example:**
```bash
@@ -131,8 +133,8 @@ No.|parameter name||type|description |notes
```
+### SQL Node
-## SQL Node
Perform data query and update operations on the specified datasource through SQL.
**The node data structure is as follows:**
@@ -168,7 +170,6 @@ No.|parameter name||type|description |note
28|workerGroup | |String |Worker group| |
29|preTasks | |Array|preposition tasks | |
-
**Node data example:**
```bash
@@ -230,12 +231,11 @@ No.|parameter name||type|description |note
}
```
-
-## PROCEDURE [stored procedures] Node
+### Procedure [stored procedures] Node
**The node data structure is as follows:**
**Node data example:**
-## SPARK Node
+### Spark Node
**The node data structure is as follows:**
No.|parameter name||type|description |notes
@@ -271,7 +271,6 @@ No.|parameter name||type|description |notes
29|workerGroup | |String |Worker group| |
30|preTasks | |Array|preposition tasks| |
-
**Node data example:**
```bash
@@ -333,9 +332,8 @@ No.|parameter name||type|description |notes
}
```
+### MapReduce(MR) Node
-
-## MapReduce(MR) Node
**The node data structure is as follows:**
No.|parameter name||type|description |notes
@@ -364,8 +362,6 @@ No.|parameter name||type|description |notes
22|workerGroup | |String |Worker group| |
23|preTasks | |Array|preposition tasks| |
-
-
**Node data example:**
```bash
@@ -420,8 +416,8 @@ No.|parameter name||type|description |notes
}
```
+### Python Node
-## Python Node
**The node data structure is as follows:**
No.|parameter name||type|description |notes
-------- | ---------| ---------| -------- | --------- | ---------
@@ -445,7 +441,6 @@ No.|parameter name||type|description |notes
18|workerGroup | |String |Worker group| |
19|preTasks | |Array|preposition tasks| |
-
**Node data example:**
```bash
@@ -494,10 +489,8 @@ No.|parameter name||type|description |notes
}
```
+### Flink Node
-
-
-## Flink Node
**The node data structure is as follows:**
No.|parameter name||type|description |notes
@@ -531,7 +524,6 @@ No.|parameter name||type|description |notes
27|workerGroup | |String |Worker group| |
38|preTasks | |Array|preposition tasks| |
-
**Node data example:**
```bash
@@ -592,7 +584,8 @@ No.|parameter name||type|description |notes
}
```
-## HTTP Node
+### HTTP Node
+
**The node data structure is as follows:**
No.|parameter name||type|description |notes
@@ -620,7 +613,6 @@ No.|parameter name||type|description |notes
21|workerGroup | |String |Worker group| |
22|preTasks | |Array|preposition tasks| |
-
**Node data example:**
```bash
@@ -677,9 +669,8 @@ No.|parameter name||type|description |notes
}
```
+### DataX Node
-
-## DataX Node
**The node data structure is as follows:**
No.|parameter name||type|description |notes
-------- | ---------| ---------| -------- | --------- | ---------
@@ -713,11 +704,8 @@ No.|parameter name||type|description |notes
28|workerGroup | |String |Worker group| |
29|preTasks | |Array|preposition tasks| |
-
-
**Node data example:**
-
```bash
{
"type":"DATAX",
@@ -768,7 +756,8 @@ No.|parameter name||type|description |notes
}
```
-## Sqoop Node
+### Sqoop Node
+
**The node data structure is as follows:**
No.|parameter name||type|description |notes
-------- | ---------| ---------| -------- | --------- | ---------
@@ -796,9 +785,6 @@ No.|parameter name||type|description |notes
22|workerGroup | |String |Worker group| |
23|preTasks | |Array|preposition tasks| |
-
-
-
**Node data example:**
```bash
@@ -845,7 +831,8 @@ No.|parameter name||type|description |notes
}
```
-## Condition Branch Node
+### Condition Branch Node
+
**The node data structure is as follows:**
No.|parameter name||type|description |notes
-------- | ---------| ---------| -------- | --------- | ---------
@@ -866,7 +853,6 @@ No.|parameter name||type|description |notes
15|workerGroup | |String |Worker group| |
16|preTasks | |Array|preposition tasks| |
-
**Node data example:**
```bash
@@ -909,8 +895,8 @@ No.|parameter name||type|description |notes
}
```
+### Subprocess Node
-## Subprocess Node
**The node data structure is as follows:**
No.|parameter name||type|description |notes
-------- | ---------| ---------| -------- | --------- | ---------
@@ -932,7 +918,6 @@ No.|parameter name||type|description |notes
16|workerGroup | |String |Worker group| |
17|preTasks | |Array|preposition tasks| |
-
**Node data example:**
```bash
@@ -969,9 +954,8 @@ No.|parameter name||type|description |notes
}
```
+### Dependent Node
-
-## DEPENDENT Node
**The node data structure is as follows:**
No.|parameter name||type|description |notes
-------- | ---------| ---------| -------- | --------- | ---------
@@ -997,7 +981,6 @@ No.|parameter name||type|description |notes
20|workerGroup | |String |Worker group| |
21|preTasks | |Array|preposition tasks| |
-
**Node data example:**
```bash
@@ -1128,4 +1111,4 @@ No.|parameter name||type|description |notes
]
}
-```
+```
\ No newline at end of file
diff --git a/docs/en-us/2.0.3/user_doc/guide/alert/alert_plugin_user_guide.md b/docs/en-us/2.0.3/user_doc/guide/alert/alert_plugin_user_guide.md
index fa38d81..a013f8e 100644
--- a/docs/en-us/2.0.3/user_doc/guide/alert/alert_plugin_user_guide.md
+++ b/docs/en-us/2.0.3/user_doc/guide/alert/alert_plugin_user_guide.md
@@ -1,4 +1,6 @@
-## How to create alert plugins and alert groups
+# Alert Component User Guide
+
+## How to Create Alert Plugins and Alert Groups
In version 2.0.3, users need to create alert instances, and then associate them with alert groups, and an alert group can use multiple alert instances, and we will notify them one by one.
diff --git a/docs/en-us/2.0.3/user_doc/guide/alert/enterprise-wechat.md b/docs/en-us/2.0.3/user_doc/guide/alert/enterprise-wechat.md
index 2baea45..4d49c22 100644
--- a/docs/en-us/2.0.3/user_doc/guide/alert/enterprise-wechat.md
+++ b/docs/en-us/2.0.3/user_doc/guide/alert/enterprise-wechat.md
@@ -1,5 +1,7 @@
# Enterprise WeChat
+## How to Create Enterprise WeChat Alert
+
If you need to use Enterprise WeChat to alert, please create an alarm Instance in warning instance manage, and then choose the wechat plugin. The configuration example of enterprise WeChat is as follows
diff --git a/docs/en-us/2.0.3/user_doc/guide/datasource/hive.md b/docs/en-us/2.0.3/user_doc/guide/datasource/hive.md
index 20d86d8..25f2106 100644
--- a/docs/en-us/2.0.3/user_doc/guide/datasource/hive.md
+++ b/docs/en-us/2.0.3/user_doc/guide/datasource/hive.md
@@ -20,7 +20,7 @@
> configure `common.properties`. It is helpful when you try to set env before running HIVE SQL. Parameter
> `support.hive.oneSession` default value is `false` and SQL would run in different session if their more than one.
-## Use HiveServer2 HA Zookeeper
+## Use HiveServer2 HA ZooKeeper
<p align="center">
<img src="/img/hive1-en.png" width="80%" />
diff --git a/docs/en-us/2.0.3/user_doc/guide/datasource/introduction.md b/docs/en-us/2.0.3/user_doc/guide/datasource/introduction.md
index c112812..fc387cb 100644
--- a/docs/en-us/2.0.3/user_doc/guide/datasource/introduction.md
+++ b/docs/en-us/2.0.3/user_doc/guide/datasource/introduction.md
@@ -1,4 +1,3 @@
-
# Data Source
Data source center supports MySQL, POSTGRESQL, HIVE/IMPALA, SPARK, CLICKHOUSE, ORACLE, SQLSERVER and other data sources
diff --git a/docs/en-us/2.0.3/user_doc/guide/datasource/mysql.md b/docs/en-us/2.0.3/user_doc/guide/datasource/mysql.md
index 7807a00..fdd84d0 100644
--- a/docs/en-us/2.0.3/user_doc/guide/datasource/mysql.md
+++ b/docs/en-us/2.0.3/user_doc/guide/datasource/mysql.md
@@ -1,6 +1,5 @@
# MySQL
-
- Data source: select MYSQL
- Data source name: enter the name of the data source
- Description: Enter a description of the data source
diff --git a/docs/en-us/2.0.3/user_doc/guide/datasource/postgresql.md b/docs/en-us/2.0.3/user_doc/guide/datasource/postgresql.md
index 77a4fd7..6b616f8 100644
--- a/docs/en-us/2.0.3/user_doc/guide/datasource/postgresql.md
+++ b/docs/en-us/2.0.3/user_doc/guide/datasource/postgresql.md
@@ -1,4 +1,4 @@
-# POSTGRESQL
+# PostgreSQL
- Data source: select POSTGRESQL
- Data source name: enter the name of the data source
diff --git a/docs/en-us/2.0.3/user_doc/guide/expansion-reduction.md b/docs/en-us/2.0.3/user_doc/guide/expansion-reduction.md
index 49e657e..144cd54 100644
--- a/docs/en-us/2.0.3/user_doc/guide/expansion-reduction.md
+++ b/docs/en-us/2.0.3/user_doc/guide/expansion-reduction.md
@@ -2,14 +2,14 @@
# DolphinScheduler Expansion and Reduction
-## 1. Expansion
+## Expansion
This article describes how to add a new master service or worker service to an existing DolphinScheduler cluster.
```
Attention: There cannot be more than one master service process or worker service process on a physical machine.
If the physical machine where the expansion master or worker node is located has already installed the scheduled service, skip to [1.4 Modify configuration] Edit the configuration file `conf/config/install_config.conf` on **all ** nodes, add masters or workers parameter, and restart the scheduling cluster.
```
-### 1.1 Basic software installation (please install the mandatory items yourself)
+### Basic Software Installation
* [required] [JDK](https://www.oracle.com/technetwork/java/javase/downloads/index.html) (1.8+):Must be installed, please install and configure JAVA_HOME and PATH variables under /etc/profile
* [optional] If the expansion is a worker node, you need to consider whether to install an external client, such as Hadoop, Hive, Spark Client.
@@ -19,7 +19,7 @@ This article describes how to add a new master service or worker service to an e
Attention: DolphinScheduler itself does not depend on Hadoop, Hive, Spark, but will only call their Client for the corresponding task submission.
```
-### 1.2 Get installation package
+### Get Installation Package
- Check which version of DolphinScheduler is used in your existing environment, and get the installation package of the corresponding version, if the versions are different, there may be compatibility problems.
- Confirm the unified installation directory of other nodes, this article assumes that DolphinScheduler is installed in /opt/ directory, and the full path is /opt/dolphinscheduler.
- Please download the corresponding version of the installation package to the server installation directory, uncompress it and rename it to dolphinscheduler and store it in the /opt directory.
@@ -38,7 +38,7 @@ mv apache-dolphinscheduler-2.0.3-bin dolphinscheduler
Attention: The installation package can be copied directly from an existing environment to an expanded physical machine for use.
```
-### 1.3 Create Deployment Users
+### Create Deployment Users
- Create deployment users on **all** expansion machines, and be sure to configure sudo-free. If we plan to deploy scheduling on four expansion machines, ds1, ds2, ds3, and ds4, we first need to create deployment users on each machine
@@ -62,7 +62,7 @@ sed -i 's/Defaults requirett/#Defaults requirett/g' /etc/sudoers
- If resource uploads are used, you also need to assign read and write permissions to the deployment user on `HDFS or MinIO`.
```
-### 1.4 Modify configuration
+### Modify Configuration
- From an existing node such as Master/Worker, copy the conf directory directly to replace the conf directory in the new node. After copying, check if the configuration items are correct.
@@ -126,7 +126,7 @@ workers="existing worker01:default,existing worker02:default,ds3:default,ds4:def
sudo chown -R dolphinscheduler:dolphinscheduler dolphinscheduler
```
-### 1.4. Restart the cluster & verify
+### Restart the Cluster and Verify
- restart the cluster
@@ -182,11 +182,11 @@ If the above services are started normally and the scheduling system page is nor
-----------------------------------------------------------------------------
-## 2. Reduction
+## Reduction
The reduction is to reduce the master or worker services for the existing DolphinScheduler cluster.
There are two steps for shrinking. After performing the following two steps, the shrinking operation can be completed.
-### 2.1 Stop the service on the scaled-down node
+### Stop the Service on the Scaled-Down Node
* If you are scaling down the master node, identify the physical machine where the master service is located, and stop the master service on the physical machine.
* If the worker node is scaled down, determine the physical machine where the worker service is to be scaled down and stop the worker and logger services on the physical machine.
@@ -228,7 +228,7 @@ sh bin/dolphinscheduler-daemon.sh start alert-server # start alert service
If the corresponding master service or worker service does not exist, then the master/worker service is successfully shut down.
-### 2.2 Modify the configuration file
+### Modify the Configuration File
- modify the configuration file `conf/config/install_config.conf` on the **all** nodes, synchronizing the following configuration.
diff --git a/docs/en-us/2.0.3/user_doc/guide/flink-call.md b/docs/en-us/2.0.3/user_doc/guide/flink-call.md
index 2b86d7c..890429d 100644
--- a/docs/en-us/2.0.3/user_doc/guide/flink-call.md
+++ b/docs/en-us/2.0.3/user_doc/guide/flink-call.md
@@ -1,6 +1,6 @@
# Flink Calls Operating steps
-### Create a queue
+## Create a Queue
1. Log in to the scheduling system, click "Security", then click "Queue manage" on the left, and click "Create queue" to create a queue.
2. Fill in the name and value of the queue, and click "Submit"
@@ -12,7 +12,7 @@
-### Create a tenant
+## Create a Tenant
```
1. The tenant corresponds to a Linux user, which the user worker uses to submit jobs. If Linux OS environment does not have this user, the worker will create this user when executing the script.
@@ -27,7 +27,7 @@
-### Create a user
+## Create a User
<p align="center">
<img src="/img/api/create_user.png" width="80%" />
@@ -36,7 +36,7 @@
-### Create a token
+## Create a Token
1. Log in to the scheduling system, click "Security", then click "Token manage" on the left, and click "Create token" to create a token.
@@ -52,7 +52,7 @@
</p>
-### Use token
+## Use Token
1. Open the API documentation page
@@ -80,7 +80,7 @@
-### User authorization
+## User Authorization
<p align="center">
<img src="/img/api/user_authorization.png" width="80%" />
@@ -89,7 +89,7 @@
-### User login
+## User Login
```
http://192.168.1.163:12345/dolphinscheduler/ui/#/monitor/servers/master
@@ -102,7 +102,7 @@ http://192.168.1.163:12345/dolphinscheduler/ui/#/monitor/servers/master
-### Upload the resource
+## Upload the Resource
<p align="center">
<img src="/img/api/upload_resource.png" width="80%" />
@@ -111,7 +111,7 @@ http://192.168.1.163:12345/dolphinscheduler/ui/#/monitor/servers/master
-### Create a workflow
+## Create a Workflow
<p align="center">
<img src="/img/api/create_workflow1.png" width="80%" />
@@ -135,7 +135,7 @@ http://192.168.1.163:12345/dolphinscheduler/ui/#/monitor/servers/master
-### View the execution result
+## View the Execution Result
<p align="center">
<img src="/img/api/execution_result.png" width="80%" />
@@ -144,7 +144,7 @@ http://192.168.1.163:12345/dolphinscheduler/ui/#/monitor/servers/master
-### View log
+## View Log
<p align="center">
<img src="/img/api/log.png" width="80%" />
diff --git a/docs/en-us/2.0.3/user_doc/guide/installation/cluster.md b/docs/en-us/2.0.3/user_doc/guide/installation/cluster.md
index be179f8..ee272d6 100644
--- a/docs/en-us/2.0.3/user_doc/guide/installation/cluster.md
+++ b/docs/en-us/2.0.3/user_doc/guide/installation/cluster.md
@@ -8,11 +8,11 @@ If you are a green hand and want to experience DolphinScheduler, we recommended
Cluster deployment uses the same scripts and configuration files as we deploy in [pseudo-cluster deployment](pseudo-cluster.md), so the prepare and required are the same as pseudo-cluster deployment. The difference is that [pseudo-cluster deployment](pseudo-cluster.md) is for one machine, while cluster deployment (Cluster) for multiple. and the steps of "Modify configuration" are quite different between pseudo-cluster deployment and cluster deployment.
-### Prepare && DolphinScheduler startup environment
+### Prepare and DolphinScheduler Startup Environment
-Because of cluster deployment for multiple machine, so you have to run you "Prepare" and "startup" in every machine in [pseudo-cluster.md](pseudo-cluster.md), except section "Configure machine SSH password-free login", "Start zookeeper", "Initialize the database", which is only for deployment or just need an single server
+Because of cluster deployment for multiple machine, so you have to run you "Prepare" and "startup" in every machine in [pseudo-cluster.md](pseudo-cluster.md), except section "Configure machine SSH password-free login", "Start ZooKeeper", "Initialize the database", which is only for deployment or just need an single server
-### Modify configuration
+### Modify Configuration
This is a step that is quite different from [pseudo-cluster.md](pseudo-cluster.md), because the deployment script will transfer the resources required for installation machine to each deployment machine using `scp`. And we have to declare all machine we want to install DolphinScheduler and then run script `install.sh`. The configuration file is under the path `conf/config/install_config.conf`, here we only need to modify section **INSTALL MACHINE**, **DolphinScheduler ENV, Database, Regi [...]
@@ -31,6 +31,10 @@ apiServers="ds5"
pythonGatewayServers="ds5"
```
-## Start DolphinScheduler && Login DolphinScheduler && Server Start And Stop
+## Start and Login DolphinScheduler
Same as pseudo-cluster.md](pseudo-cluster.md)
+
+## Start and Stop Server
+
+Same as pseudo-cluster.md](pseudo-cluster.md)
\ No newline at end of file
diff --git a/docs/en-us/2.0.3/user_doc/guide/installation/docker.md b/docs/en-us/2.0.3/user_doc/guide/installation/docker.md
index 2e52184..f08b42b 100644
--- a/docs/en-us/2.0.3/user_doc/guide/installation/docker.md
+++ b/docs/en-us/2.0.3/user_doc/guide/installation/docker.md
@@ -5,17 +5,17 @@
- [Docker](https://docs.docker.com/engine/install/) 1.13.1+
- [Docker Compose](https://docs.docker.com/compose/) 1.11.0+
-## How to use this Docker image
+## How to Use this Docker Image
Here're 3 ways to quickly install DolphinScheduler
-### The First Way: Start a DolphinScheduler by docker-compose (recommended)
+### The First Way: Start a DolphinScheduler by Docker Compose (Recommended)
In this way, you need to install [docker-compose](https://docs.docker.com/compose/) as a prerequisite, please install it yourself according to the rich docker-compose installation guidance on the Internet
For Windows 7-10, you can install [Docker Toolbox](https://github.com/docker/toolbox/releases). For Windows 10 64-bit, you can install [Docker Desktop](https://docs.docker.com/docker-for-windows/install/), and pay attention to the [system requirements](https://docs.docker.com/docker-for-windows/install/#system-requirements)
-#### 0. Configure memory not less than 4GB
+#### Configure Memory not Less Than 4GB
For Mac user, click `Docker Desktop -> Preferences -> Resources -> Memory`
@@ -28,11 +28,11 @@ For Windows Docker Desktop user
- **Hyper-V mode**: Click `Docker Desktop -> Settings -> Resources -> Memory`
- **WSL 2 mode**: Refer to [WSL 2 utility VM](https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-global-options-with-wslconfig)
-#### 1. Download the Source Code Package
+#### Download the Source Code Package
Please download the source code package apache-dolphinscheduler-2.0.3-src.tar.gz, download address: [download](/en-us/download/download.html)
-#### 2. Pull Image and Start the Service
+#### Pull Image and Start the Service
> For Mac and Linux user, open **Terminal**
> For Windows Docker Toolbox user, open **Docker Quickstart Terminal**
@@ -50,7 +50,7 @@ $ docker-compose up -d
The **PostgreSQL** (with username `root`, password `root` and database `dolphinscheduler`) and **ZooKeeper** services will start by default
-#### 3. Login
+#### Login
Visit the Web UI: http://localhost:12345/dolphinscheduler (The local address is http://localhost:12345/dolphinscheduler)
@@ -62,21 +62,21 @@ The default username is `admin` and the default password is `dolphinscheduler123
Please refer to the `Quick Start` in the chapter [Quick Start](../quick-start.md) to explore how to use DolphinScheduler
-### The Second Way: Start via specifying the existing PostgreSQL and ZooKeeper service
+### The Second Way: Start via Specifying the Existing PostgreSQL and ZooKeeper Service
In this way, you need to install [docker](https://docs.docker.com/engine/install/) as a prerequisite, please install it yourself according to the rich docker installation guidance on the Internet
-#### 1. Basic Required Software (please install by yourself)
+#### Basic Required Software
- [PostgreSQL](https://www.postgresql.org/download/) (8.2.15+)
- [ZooKeeper](https://zookeeper.apache.org/releases.html) (3.4.6+)
- [Docker](https://docs.docker.com/engine/install/) (1.13.1+)
-#### 2. Please login to the PostgreSQL database and create a database named `dolphinscheduler`
+#### Please Login to the PostgreSQL Database and Create a Database Named `dolphinscheduler`
-#### 3. Initialize the database, import `sql/dolphinscheduler_postgre.sql` to create tables and initial data
+#### Initialize the Database, Import `sql/dolphinscheduler_postgre.sql` to Create Tables and Initial Data
-#### 4. Download the DolphinScheduler Image
+#### Download the DolphinScheduler Image
We have already uploaded user-oriented DolphinScheduler image to the Docker repository so that you can pull the image from the docker repository:
@@ -97,11 +97,11 @@ apache/dolphinscheduler:2.0.3 all
Note: database username test and password test need to be replaced with your actual PostgreSQL username and password, 192.168.x.x need to be replaced with your relate PostgreSQL and ZooKeeper host IP
-#### 6. Login
+#### Login
Same as above
-### The Third Way: Start a standalone DolphinScheduler server
+### The Third Way: Start a Standalone DolphinScheduler Server
The following services are automatically started when the container starts:
@@ -212,7 +212,7 @@ Especially, it can be configured through the environment variable configuration
## FAQ
-### How to manage DolphinScheduler by docker-compose?
+### How to Manage DolphinScheduler by Docker Compose?
Start, restart, stop or list containers:
@@ -235,7 +235,7 @@ Stop containers and remove containers, networks and volumes:
docker-compose down -v
```
-### How to view the logs of a container?
+### How to View the Logs of a Container?
List all running containers:
@@ -252,7 +252,7 @@ docker logs -f docker-swarm_dolphinscheduler-api_1 # follow log output
docker logs --tail 10 docker-swarm_dolphinscheduler-api_1 # show last 10 lines from the end of the logs
```
-### How to scale master and worker by docker-compose?
+### How to Scale Master and Worker by Docker Compose?
Scale master to 2 instances:
@@ -266,7 +266,7 @@ Scale worker to 3 instances:
docker-compose up -d --scale dolphinscheduler-worker=3 dolphinscheduler-worker
```
-### How to deploy DolphinScheduler on Docker Swarm?
+### How to Deploy DolphinScheduler on Docker Swarm?
Assuming that the Docker Swarm cluster has been created (If there is no Docker Swarm cluster, please refer to [create-swarm](https://docs.docker.com/engine/swarm/swarm-tutorial/create-swarm/))
@@ -294,7 +294,7 @@ Remove the volumes of the stack named dolphinscheduler:
docker volume rm -f $(docker volume ls --format "{{.Name}}" | grep -e "^dolphinscheduler")
```
-### How to scale master and worker on Docker Swarm?
+### How to Scale Master and Worker on Docker Swarm?
Scale master of the stack named dolphinscheduler to 2 instances:
@@ -308,9 +308,9 @@ Scale worker of the stack named dolphinscheduler to 3 instances:
docker service scale dolphinscheduler_dolphinscheduler-worker=3
```
-### How to build a Docker image?
+### How to Build a Docker Image?
-#### Build from the source code (Require Maven 3.3+ & JDK 1.8+)
+#### Build from the Source Code (Require Maven 3.3+ and JDK 1.8+)
In Unix-Like, execute in Terminal:
@@ -326,7 +326,7 @@ C:\dolphinscheduler-src>.\docker\build\hooks\build.bat
Please read `./docker/build/hooks/build` `./docker/build/hooks/build.bat` script files if you don't understand
-#### Build from the binary distribution (Not require Maven 3.3+ & JDK 1.8+)
+#### Build from the Binary Distribution (Not require Maven 3.3+ and JDK 1.8+)
Please download the binary distribution package apache-dolphinscheduler-2.0.3-bin.tar.gz, download address: [download](/en-us/download/download.html). And put apache-dolphinscheduler-2.0.3-bin.tar.gz into the `apache-dolphinscheduler-2.0.3-src/docker/build` directory, execute in Terminal or PowerShell:
@@ -337,7 +337,7 @@ $ docker build --build-arg VERSION=2.0.3 -t apache/dolphinscheduler:2.0.3 .
> PowerShell should use `cd apache-dolphinscheduler-2.0.3-src/docker/build`
-#### Build multi-platform images
+#### Build Multi-Platform Images
Currently support to build images including `linux/amd64` and `linux/arm64` platform architecture, requirements:
@@ -351,7 +351,7 @@ $ docker login # login to push apache/dolphinscheduler
$ bash ./docker/build/hooks/build
```
-### How to add an environment variable for Docker?
+### How to Add an Environment Variable for Docker?
If you would like to do additional initialization in an image derived from this one, add one or more environment variables under `/root/start-init-conf.sh`, and modify template files in `/opt/dolphinscheduler/conf/*.tpl`.
@@ -378,7 +378,7 @@ EOF
done
```
-### How to use MySQL as the DolphinScheduler's database instead of PostgreSQL?
+### How to Use MySQL as the DolphinScheduler's Database Instead of PostgreSQL?
> Because of the commercial license, we cannot directly use the driver of MySQL.
>
@@ -424,7 +424,7 @@ DATABASE_PARAMS=useUnicode=true&characterEncoding=UTF-8
8. Run a dolphinscheduler (See **How to use this docker image**)
-### How to support MySQL datasource in `Datasource manage`?
+### How to Support MySQL Datasource in `Datasource manage`?
> Because of the commercial license, we cannot directly use the driver of MySQL.
>
@@ -453,7 +453,7 @@ docker build -t apache/dolphinscheduler:mysql-driver .
6. Add a MySQL datasource in `Datasource manage`
-### How to support Oracle datasource in `Datasource manage`?
+### How to Support Oracle Datasource in `Datasource manage`?
> Because of the commercial license, we cannot directly use the driver of Oracle.
>
@@ -482,7 +482,7 @@ docker build -t apache/dolphinscheduler:oracle-driver .
6. Add an Oracle datasource in `Datasource manage`
-### How to support Python 2 pip and custom requirements.txt?
+### How to Support Python 2 pip and Custom requirements.txt?
1. Create a new `Dockerfile` to install pip:
@@ -515,7 +515,7 @@ docker build -t apache/dolphinscheduler:pip .
5. Verify pip under a new Python task
-### How to support Python 3?
+### How to Support Python 3?
1. Create a new `Dockerfile` to install Python 3:
@@ -548,7 +548,7 @@ docker build -t apache/dolphinscheduler:python3 .
6. Verify Python 3 under a new Python task
-### How to support Hadoop, Spark, Flink, Hive or DataX?
+### How to Support Hadoop, Spark, Flink, Hive or DataX?
Take Spark 2.4.7 as an example:
@@ -602,7 +602,7 @@ Spark on YARN (Deploy Mode is `cluster` or `client`) requires Hadoop support. Si
Ensure that `$HADOOP_HOME` and `$HADOOP_CONF_DIR` exists
-### How to support Spark 3?
+### How to Support Spark 3?
In fact, the way to submit applications with `spark-submit` is the same, regardless of Spark 1, 2 or 3. In other words, the semantics of `SPARK_HOME2` is the second `SPARK_HOME` instead of `SPARK2`'s `HOME`, so just set `SPARK_HOME2=/path/to/spark3`
@@ -639,7 +639,7 @@ $SPARK_HOME2/bin/spark-submit --class org.apache.spark.examples.SparkPi $SPARK_H
Check whether the task log contains the output like `Pi is roughly 3.146015`
-### How to support shared storage between Master, Worker and Api server?
+### How to Support Shared Storage between Master, Worker and Api server?
> **Note**: If it is deployed on a single machine by `docker-compose`, step 1 and 2 can be skipped directly, and execute the command like `docker cp hadoop-3.2.2.tar.gz docker-swarm_dolphinscheduler-worker_1:/opt/soft` to put Hadoop into the shared directory `/opt/soft` in the container
@@ -662,7 +662,7 @@ volumes:
3. Ensure that `$HADOOP_HOME` and `$HADOOP_CONF_DIR` are correct
-### How to support local file resource storage instead of HDFS and S3?
+### How to Support Local File Resource Storage Instead of HDFS and S3?
> **Note**: If it is deployed on a single machine by `docker-compose`, step 2 can be skipped directly
@@ -686,7 +686,7 @@ volumes:
device: ":/path/to/resource/dir"
```
-### How to support S3 resource storage like MinIO?
+### How to Support S3 Resource Storage Like MinIO?
Take MinIO as an example: Modify the following environment variables in `config.env.sh`
@@ -703,7 +703,7 @@ FS_S3A_SECRET_KEY=MINIO_SECRET_KEY
> **Note**: `MINIO_IP` can only use IP instead of the domain name, because DolphinScheduler currently doesn't support S3 path style access
-### How to configure SkyWalking?
+### How to Configure SkyWalking?
Modify SkyWalking environment variables in `config.env.sh`:
@@ -770,13 +770,13 @@ This environment variable sets the database for the database. The default value
**`ZOOKEEPER_QUORUM`**
-This environment variable sets zookeeper quorum. The default value is `127.0.0.1:2181`.
+This environment variable sets ZooKeeper quorum. The default value is `127.0.0.1:2181`.
**Note**: You must specify it when starting a standalone dolphinscheduler server. Like `master-server`, `worker-server`, `api-server`.
**`ZOOKEEPER_ROOT`**
-This environment variable sets zookeeper root directory for dolphinscheduler. The default value is `/dolphinscheduler`.
+This environment variable sets ZooKeeper root directory for dolphinscheduler. The default value is `/dolphinscheduler`.
### Common
diff --git a/docs/en-us/2.0.3/user_doc/guide/installation/hardware.md b/docs/en-us/2.0.3/user_doc/guide/installation/hardware.md
index 0c5df7f..1303276 100644
--- a/docs/en-us/2.0.3/user_doc/guide/installation/hardware.md
+++ b/docs/en-us/2.0.3/user_doc/guide/installation/hardware.md
@@ -2,7 +2,7 @@
DolphinScheduler, as an open-source distributed workflow task scheduling system, can be well deployed and run in Intel architecture server environments and mainstream virtualization environments, and supports mainstream Linux operating system environments.
-## 1. Linux Operating System Version Requirements
+## Linux Operating System Version Requirements
| OS | Version |
| :----------------------- | :----------: |
@@ -14,8 +14,10 @@ DolphinScheduler, as an open-source distributed workflow task scheduling system,
> **Attention:**
>The above Linux operating systems can run on physical servers and mainstream virtualization environments such as VMware, KVM, and XEN.
-## 2. Recommended Server Configuration
+## Recommended Server Configuration
+
DolphinScheduler supports 64-bit hardware platforms with Intel x86-64 architecture. The following recommendation is made for server hardware configuration in a production environment:
+
### Production Environment
| **CPU** | **MEM** | **HD** | **NIC** | **Num** |
@@ -27,7 +29,7 @@ DolphinScheduler supports 64-bit hardware platforms with Intel x86-64 architectu
> - The hard disk size configuration is recommended by more than 50GB. The system disk and data disk are separated.
-## 3. Network Requirements
+## Network Requirements
DolphinScheduler provides the following network port configurations for normal operation:
@@ -41,7 +43,7 @@ DolphinScheduler provides the following network port configurations for normal o
> - MasterServer and WorkerServer do not need to enable communication between the networks. As long as the local ports do not conflict.
> - Administrators can adjust relevant ports on the network side and host-side according to the deployment plan of DolphinScheduler components in the actual environment.
-## 4. Browser Requirements
+## Browser Requirements
DolphinScheduler recommends Chrome and the latest browsers which using Chrome Kernel to access the front-end visual operator page.
diff --git a/docs/en-us/2.0.3/user_doc/guide/installation/kubernetes.md b/docs/en-us/2.0.3/user_doc/guide/installation/kubernetes.md
index 659b019..86fc7c3 100644
--- a/docs/en-us/2.0.3/user_doc/guide/installation/kubernetes.md
+++ b/docs/en-us/2.0.3/user_doc/guide/installation/kubernetes.md
@@ -10,7 +10,7 @@ If you are a green hand and want to experience DolphinScheduler, we recommended
- [Kubernetes](https://kubernetes.io/) 1.12+
- PV provisioner support in the underlying infrastructure
-## Installing the Chart
+## Install the Chart
Please download the source code package apache-dolphinscheduler-2.0.3-src.tar.gz, download address: [download](/en-us/download/download.html)
@@ -69,7 +69,7 @@ The default username is `admin` and the default password is `dolphinscheduler123
Please refer to the `Quick Start` in the chapter [Quick Start](../quick-start.md) to explore how to use DolphinScheduler
-## Uninstalling the Chart
+## Uninstall the Chart
To uninstall/delete the `dolphinscheduler` deployment:
@@ -128,7 +128,7 @@ The configuration file is `values.yaml`, and the [Appendix-Configuration](#appen
## FAQ
-### How to view the logs of a pod container?
+### How to View the Logs of a Pod Container?
List all pods (aka `po`):
@@ -145,7 +145,7 @@ kubectl logs -f dolphinscheduler-master-0 # follow log output
kubectl logs --tail 10 dolphinscheduler-master-0 -n test # show last 10 lines from the end of the logs
```
-### How to scale api, master and worker on Kubernetes?
+### How to Scale api, master and worker on Kubernetes?
List all deployments (aka `deploy`):
@@ -182,7 +182,7 @@ kubectl scale --replicas=6 sts dolphinscheduler-worker
kubectl scale --replicas=6 sts dolphinscheduler-worker -n test # with test namespace
```
-### How to use MySQL as the DolphinScheduler's database instead of PostgreSQL?
+### How to Use MySQL as the DolphinScheduler's Database Instead of PostgreSQL?
> Because of the commercial license, we cannot directly use the driver of MySQL.
>
@@ -225,7 +225,7 @@ externalDatabase:
8. Run a DolphinScheduler release in Kubernetes (See **Installing the Chart**)
-### How to support MySQL datasource in `Datasource manage`?
+### How to Support MySQL Datasource in `Datasource manage`?
> Because of the commercial license, we cannot directly use the driver of MySQL.
>
@@ -254,7 +254,7 @@ docker build -t apache/dolphinscheduler:mysql-driver .
7. Add a MySQL datasource in `Datasource manage`
-### How to support Oracle datasource in `Datasource manage`?
+### How to Support Oracle Datasource in `Datasource manage`?
> Because of the commercial license, we cannot directly use the driver of Oracle.
>
@@ -283,7 +283,7 @@ docker build -t apache/dolphinscheduler:oracle-driver .
7. Add an Oracle datasource in `Datasource manage`
-### How to support Python 2 pip and custom requirements.txt?
+### How to Support Python 2 pip and Custom requirements.txt?
1. Create a new `Dockerfile` to install pip:
@@ -316,7 +316,7 @@ docker build -t apache/dolphinscheduler:pip .
6. Verify pip under a new Python task
-### How to support Python 3?
+### How to Support Python 3?
1. Create a new `Dockerfile` to install Python 3:
@@ -349,7 +349,7 @@ docker build -t apache/dolphinscheduler:python3 .
7. Verify Python 3 under a new Python task
-### How to support Hadoop, Spark, Flink, Hive or DataX?
+### How to Support Hadoop, Spark, Flink, Hive or DataX?
Take Spark 2.4.7 as an example:
@@ -407,7 +407,7 @@ Spark on YARN (Deploy Mode is `cluster` or `client`) requires Hadoop support. Si
Ensure that `$HADOOP_HOME` and `$HADOOP_CONF_DIR` exists
-### How to support Spark 3?
+### How to Support Spark 3?
In fact, the way to submit applications with `spark-submit` is the same, regardless of Spark 1, 2 or 3. In other words, the semantics of `SPARK_HOME2` is the second `SPARK_HOME` instead of `SPARK2`'s `HOME`, so just set `SPARK_HOME2=/path/to/spark3`
@@ -448,7 +448,7 @@ $SPARK_HOME2/bin/spark-submit --class org.apache.spark.examples.SparkPi $SPARK_H
Check whether the task log contains the output like `Pi is roughly 3.146015`
-### How to support shared storage between Master, Worker and Api server?
+### How to Support Shared Storage Between Master, Worker and Api Server?
For example, Master, Worker and API server may use Hadoop at the same time
@@ -473,7 +473,7 @@ common:
3. Ensure that `$HADOOP_HOME` and `$HADOOP_CONF_DIR` are correct
-### How to support local file resource storage instead of HDFS and S3?
+### How to Support Local File Resource Storage Instead of HDFS and S3?
Modify the following configurations in `values.yaml`
@@ -495,7 +495,7 @@ common:
> **Note**: `storageClassName` must support the access mode: `ReadWriteMany`
-### How to support S3 resource storage like MinIO?
+### How to Support S3 Resource Storage Like MinIO?
Take MinIO as an example: Modify the following configurations in `values.yaml`
@@ -514,7 +514,7 @@ common:
> **Note**: `MINIO_IP` can only use IP instead of domain name, because DolphinScheduler currently doesn't support S3 path style access
-### How to configure SkyWalking?
+### How to Configure SkyWalking?
Modify SKYWALKING configurations in `values.yaml`:
@@ -554,14 +554,14 @@ common:
| `externalDatabase.database` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database database will use it | `dolphinscheduler` |
| `externalDatabase.params` | If exists external PostgreSQL, and set `postgresql.enabled` value to false. DolphinScheduler's database params will use it | `characterEncoding=utf8` |
| | | |
-| `zookeeper.enabled` | If not exists external Zookeeper, by default, the DolphinScheduler will use a internal Zookeeper | `true` |
+| `zookeeper.enabled` | If not exists external ZooKeeper, by default, the DolphinScheduler will use a internal Zookeeper | `true` |
| `zookeeper.fourlwCommandsWhitelist` | A list of comma separated Four Letter Words commands to use | `srvr,ruok,wchs,cons` |
| `zookeeper.persistence.enabled` | Set `zookeeper.persistence.enabled` to `true` to mount a new volume for internal Zookeeper | `false` |
| `zookeeper.persistence.size` | `PersistentVolumeClaim` size | `20Gi` |
-| `zookeeper.persistence.storageClass` | Zookeeper data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
+| `zookeeper.persistence.storageClass` | ZooKeeper data persistent volume storage class. If set to "-", storageClassName: "", which disables dynamic provisioning | `-` |
| `zookeeper.zookeeperRoot` | Specify dolphinscheduler root directory in Zookeeper | `/dolphinscheduler` |
-| `externalZookeeper.zookeeperQuorum` | If exists external Zookeeper, and set `zookeeper.enabled` value to false. Specify Zookeeper quorum | `127.0.0.1:2181` |
-| `externalZookeeper.zookeeperRoot` | If exists external Zookeeper, and set `zookeeper.enabled` value to false. Specify dolphinscheduler root directory in Zookeeper | `/dolphinscheduler` |
+| `externalZookeeper.zookeeperQuorum` | If exists external ZooKeeper, and set `zookeeper.enabled` value to false. Specify Zookeeper quorum | `127.0.0.1:2181` |
+| `externalZookeeper.zookeeperRoot` | If exists external ZooKeeper, and set `zookeeper.enabled` value to false. Specify dolphinscheduler root directory in Zookeeper | `/dolphinscheduler` |
| | | |
| `common.configmap.DOLPHINSCHEDULER_OPTS` | The jvm options for dolphinscheduler, suitable for all servers | `""` |
| `common.configmap.DATA_BASEDIR_PATH` | User data directory path, self configuration, please make sure the directory exists and have read write permissions | `/tmp/dolphinscheduler` |
diff --git a/docs/en-us/2.0.3/user_doc/guide/installation/pseudo-cluster.md b/docs/en-us/2.0.3/user_doc/guide/installation/pseudo-cluster.md
index 5b9a42c..242c5b2 100644
--- a/docs/en-us/2.0.3/user_doc/guide/installation/pseudo-cluster.md
+++ b/docs/en-us/2.0.3/user_doc/guide/installation/pseudo-cluster.md
@@ -18,9 +18,9 @@ Pseudo-cluster deployment of DolphinScheduler requires external software support
> **_Note:_** DolphinScheduler itself does not depend on Hadoop, Hive, Spark, but if you need to run tasks that depend on them, you need to have the corresponding environment support
-## DolphinScheduler startup environment
+## DolphinScheduler Startup Environment
-### Configure user exemption and permissions
+### Configure User Exemption and Permissions
Create a deployment user, and be sure to configure `sudo` without password. We here make a example for user dolphinscheduler.
@@ -44,7 +44,7 @@ chown -R dolphinscheduler:dolphinscheduler apache-dolphinscheduler-*-bin
> * Because DolphinScheduler's multi-tenant task switch user by command `sudo -u {linux-user}`, the deployment user needs to have sudo privileges and is password-free. If novice learners don’t understand, you can ignore this point for the time being.
> * If you find the line "Defaults requirest" in the `/etc/sudoers` file, please comment it
-### Configure machine SSH password-free login
+### Configure Machine SSH Password-Free Login
Since resources need to be sent to different machines during installation, SSH password-free login is required between each machine. The steps to configure password-free login are as follows
@@ -58,16 +58,16 @@ chmod 600 ~/.ssh/authorized_keys
> **_Notice:_** After the configuration is complete, you can run the command `ssh localhost` to test if it work or not, if you can login with ssh without password.
-### Start zookeeper
+### Start ZooKeeper
-Go to the zookeeper installation directory, copy configure file `zoo_sample.cfg` to `conf/zoo.cfg`, and change value of dataDir in `conf/zoo.cfg` to `dataDir=./tmp/zookeeper`
+Go to the ZooKeeper installation directory, copy configure file `zoo_sample.cfg` to `conf/zoo.cfg`, and change value of dataDir in `conf/zoo.cfg` to `dataDir=./tmp/zookeeper`
```shell
-# Start zookeeper
+# Start ZooKeeper
./bin/zkServer.sh start
```
-## Modify configuration
+## Modify Configuration
After completing the preparation of the basic environment, you need to modify the configuration file according to your environment. The configuration file is in the path of `conf/config/install_config.conf`. Generally, you just needs to modify the **INSTALL MACHINE, DolphinScheduler ENV, Database, Registry Server** part to complete the deployment, the following describes the parameters that must be modified
@@ -109,11 +109,11 @@ SPRING_DATASOURCE_PASSWORD="dolphinscheduler"
# ---------------------------------------------------------
# Registry Server
# ---------------------------------------------------------
-# Registration center address, the address of zookeeper service
+# Registration center address, the address of ZooKeeper service
registryServers="localhost:2181"
```
-## Initialize the database
+## Initialize the Database
DolphinScheduler metadata is stored in relational database. Currently, PostgreSQL and MySQL are supported. If you use MySQL, you need to manually download [mysql-connector-java driver][mysql] (8.0.16) and move it to the lib directory of DolphinScheduler. Let's take MySQL as an example for how to initialize the database
@@ -150,7 +150,7 @@ sh install.sh
The browser access address http://localhost:12345/dolphinscheduler can login DolphinScheduler UI. The default username and password are **admin/dolphinscheduler123**
-## Start or stop server
+## Start or Stop Server
```shell
# Stop all DolphinScheduler server
diff --git a/docs/en-us/2.0.3/user_doc/guide/installation/standalone.md b/docs/en-us/2.0.3/user_doc/guide/installation/standalone.md
index 9ab7b79..143ca65 100644
--- a/docs/en-us/2.0.3/user_doc/guide/installation/standalone.md
+++ b/docs/en-us/2.0.3/user_doc/guide/installation/standalone.md
@@ -4,7 +4,7 @@ Standalone only for quick look for DolphinScheduler.
If you are a green hand and want to experience DolphinScheduler, we recommended you install follow [Standalone](standalone.md). If you want to experience more complete functions or schedule large tasks number, we recommended you install follow [pseudo-cluster deployment](pseudo-cluster.md). If you want to using DolphinScheduler in production, we recommended you follow [cluster deployment](cluster.md) or [kubernetes](kubernetes.md)
-> **_Note:_** Standalone only recommends the use of less than 20 workflows, because it uses H2 Database, Zookeeper Testing Server, too many tasks may cause instability
+> **_Note:_** Standalone only recommends the use of less than 20 workflows, because it uses H2 Database, ZooKeeper Testing Server, too many tasks may cause instability
## Prepare
@@ -13,7 +13,7 @@ If you are a green hand and want to experience DolphinScheduler, we recommended
## Start DolphinScheduler Standalone Server
-### Extract and start DolphinScheduler
+### Extract and Start DolphinScheduler
There is a standalone startup script in the binary compressed package, which can be quickly started after extract. Switch to a user with sudo permission and run the script
@@ -28,7 +28,7 @@ sh ./bin/dolphinscheduler-daemon.sh start standalone-server
The browser access address http://localhost:12345/dolphinscheduler can login DolphinScheduler UI. The default username and password are **admin/dolphinscheduler123**
-## start/stop server
+### Start or Stop Server
The script `./bin/dolphinscheduler-daemon.sh` can not only quickly start standalone, but also stop the service operation. All the commands are as follows
diff --git a/docs/en-us/2.0.3/user_doc/guide/monitor.md b/docs/en-us/2.0.3/user_doc/guide/monitor.md
index 2bad35e..206aea2 100644
--- a/docs/en-us/2.0.3/user_doc/guide/monitor.md
+++ b/docs/en-us/2.0.3/user_doc/guide/monitor.md
@@ -1,18 +1,17 @@
-
# Monitor
-## Service management
+## Service Management
- Service management is mainly to monitor and display the health status and basic information of each service in the system
-## master monitoring
+## Monitor Master Server
- Mainly related to master information.
<p align="center">
<img src="/img/master-jk-en.png" width="80%" />
</p>
-## worker monitoring
+## Monitor Worker Server
- Mainly related to worker information.
@@ -20,7 +19,7 @@
<img src="/img/worker-jk-en.png" width="80%" />
</p>
-## Zookeeper monitoring
+## Monitor ZooKeeper
- Mainly related configuration information of each worker and master in ZooKeeper.
@@ -28,7 +27,7 @@
<img src="/img/zookeeper-monitor-en.png" width="80%" />
</p>
-## DB monitoring
+## Monitor DB
- Mainly the health of the DB
@@ -36,7 +35,7 @@
<img src="/img/mysql-jk-en.png" width="80%" />
</p>
-## Statistics management
+## Statistics Management
<p align="center">
<img src="/img/statistics-en.png" width="80%" />
@@ -44,5 +43,5 @@
- Number of commands to be executed: statistics on the t_ds_command table
- The number of failed commands: statistics on the t_ds_error_command table
-- Number of tasks to run: Count the data of task_queue in Zookeeper
-- Number of tasks to be killed: Count the data of task_kill in Zookeeper
+- Number of tasks to run: Count the data of task_queue in ZooKeeper
+- Number of tasks to be killed: Count the data of task_kill in ZooKeeper
diff --git a/docs/en-us/2.0.3/user_doc/guide/observability/skywalking-agent.md b/docs/en-us/2.0.3/user_doc/guide/observability/skywalking-agent.md
index cdb4d2d..6d484da 100644
--- a/docs/en-us/2.0.3/user_doc/guide/observability/skywalking-agent.md
+++ b/docs/en-us/2.0.3/user_doc/guide/observability/skywalking-agent.md
@@ -5,11 +5,11 @@ The dolphinscheduler-skywalking module provides [SkyWalking](https://skywalking.
This document describes how to enable SkyWalking 8.4+ support with this module (recommended to use SkyWalking 8.5.0).
-# Installation
+## Installation
The following configuration is used to enable SkyWalking agent.
-### Through environment variable configuration (for Docker Compose)
+### Through Environment Variable Configuration (for Docker Compose)
Modify SkyWalking environment variables in `docker/docker-swarm/config.env.sh`:
@@ -26,7 +26,7 @@ And run
$ docker-compose up -d
```
-### Through environment variable configuration (for Docker)
+### Through Environment Variable Configuration (for Docker)
```shell
$ docker run -d --name dolphinscheduler \
@@ -41,7 +41,7 @@ $ docker run -d --name dolphinscheduler \
apache/dolphinscheduler:2.0.3 all
```
-### Through install_config.conf configuration (for DolphinScheduler install.sh)
+### Through install_config.conf Configuration (for DolphinScheduler install.sh)
Add the following configurations to `${workDir}/conf/config/install_config.conf`.
@@ -59,11 +59,11 @@ skywalkingLogReporterPort="11800"
```
-# Usage
+## Usage
### Import Dashboard
-#### Import DolphinScheduler Dashboard to SkyWalking Sever
+#### Import DolphinScheduler Dashboard to SkyWalking Server
Copy the `${dolphinscheduler.home}/ext/skywalking-agent/dashboard/dolphinscheduler.yml` file into `${skywalking-oap-server.home}/config/ui-initialized-templates/` directory, and restart SkyWalking oap-server.
diff --git a/docs/en-us/2.0.3/user_doc/guide/open-api.md b/docs/en-us/2.0.3/user_doc/guide/open-api.md
index e93737a..dd9436f 100644
--- a/docs/en-us/2.0.3/user_doc/guide/open-api.md
+++ b/docs/en-us/2.0.3/user_doc/guide/open-api.md
@@ -5,7 +5,7 @@ Generally, projects and processes are created through pages, but integration wit
## The Operation Steps of DS API Calls
-### Create a token
+### Create a Token
1. Log in to the scheduling system, click "Security", then click "Token manage" on the left, and click "Create token" to create a token.
<p align="center">
@@ -18,7 +18,7 @@ Generally, projects and processes are created through pages, but integration wit
<img src="/img/create-token-en1.png" width="80%" />
</p>
-### Use token
+### Use Token
1. Open the API documentation page
> Address：http://{api server ip}:12345/dolphinscheduler/doc.html?language=en_US&lang=en
<p align="center">
@@ -36,7 +36,7 @@ Generally, projects and processes are created through pages, but integration wit
<img src="/img/test-api.png" width="80%" />
</p>
-### Create a project
+### Create a Project
Here is an example of creating a project named "wudl-flink-test"：
<p align="center">
<img src="/img/api/create_project1.png" width="80%" />
@@ -52,7 +52,7 @@ Here is an example of creating a project named "wudl-flink-test"：
The returned msg information is "success", indicating that we have successfully created the project through API.
If you are interested in the source code of the project, please continue to read the following：
-### Appendix：The source code of creating a project
+### Appendix：The Source Code of Creating a Project
<p align="center">
<img src="/img/api/create_source1.png" width="80%" />
</p>
diff --git a/docs/en-us/2.0.3/user_doc/guide/parameter/context.md b/docs/en-us/2.0.3/user_doc/guide/parameter/context.md
index b37e97c..b04eebf 100644
--- a/docs/en-us/2.0.3/user_doc/guide/parameter/context.md
+++ b/docs/en-us/2.0.3/user_doc/guide/parameter/context.md
@@ -2,7 +2,7 @@
DolphinScheduler provides the ability to refer to each other between parameters, including: local parameters refer to global parameters, and upstream and downstream parameter transfer. Because of the existence of references, it involves the priority of parameters when the parameter names are the same. see also [Parameter Priority](priority.md)
-## Local task use global parameter
+## Local Task Use Global Parameter
The premise of local tasks referencing global parameters is that you have already defined [Global Parameter](global.md). The usage is similar to the usage in [local parameters](local.md), but the value of the parameter needs to be configured as the key in the global parameter
@@ -10,7 +10,7 @@ The premise of local tasks referencing global parameters is that you have alread
As shown in the figure above, `${biz_date}` and `${biz_curdate}` are examples of local parameters referencing global parameters. Observe the last line of the above figure, local_param_bizdate uses \${global_bizdate} to refer to the global parameter. In the shell script, you can use \${local_param_bizdate} to refer to the value of the global variable global_bizdate, or set the value of local_param_bizdate directly through JDBC. In the same way, local_param refers to the global parameters [...]
-## Pass parameter from upstream task to downstream
+## Pass Parameter from Upstream Task to Downstream
DolphinScheduler Parameter transfer between tasks is allowed, and the current transfer direction only supports one-way transfer from upstream to downstream. The task types currently supporting this feature are：
diff --git a/docs/en-us/2.0.3/user_doc/guide/project/project-list.md b/docs/en-us/2.0.3/user_doc/guide/project/project-list.md
index 37c7b9f..48b3d5e 100644
--- a/docs/en-us/2.0.3/user_doc/guide/project/project-list.md
+++ b/docs/en-us/2.0.3/user_doc/guide/project/project-list.md
@@ -1,6 +1,6 @@
# Project
-## Create project
+## Create Project
- Click "Project Management" to enter the project management page, click the "Create Project" button, enter the project name, project description, and click "Submit" to create a new project.
@@ -8,7 +8,7 @@
<img src="/img/create_project_en1.png" width="80%" />
</p>
-## Project home
+## Project Home
- Click the project name link on the project management page to enter the project home page, as shown in the figure below, the project home page contains the task status statistics, process status statistics, and workflow definition statistics of the project. The introduction for those metric:
diff --git a/docs/en-us/2.0.3/user_doc/guide/project/task-instance.md b/docs/en-us/2.0.3/user_doc/guide/project/task-instance.md
index 6d02cdc..b95c1c1 100644
--- a/docs/en-us/2.0.3/user_doc/guide/project/task-instance.md
+++ b/docs/en-us/2.0.3/user_doc/guide/project/task-instance.md
@@ -1,5 +1,4 @@
-
-## Task instance
+# Task Instance
- Click Project Management -> Workflow -> Task Instance to enter the task instance page, as shown in the figure below, click the name of the workflow instance, you can jump to the workflow instance DAG chart to view the task status.
<p align="center">
diff --git a/docs/en-us/2.0.3/user_doc/guide/project/workflow-definition.md b/docs/en-us/2.0.3/user_doc/guide/project/workflow-definition.md
index ddb9d2f..485c046 100644
--- a/docs/en-us/2.0.3/user_doc/guide/project/workflow-definition.md
+++ b/docs/en-us/2.0.3/user_doc/guide/project/workflow-definition.md
@@ -1,6 +1,6 @@
-# Workflow definition
+# Workflow Definition
-## <span id=creatDag> Create workflow definition</span>
+## <span id=creatDag> Create Workflow Definition</span>
- Click Project Management -> Workflow -> Workflow Definition to enter the workflow definition page, and click the "Create Workflow" button to enter the **workflow DAG edit** page, as shown in the following figure:
<p align="center">
@@ -37,7 +37,7 @@
</p>
> For other types of tasks, please refer to [Task Node Type and Parameter Settings](#TaskParamers).
-## Workflow definition operation function
+## Workflow Definition Operation Function
Click Project Management -> Workflow -> Workflow Definition to enter the workflow definition page, as shown below:
@@ -59,7 +59,7 @@ The operation functions of the workflow definition list are as follows:
<img src="/img/tree_en.png" width="80%" />
</p>
-## <span id=runWorkflow>Run the workflow</span>
+## <span id=runWorkflow>Run the Workflow</span>
- Click Project Management -> Workflow -> Workflow Definition to enter the workflow definition page, as shown in the figure below, click the "Go Online" button <img src="/img/online.png" width="35"/>，Go online workflow.
<p align="center">
@@ -91,7 +91,7 @@ The operation functions of the workflow definition list are as follows:
> Parallel mode: The tasks from May 1 to may 10 are executed simultaneously, and 10 process instances are generated on the process instance page.
-## <span id=creatTiming>Workflow timing</span>
+## <span id=creatTiming>Workflow Timing</span>
- Create timing: Click Project Management->Workflow->Workflow Definition, enter the workflow definition page, go online the workflow, click the "timing" button <img src="/img/timing.png" width="35"/> ,The timing parameter setting dialog box pops up, as shown in the figure below:
<p align="center">
@@ -109,6 +109,6 @@ The operation functions of the workflow definition list are as follows:
<img src="/img/time-manage-list-en.png" width="80%" />
</p>
-## Import workflow
+## Import Workflow
Click Project Management -> Workflow -> Workflow Definition to enter the workflow definition page, click the "Import Workflow" button to import the local workflow file, the workflow definition list displays the imported workflow, and the status is offline.
diff --git a/docs/en-us/2.0.3/user_doc/guide/project/workflow-instance.md b/docs/en-us/2.0.3/user_doc/guide/project/workflow-instance.md
index ac65ebe..1733e7a 100644
--- a/docs/en-us/2.0.3/user_doc/guide/project/workflow-instance.md
+++ b/docs/en-us/2.0.3/user_doc/guide/project/workflow-instance.md
@@ -1,6 +1,6 @@
-# Workflow instance
+# Workflow Instance
-## View workflow instance
+## View Workflow Instance
- Click Project Management -> Workflow -> Workflow Instance to enter the Workflow Instance page, as shown in the figure below:
<p align="center">
@@ -11,7 +11,7 @@
<img src="/img/instance-runs-en.png" width="80%" />
</p>
-## View task log
+## View Task Log
- Enter the workflow instance page, click the workflow name, enter the DAG view page, double-click the task node, as shown in the following figure:
<p align="center">
@@ -22,7 +22,7 @@
<img src="/img/task-log-en.png" width="80%" />
</p>
-## View task history
+## View Task History
- Click Project Management -> Workflow -> Workflow Instance to enter the workflow instance page, and click the workflow name to enter the workflow DAG page;
- Double-click the task node, as shown in the figure below, click "View History" to jump to the task instance page, and display a list of task instances running by the workflow instance
@@ -30,7 +30,7 @@
<img src="/img/task_history_en.png" width="80%" />
</p>
-## View operating parameters
+## View Operating Parameters
- Click Project Management -> Workflow -> Workflow Instance to enter the workflow instance page, and click the workflow name to enter the workflow DAG page;
- Click the icon in the upper left corner <img src="/img/run_params_button.png" width="35"/>，View the startup parameters of the workflow instance; click the icon <img src="/img/global_param.png" width="35"/>，View the global and local parameters of the workflow instance, as shown in the following figure:
@@ -38,7 +38,7 @@
<img src="/img/run_params_en.png" width="80%" />
</p>
-## Workflow instance operation function
+## Workflow Instance Operation Function
Click Project Management -> Workflow -> Workflow Instance to enter the Workflow Instance page, as shown in the figure below:
diff --git a/docs/en-us/2.0.3/user_doc/guide/resource.md b/docs/en-us/2.0.3/user_doc/guide/resource.md
index 58e6fc0..b44047d 100644
--- a/docs/en-us/2.0.3/user_doc/guide/resource.md
+++ b/docs/en-us/2.0.3/user_doc/guide/resource.md
@@ -7,7 +7,7 @@ If you want to use the resource upload function, you can select the local file d
> * If the resource upload function is used, the deployment user in [installation and deployment](installation/standalone.md) must to have operation authority
> * If you using Hadoop cluster with HA, you need to enable HDFS resource upload, and you need to copy the `core-site.xml` and `hdfs-site.xml` under the Hadoop cluster to `/opt/dolphinscheduler/conf`, otherwise Skip step
-## hdfs resource configuration
+## HDFS Resource Configuration
- Upload resource files and udf functions, all uploaded files and resources will be stored on hdfs, so the following configuration items are required:
@@ -42,7 +42,7 @@ conf/common/hadoop.properties
- Only one address needs to be configured for yarn.resourcemanager.ha.rm.ids and yarn.application.status.address, and the other address is empty.
- You need to copy core-site.xml and hdfs-site.xml from the conf directory of the Hadoop cluster to the conf directory of the dolphinscheduler project, and restart the api-server service.
-## File management
+## File Management
> It is the management of various resource files, including creating basic txt/log/sh/conf/py/java and other files, uploading jar packages and other types of files, and can do edit, rename, download, delete and other operations.
@@ -86,9 +86,9 @@ conf/common/hadoop.properties
- delete
> File list -> Click the "Delete" button to delete the specified file
-## UDF management
+## UDF Management
-### Resource management
+### Resource Management
> The resource management and file management functions are similar. The difference is that the resource management is the uploaded UDF function, and the file management uploads the user program, script and configuration file.
> Operation function: rename, download, delete.
@@ -96,7 +96,7 @@ conf/common/hadoop.properties
- Upload udf resources
> Same as uploading files.
-### Function management
+### Function Management
- Create UDF function
> Click "Create UDF Function", enter the udf function parameters, select the udf resource, and click "Submit" to create the udf function.
diff --git a/docs/en-us/2.0.3/user_doc/guide/security.md b/docs/en-us/2.0.3/user_doc/guide/security.md
index bbab492..d4578b5 100644
--- a/docs/en-us/2.0.3/user_doc/guide/security.md
+++ b/docs/en-us/2.0.3/user_doc/guide/security.md
@@ -1,10 +1,9 @@
-
# Security
* Only the administrator account in the security center has the authority to operate. It has functions such as queue management, tenant management, user management, alarm group management, worker group management, token management, etc. In the user management module, resources, data sources, projects, etc. Authorization
* Administrator login, default user name and password: admin/dolphinscheduler123
-## Create queue
+## Create Queue
- Queue is used when the "queue" parameter is needed to execute programs such as spark and mapreduce.
- The administrator enters the Security Center->Queue Management page and clicks the "Create Queue" button to create a queue.
@@ -12,7 +11,7 @@
<img src="/img/create-queue-en.png" width="80%" />
</p>
-## Add tenant
+## Add Tenant
- The tenant corresponds to the Linux user, which is used by the worker to submit the job. Task will fail if Linux does not exists this user. You can set the parameter `worker.tenant.auto.create` as `true` in configuration file `worker.properties`. After that DolphinScheduler would create user if not exists, The property `worker.tenant.auto.create=true` requests worker run `sudo` command without password.
- Tenant Code: **Tenant Code is the only user on Linux and cannot be repeated**
@@ -22,7 +21,7 @@
<img src="/img/addtenant-en.png" width="80%" />
</p>
-## Create normal user
+## Create Normal User
- Users are divided into **administrator users** and **normal users**
@@ -45,7 +44,7 @@
- The administrator enters the Security Center->User Management page and clicks the "Edit" button. When editing user information, enter the new password to modify the user password.
- After a normal user logs in, click the user information in the user name drop-down box to enter the password modification page, enter the password and confirm the password and click the "Edit" button, then the password modification is successful.
-## Create alarm group
+## Create Alarm Group
- The alarm group is a parameter set at startup. After the process ends, the status of the process and other information will be sent to the alarm group in the form of email.
@@ -54,7 +53,7 @@
<p align="center">
<img src="/img/mail-en.png" width="80%" />
-## Token management
+## Token Management
> Since the back-end interface has login check, token management provides a way to perform various operations on the system by calling the interface.
@@ -121,7 +120,7 @@
- Resources, data sources, and UDF function authorization are the same as project authorization.
-## Worker grouping
+## Worker Grouping
Each worker node will belong to its own worker group, and the default group is "default".
diff --git a/docs/en-us/2.0.3/user_doc/guide/task/conditions.md b/docs/en-us/2.0.3/user_doc/guide/task/conditions.md
index 345bee8..d2e262e 100644
--- a/docs/en-us/2.0.3/user_doc/guide/task/conditions.md
+++ b/docs/en-us/2.0.3/user_doc/guide/task/conditions.md
@@ -31,6 +31,6 @@ Drag in the toolbar<img src="/img/conditions.png" width="20"/>The task node to t
- Add the upstream dependency: Use the first parameter to choose task name, and the second parameter for status of the upsteam task.
- Upstream task relationship: we use `and` and `or` operators to handle complex relationship of upstream when multiple upstream tasks for Conditions task
-## Related task
+## Related Task
[switch](switch.md): [Condition](conditions.md)task mainly executes the corresponding branch based on the execution status (success, failure) of the upstream node. The [Switch](switch.md) task mainly executes the corresponding branch based on the value of the [global parameter](../parameter/global.md) and the judgment expression result written by the user.
diff --git a/docs/en-us/2.0.3/user_doc/guide/task/datax.md b/docs/en-us/2.0.3/user_doc/guide/task/datax.md
index f6436bc..a13dec7 100644
--- a/docs/en-us/2.0.3/user_doc/guide/task/datax.md
+++ b/docs/en-us/2.0.3/user_doc/guide/task/datax.md
@@ -1,5 +1,4 @@
-
-# DATAX
+# DataX
- Drag in the toolbar<img src="/img/datax.png" width="35"/>Task node into the drawing board
diff --git a/docs/en-us/2.0.3/user_doc/guide/task/dependent.md b/docs/en-us/2.0.3/user_doc/guide/task/dependent.md
index 97c2940..88868c9 100644
--- a/docs/en-us/2.0.3/user_doc/guide/task/dependent.md
+++ b/docs/en-us/2.0.3/user_doc/guide/task/dependent.md
@@ -1,4 +1,4 @@
-# DEPENDENT
+# Dependent
- Dependent nodes are **dependency check nodes**. For example, process A depends on the successful execution of process B yesterday, and the dependent node will check whether process B has a successful execution yesterday.
diff --git a/docs/en-us/2.0.3/user_doc/guide/task/flink.md b/docs/en-us/2.0.3/user_doc/guide/task/flink.md
index 18c15f0..0900072 100644
--- a/docs/en-us/2.0.3/user_doc/guide/task/flink.md
+++ b/docs/en-us/2.0.3/user_doc/guide/task/flink.md
@@ -4,7 +4,7 @@
Flink task type for executing Flink programs. For Flink nodes, the worker submits the task by using the flink command `flink run`. See [flink cli](https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/cli/) for more details.
-## Create task
+## Create Task
- Click Project Management -> Project Name -> Workflow Definition, and click the "Create Workflow" button to enter the DAG editing page.
- Drag the <img src="/img/tasks/icons/flink.png" width="15"/> from the toolbar to the drawing board.
@@ -42,11 +42,11 @@ Flink task type for executing Flink programs. For Flink nodes, the worker submit
## Task Example
-### Execute the WordCount program
+### Execute the WordCount Program
This is a common introductory case in the Big Data ecosystem, which often applied to computational frameworks such as MapReduce, Flink and Spark. The main purpose is to count the number of identical words in the input text. (Flink's releases come with this example job)
-#### Uploading the main package
+#### Upload the Main Package
When using the Flink task node, you will need to use the Resource Centre to upload the jar package for the executable. Refer to the [resource center](../resource.md).
@@ -54,7 +54,7 @@ After configuring the Resource Centre, you can upload the required target files
-#### Configuring Flink nodes
+#### Configure Flink Nodes
Simply configure the required content according to the parameter descriptions above.
diff --git a/docs/en-us/2.0.3/user_doc/guide/task/http.md b/docs/en-us/2.0.3/user_doc/guide/task/http.md
index 6072e66..d578180 100644
--- a/docs/en-us/2.0.3/user_doc/guide/task/http.md
+++ b/docs/en-us/2.0.3/user_doc/guide/task/http.md
@@ -1,4 +1,3 @@
-
# HTTP
- Drag in the toolbar<img src="/img/http.png" width="35"/>The task node to the drawing board, as shown in the following figure:
diff --git a/docs/en-us/2.0.3/user_doc/guide/task/map-reduce.md b/docs/en-us/2.0.3/user_doc/guide/task/map-reduce.md
index 5fa23ab..7d71e89 100644
--- a/docs/en-us/2.0.3/user_doc/guide/task/map-reduce.md
+++ b/docs/en-us/2.0.3/user_doc/guide/task/map-reduce.md
@@ -8,6 +8,7 @@
- Click Project Management-Project Name-Workflow Definition, and click the "Create Workflow" button to enter the DAG editing page.
- Drag the <img src="/img/tasks/icons/mr.png" width="15"/> from the toolbar to the drawing board.
+
## Task Parameter
- **Node name**: The node name in a workflow definition is unique.
@@ -47,11 +48,11 @@
## Task Example
-### Execute the WordCount program
+### Execute the WordCount Program
This example is a common introductory type of MapReduce application, which is designed to count the number of identical words in the input text.
-#### Uploading the main package
+#### Upload the Main Package
When using the MapReduce task node, you will need to use the Resource Centre to upload the jar package for the executable. Refer to the [resource centre](../resource.md).
@@ -59,7 +60,7 @@ After configuring the Resource Centre, you can upload the required target files
-#### Configuring MapReduce nodes
+#### Configure MapReduce Nodes
Simply configure the required content according to the parameter descriptions above.
diff --git a/docs/en-us/2.0.3/user_doc/guide/task/spark.md b/docs/en-us/2.0.3/user_doc/guide/task/spark.md
index 9543d18..561a457 100644
--- a/docs/en-us/2.0.3/user_doc/guide/task/spark.md
+++ b/docs/en-us/2.0.3/user_doc/guide/task/spark.md
@@ -4,7 +4,7 @@
Spark task type for executing Spark programs. For Spark nodes, the worker submits the task by using the spark command `spark submit`. See [spark-submit](https://spark.apache.org/docs/3.2.1/submitting-applications.html#launching-applications-with-spark-submit) for more details.
-## Create task
+## Create Task
- Click Project Management -> Project Name -> Workflow Definition, and click the "Create Workflow" button to enter the DAG editing page.
- Drag the <img src="/img/tasks/icons/spark.png" width="15"/> from the toolbar to the drawing board.
@@ -39,11 +39,11 @@ Spark task type for executing Spark programs. For Spark nodes, the worker submit
## Task Example
-### Execute the WordCount program
+### Execute the WordCount Program
This is a common introductory case in the Big Data ecosystem, which often applied to computational frameworks such as MapReduce, Flink and Spark. The main purpose is to count the number of identical words in the input text.
-#### Uploading the main package
+#### Upload the Main Package
When using the Spark task node, you will need to use the Resource Center to upload the jar package for the executable. Refer to the [resource center](../resource.md).
@@ -51,7 +51,7 @@ After configuring the Resource Center, you can upload the required target files
-#### Configuring Spark nodes
+#### Configure Spark Nodes
Simply configure the required content according to the parameter descriptions above.
diff --git a/docs/en-us/2.0.3/user_doc/guide/task/sql.md b/docs/en-us/2.0.3/user_doc/guide/task/sql.md
index 4cbb582..af1a505 100644
--- a/docs/en-us/2.0.3/user_doc/guide/task/sql.md
+++ b/docs/en-us/2.0.3/user_doc/guide/task/sql.md
@@ -4,7 +4,7 @@
SQL task, used to connect to database and execute SQL.
-## create data source
+## Create Data Source
Refer to [Data Source](../datasource/introduction.md)
@@ -26,13 +26,13 @@ Refer to [Data Source](../datasource/introduction.md)
## Task Example
-### Create a temporary table in hive and write data
+### Create a Temporary Table in Hive and Write Data
This example creates a temporary table `tmp_hello_world` in hive and write a row of data. Before creating a temporary table, we need to ensure that the table does not exist, so we will use custom parameters to obtain the time of the day as the suffix of the table name every time we run, so that this task can run every day. The format of the created table name is: `tmp_hello_world_{yyyyMMdd}`.
-### After running the task successfully, query the results in hive.
+### After Running the Task Successfully, Query the Results in Hive.
Log in to the bigdata cluster and use 'hive' command or 'beeline' or 'JDBC' and other methods to connect to the 'Apache Hive' for the query. The query SQL is `select * from tmp_hello_world_{yyyyMMdd}`, please replace '{yyyyMMdd}' with the date of the running day. The query screenshot is as follows:
diff --git a/docs/en-us/2.0.3/user_doc/guide/upgrade.md b/docs/en-us/2.0.3/user_doc/guide/upgrade.md
index 0f86c1a..dea6c65 100644
--- a/docs/en-us/2.0.3/user_doc/guide/upgrade.md
+++ b/docs/en-us/2.0.3/user_doc/guide/upgrade.md
@@ -1,18 +1,17 @@
+# DolphinScheduler Upgrade Documentation
-# DolphinScheduler upgrade documentation
+## Back Up Previous Version's Files and Database
-## 1. Back Up Previous Version's Files and Database.
-
-## 2. Stop All Services of DolphinScheduler.
+## Stop All Services of DolphinScheduler
`sh ./script/stop-all.sh`
-## 3. Download the New Version's Installation Package.
+## Download the New Version's Installation Package
- [Download](/en-us/download/download.html) the latest version of the installation packages.
- The following upgrade operations need to be performed in the new version's directory.
-## 4. Database Upgrade
+## Database Upgrade
- Modify the following properties in `conf/config/install_config.conf`.
- If you use MySQL as the database to run DolphinScheduler, please comment out PostgreSQL related configurations, and add mysql connector jar into lib dir, here we download mysql-connector-java-8.0.16.jar, and then correctly config database connect information. You can download mysql connector jar [here](https://downloads.MySQL.com/archives/c-j/). Alternatively, if you use Postgres as database, you just need to comment out Mysql related configurations, and correctly config database conne [...]
@@ -31,9 +30,9 @@ SPRING_DATASOURCE_PASSWORD="dolphinscheduler"
`sh ./script/create-dolphinscheduler.sh`
-## 5. Backend Service Upgrade.
+## Backend Service Upgrade
-### 5.1 Modify the Content in `conf/config/install_config.conf` File.
+### Modify the Content in `conf/config/install_config.conf` File
- Standalone Deployment please refer the [6, Modify running arguments] in [Standalone-Deployment](./installation/standalone.md).
- Cluster Deployment please refer the [6, Modify running arguments] in [Cluster-Deployment](./installation/cluster.md).
@@ -55,7 +54,7 @@ To keep worker group config consistent with the previous version, we need to mod
workers="ds1:service1,ds2:service2,ds3:service2"
```
-### 5.2 Execute Deploy Script.
+### Execute Deploy Script
```shell
`sh install.sh`
```