You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@dolphinscheduler.apache.org by li...@apache.org on 2019/11/29 11:57:09 UTC
[incubator-dolphinscheduler-website] branch master updated: Added
English page for plugin development page,
migrated developer guide to developer,
removed blog system architecture design page and style modification
This is an automated email from the ASF dual-hosted git repository.
lidongdai pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-dolphinscheduler-website.git
The following commit(s) were added to refs/heads/master by this push:
new e7bbc7f Added English page for plugin development page, migrated developer guide to developer, removed blog system architecture design page and style modification
new 4bf11bd Merge pull request #42 from break60/master
e7bbc7f is described below
commit e7bbc7fe2f510767941c34e277206f60934d3913
Author: break60 <79...@qq.com>
AuthorDate: Fri Nov 29 17:32:11 2019 +0800
Added English page for plugin development page, migrated developer guide to developer, removed blog system architecture design page and style modification
---
.../architecture-design.md | 0
.../backend-development.md | 0
.../frontend-development.md | 0
docs/en-us/development/plugin-development.md | 54 +++++++++++++++
.../architecture-design.md | 78 ++++++++++++++++++++++
.../backend-development.md | 0
.../frontend-development.md | 0
.../plugin-development.md | 0
md_json/docs.json | 39 ++++++-----
site_config/blog.js | 14 ----
site_config/development.js | 52 +++++++++++++++
site_config/docs.js | 48 -------------
site_config/site.js | 4 +-
src/components/header/index.jsx | 20 +++---
src/pages/blogDetail/index.scss | 2 +-
15 files changed, 217 insertions(+), 94 deletions(-)
diff --git a/docs/en-us/developer_guide/architecture-design.md b/docs/en-us/development/architecture-design.md
similarity index 100%
rename from docs/en-us/developer_guide/architecture-design.md
rename to docs/en-us/development/architecture-design.md
diff --git a/docs/en-us/developer_guide/backend-development.md b/docs/en-us/development/backend-development.md
similarity index 100%
rename from docs/en-us/developer_guide/backend-development.md
rename to docs/en-us/development/backend-development.md
diff --git a/docs/en-us/developer_guide/frontend-development.md b/docs/en-us/development/frontend-development.md
similarity index 100%
rename from docs/en-us/developer_guide/frontend-development.md
rename to docs/en-us/development/frontend-development.md
diff --git a/docs/en-us/development/plugin-development.md b/docs/en-us/development/plugin-development.md
new file mode 100644
index 0000000..eda2d82
--- /dev/null
+++ b/docs/en-us/development/plugin-development.md
@@ -0,0 +1,54 @@
+## Task Plugin Development
+
+Remind:Currently, task plugin development does not support hot deployment.
+
+### Shell-based tasks
+
+#### YARN-based calculations (see MapReduceTask)
+
+- Need to be **cn.dolphinscheduler.server.worker.task** Down **TaskManager** Create a custom task in the class (also need to register the corresponding task type in TaskType)
+- Need to inherit**cn.dolphinscheduler.server.worker.task** Down **AbstractYarnTask**
+- Constructor Scheduling **AbstractYarnTask** Construction method
+- Inherit **AbstractParameters** Custom task parameter entity
+- Rewrite **AbstractTask** of **init** Parsing in method**Custom task parameters**
+- Rewrite **buildCommand** Encapsulation command
+
+
+
+#### Non-YARN-based calculations (see ShellTask)
+- Need to be **cn.dolphinscheduler.server.worker.task** Down **TaskManager** A custom task
+
+- Need to inherit**cn.dolphinscheduler.server.worker.task** Down **AbstractTask**
+
+- Instantiation in constructor **ShellCommandExecutor**
+
+ ```
+ public ShellTask(TaskProps props, Logger logger) {
+ super(props, logger);
+
+ this.taskDir = props.getTaskDir();
+
+ this.processTask = new ShellCommandExecutor(this::logHandle,
+ props.getTaskDir(), props.getTaskAppId(),
+ props.getTenantCode(), props.getEnvFile(), props.getTaskStartTime(),
+ props.getTaskTimeout(), logger);
+ this.processDao = DaoFactory.getDaoInstance(ProcessDao.class);
+ }
+ ```
+
+ Incoming custom tasks **TaskProps**And custom**Logger**，TaskProps Encapsulate task information, Logger is installed with custom log information
+
+- Inherit **AbstractParameters** Custom task parameter entity
+
+- Rewrite **AbstractTask** of **init** Parsing in method**Custom task parameter entity**
+
+- Rewrite **handle** method，transfer **ShellCommandExecutor** of **run** method，The first parameter is passed in**command**，Pass the second parameter to ProcessDao and set the corresponding **exitStatusCode**
+
+### Non-SHELL-based tasks (see SqlTask)
+
+- Need to be **cn.dolphinscheduler.server.worker.task** Down **TaskManager** A custom task
+- Need to inherit**cn.dolphinscheduler.server.worker.task** Down **AbstractTask**
+- Inherit **AbstractParameters** Custom task parameter entity
+- Constructor or override **AbstractTask** of **init** in the method, parse the custom task parameter entity
+- Rewrite **handle** Methods to implement business logic and set the corresponding**exitStatusCode**
+
diff --git a/docs/zh-cn/developer_guide/architecture-design.md b/docs/zh-cn/development/architecture-design.md
similarity index 85%
rename from docs/zh-cn/developer_guide/architecture-design.md
rename to docs/zh-cn/development/architecture-design.md
index 087e834..2044f87 100755
--- a/docs/zh-cn/developer_guide/architecture-design.md
+++ b/docs/zh-cn/development/architecture-design.md
@@ -302,3 +302,81 @@ public class TaskLogFilter extends Filter<ILoggingEvent {
本文从调度出发，初步介绍了大数据分布式工作流调度系统--DolphinScheduler的架构原理及实现思路。未完待续
+ some processes need to be processed before other processes. This can be configured at the start of the process or at the time of scheduled start. There are 5 levels, followed by HIGHEST, HIGH, MEDIUM, LOW, and LOWEST. As shown below
+
+ <p align="center">
+ <img src="https://analysys.github.io/easyscheduler_docs_cn/images/process_priority.png" alt="Process Priority Configuration" width="40%" />
+ </p>
+
+ - The priority of the task is also divided into 5 levels, followed by HIGHEST, HIGH, MEDIUM, LOW, and LOWEST. As shown below
+
+ <p align="center">
+ <img src="https://analysys.github.io/easyscheduler_docs_cn/images/task_priority.png" alt="task priority configuration" width="35%" />
+ </p>
+
+##### VI. Logback and gRPC implement log access
+
+- Since the Web (UI) and Worker are not necessarily on the same machine, viewing the log is not as it is for querying local files. There are two options:
+ - Put the logs on the ES search engine
+ - Obtain remote log information through gRPC communication
+- Considering the lightweightness of DolphinScheduler as much as possible, gRPC was chosen to implement remote access log information.
+
+ <p align="center">
+ <img src="https://analysys.github.io/easyscheduler_docs_cn/images/grpc.png" alt="grpc remote access" width="50%" />
+ </p>
+
+- We use a custom Logback FileAppender and Filter function to generate a log file for each task instance.
+- The main implementation of FileAppender is as follows:
+
+```java
+ /**
+ * task log appender
+ */
+ Public class TaskLogAppender extends FileAppender<ILoggingEvent {
+
+ ...
+
+ @Override
+ Protected void append(ILoggingEvent event) {
+
+ If (currentlyActiveFile == null){
+ currentlyActiveFile = getFile();
+ }
+ String activeFile = currentlyActiveFile;
+ // thread name: taskThreadName-processDefineId_processInstanceId_taskInstanceId
+ String threadName = event.getThreadName();
+ String[] threadNameArr = threadName.split("-");
+ // logId = processDefineId_processInstanceId_taskInstanceId
+ String logId = threadNameArr[1];
+ ...
+ super.subAppend(event);
+ }
+}
+```
+
+Generate a log in the form of /process definition id/process instance id/task instance id.log
+
+- Filter matches the thread name starting with TaskLogInfo:
+- TaskLogFilter is implemented as follows:
+
+```java
+ /**
+ * task log filter
+ */
+Public class TaskLogFilter extends Filter<ILoggingEvent {
+
+ @Override
+ Public FilterReply decide(ILoggingEvent event) {
+ If (event.getThreadName().startsWith("TaskLogInfo-")){
+ Return FilterReply.ACCEPT;
+ }
+ Return FilterReply.DENY;
+ }
+}
+```
+
+
+
+### summary
+
+Starting from the scheduling, this paper introduces the architecture principle and implementation ideas of the big data distributed workflow scheduling system-DolphinScheduler. To be continued
\ No newline at end of file
diff --git a/docs/zh-cn/developer_guide/backend-development.md b/docs/zh-cn/development/backend-development.md
similarity index 100%
rename from docs/zh-cn/developer_guide/backend-development.md
rename to docs/zh-cn/development/backend-development.md
diff --git a/docs/zh-cn/developer_guide/frontend-development.md b/docs/zh-cn/development/frontend-development.md
similarity index 100%
rename from docs/zh-cn/developer_guide/frontend-development.md
rename to docs/zh-cn/development/frontend-development.md
diff --git a/docs/zh-cn/developer_guide/plugin-development.md b/docs/zh-cn/development/plugin-development.md
similarity index 100%
rename from docs/zh-cn/developer_guide/plugin-development.md
rename to docs/zh-cn/development/plugin-development.md
diff --git a/md_json/docs.json b/md_json/docs.json
index d770ea8..0b8a72f 100755
--- a/md_json/docs.json
+++ b/md_json/docs.json
@@ -2,17 +2,12 @@
"en-us": [
{
"filename": "architecture-design.md",
- "link": "/en-us/docs/developer_guide/architecture-design.html",
+ "link": "/en-us/docs/development/architecture-design.html",
"meta": {}
},
{
"filename": "backend-development.md",
- "link": "/en-us/docs/developer_guide/backend-development.html",
- "meta": {}
- },
- {
- "filename": "frontend-development.md",
- "link": "/en-us/docs/developer_guide/frontend-development.html",
+ "link": "/en-us/docs/development/backend-development.html",
"meta": {}
},
{
@@ -26,6 +21,16 @@
"meta": {}
},
{
+ "filename": "frontend-development.md",
+ "link": "/en-us/docs/development/frontend-development.html",
+ "meta": {}
+ },
+ {
+ "filename": "plugin-development.md",
+ "link": "/en-us/docs/development/plugin-development.html",
+ "meta": {}
+ },
+ {
"filename": "faq.md",
"link": "/en-us/docs/faq.html",
"meta": {}
@@ -59,32 +64,32 @@
"zh-cn": [
{
"filename": "architecture-design.md",
- "link": "/zh-cn/docs/developer_guide/architecture-design.html",
+ "link": "/zh-cn/docs/development/architecture-design.html",
"meta": {}
},
{
"filename": "backend-development.md",
- "link": "/zh-cn/docs/developer_guide/backend-development.html",
+ "link": "/zh-cn/docs/development/backend-development.html",
"meta": {}
},
{
- "filename": "frontend-development.md",
- "link": "/zh-cn/docs/developer_guide/frontend-development.html",
+ "filename": "contribute.md",
+ "link": "/zh-cn/docs/development/contribute.html",
"meta": {}
},
{
- "filename": "plugin-development.md",
- "link": "/zh-cn/docs/developer_guide/plugin-development.html",
+ "filename": "developers.md",
+ "link": "/zh-cn/docs/development/developers.html",
"meta": {}
},
{
- "filename": "contribute.md",
- "link": "/zh-cn/docs/development/contribute.html",
+ "filename": "frontend-development.md",
+ "link": "/zh-cn/docs/development/frontend-development.html",
"meta": {}
},
{
- "filename": "developers.md",
- "link": "/zh-cn/docs/development/developers.html",
+ "filename": "plugin-development.md",
+ "link": "/zh-cn/docs/development/plugin-development.html",
"meta": {}
},
{
diff --git a/site_config/blog.js b/site_config/blog.js
index 50471cc..d85128c 100755
--- a/site_config/blog.js
+++ b/site_config/blog.js
@@ -9,13 +9,6 @@ export default {
dateStr: '2019-9-27',
desc: 'Apache Dolphin Scheduler(Incubating) will organize a meetup in Shanghai 2019.10.26. Welcome to rigister.',
link: '/en-us/blog/meetup_2019_10_26.html',
- },
- {
- title: 'DolphinScheduler Architecture Desgin',
- author: 'xiaochun',
- dateStr: '2019-9-20',
- desc: 'Blog description, some text to sum up the main content of the blog',
- link: '/en-us/blog/architecture-design.html'
}
],
},
@@ -29,13 +22,6 @@ export default {
dateStr: '2019-9-27',
desc: 'Apache Dolphin Scheduler(Incubating) 将于2019年10月26日在上海组织一次会议，欢迎踊跃报名。 ',
link: '/zh-cn/blog/meetup_2019_10_26.html',
- },
- {
- title: 'DolphinScheduler系统架构设计',
- author: 'xiaochun.liu',
- dateStr: '2019-9-20',
- desc: '介绍DolphinScheduler整体架构',
- link: '/zh-cn/blog/architecture-design.html',
}
],
},
diff --git a/site_config/development.js b/site_config/development.js
index 53ae908..ff0d3fa 100755
--- a/site_config/development.js
+++ b/site_config/development.js
@@ -2,6 +2,32 @@ export default {
'en-us': {
sidemenu: [
{
+ title: 'Developer guide',
+ children: [
+ {
+ title: 'Architecture Design',
+ link: '/en-us/docs/development/architecture-design.html',
+ },
+ {
+ title: 'Backend Development',
+ link: '/en-us/docs/development/backend-development.html',
+ },
+ {
+ title: 'plugin-development',
+ link: '/en-us/docs/development/plugin-development.html',
+ },
+ {
+ title: 'Frontend Development',
+ link: '/en-us/docs/development/frontend-development.html',
+ },
+ {
+ title: 'Api Document',
+ target: '_blank',
+ link: 'http://106.75.43.194:8888/easyscheduler/doc.html?language=en&lang=en'
+ },
+ ],
+ },
+ {
title: 'Committer Guide',
children: [
{
@@ -20,6 +46,32 @@ export default {
'zh-cn': {
sidemenu: [
{
+ title: '开发者指南',
+ children: [
+ {
+ title: '架构设计',
+ link: '/zh-cn/docs/development/architecture-design.html',
+ },
+ {
+ title: '后端开发',
+ link: '/zh-cn/docs/development/backend-development.html',
+ },
+ {
+ title: '插件开发',
+ link: '/zh-cn/docs/development/plugin-development.html',
+ },
+ {
+ title: '前端开发',
+ link: '/zh-cn/docs/development/frontend-development.html',
+ },
+ {
+ title: '接口文档',
+ target: '_blank',
+ link: 'http://106.75.43.194:8888/easyscheduler/doc.html?language=zh_CN&lang=zh'
+ },
+ ],
+ },
+ {
title: '提交者向导',
children: [
{
diff --git a/site_config/docs.js b/site_config/docs.js
index d0fb3ef..3a3c338 100755
--- a/site_config/docs.js
+++ b/site_config/docs.js
@@ -23,28 +23,6 @@ export default {
]
},
{
- title: 'Developer guide',
- children: [
- {
- title: 'Architecture Design',
- link: '/en-us/docs/developer_guide/architecture-design.html',
- },
- {
- title: 'Backend Development',
- link: '/en-us/docs/developer_guide/backend-development.html',
- },
- {
- title: 'Frontend Development',
- link: '/en-us/docs/developer_guide/frontend-development.html',
- },
- {
- title: 'Api Document',
- target: '_blank',
- link: 'http://106.75.43.194:8888/easyscheduler/doc.html?language=en&lang=en'
- },
- ],
- },
- {
title: 'Release',
children: [
{
@@ -89,32 +67,6 @@ export default {
],
},
{
- title: '开发者指南',
- children: [
- {
- title: '架构设计',
- link: '/zh-cn/docs/developer_guide/architecture-design.html',
- },
- {
- title: '后端开发',
- link: '/zh-cn/docs/developer_guide/backend-development.html',
- },
- {
- title: '插件开发',
- link: '/zh-cn/docs/developer_guide/architecture-design.html',
- },
- {
- title: '前端开发',
- link: '/zh-cn/docs/developer_guide/frontend-development.html',
- },
- {
- title: '接口文档',
- target: '_blank',
- link: 'http://106.75.43.194:8888/easyscheduler/doc.html?language=zh_CN&lang=zh'
- },
- ],
- },
- {
title: '版本发布',
children: [
{
diff --git a/site_config/site.js b/site_config/site.js
index 38c8b70..2c28e40 100755
--- a/site_config/site.js
+++ b/site_config/site.js
@@ -37,7 +37,7 @@ export default {
{
key: 'development',
text: 'DEVELOPMENT',
- link: '/en-us/docs/development/developers.html',
+ link: '/en-us/docs/development/architecture-design.html',
},
{
key: 'community',
@@ -129,7 +129,7 @@ export default {
{
key: 'development',
text: '开发者',
- link: '/zh-cn/docs/development/developers.html',
+ link: '/zh-cn/docs/development/architecture-design.html',
},
{
key: 'community',
diff --git a/src/components/header/index.jsx b/src/components/header/index.jsx
index 14672b1..cd0e993 100755
--- a/src/components/header/index.jsx
+++ b/src/components/header/index.jsx
@@ -48,6 +48,7 @@ class Header extends React.Component {
constructor(props) {
super(props);
this.state = {
+ current: '',
menuBodyVisible: false,
language: props.language,
search: siteConfig.defaultSearch,
@@ -69,15 +70,6 @@ class Header extends React.Component {
}
});
}
- state = {
- current: 'home',
- }
-
- handleClick = e => {
- this.setState({
- current: e.key,
- });
- }
componentWillReceiveProps(nextProps) {
this.setState({
@@ -85,9 +77,11 @@ class Header extends React.Component {
});
}
- toggleMenu() {
+
+
+ handleClick = e => {
this.setState({
- menuBodyVisible: !this.state.menuBodyVisible,
+ current: e.key,
});
}
@@ -206,6 +200,7 @@ class Header extends React.Component {
onClick={this.toggleMenu}
src={type === 'primary' ? getLink('/img/system/menu_white.png') : getLink('/img/system/menu_gray.png')}
/>
+ <div>
<Menu className={type === 'primary'? 'whiteClass': 'blackClass'} onClick={this.handleClick} selectedKeys={[this.state.current]} mode="horizontal">
{siteConfig[language].pageMenu.map(item => (
item.children ? <SubMenu
@@ -217,7 +212,7 @@ class Header extends React.Component {
>
<Menu.ItemGroup>
{item.children.map(items => (
- <Menu.Item key={items.key}><a href={getLink(items.link)} target={items.target || '_self'}>{items.text}</a></Menu.Item>
+ <Menu.Item key={items.key} ><a href={getLink(items.link)} target={items.target || '_self'}>{items.text}</a></Menu.Item>
))}
</Menu.ItemGroup>
</SubMenu> : <Menu.Item key={item.key}>
@@ -226,6 +221,7 @@ class Header extends React.Component {
))}
</Menu>
</div>
+ </div>
</div>
</header>
);
diff --git a/src/pages/blogDetail/index.scss b/src/pages/blogDetail/index.scss
index b2c9cd1..20dee13 100755
--- a/src/pages/blogDetail/index.scss
+++ b/src/pages/blogDetail/index.scss
@@ -6,6 +6,6 @@
.blog-content {
padding: 80px 20%;
margin: $headerHeight auto 0;
- max-width: 735px;
+ min-width: 735px;
}
}
\ No newline at end of file