You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by le...@apache.org on 2022/04/11 08:50:40 UTC
[dolphinscheduler] branch 3.0.0-alpha-prepare updated: [cherry-pick-3.0.0-alpha] Cherry pick dev to 3.0.0 alpha (#9429)
This is an automated email from the ASF dual-hosted git repository.
leonbao pushed a commit to branch 3.0.0-alpha-prepare
in repository https://gitbox.apache.org/repos/asf/dolphinscheduler.git
The following commit(s) were added to refs/heads/3.0.0-alpha-prepare by this push:
new 9fda8c5811 [cherry-pick-3.0.0-alpha] Cherry pick dev to 3.0.0 alpha (#9429)
9fda8c5811 is described below
commit 9fda8c5811be333854fe79d8100c9f2eb45d92b7
Author: caishunfeng <ca...@gmail.com>
AuthorDate: Mon Apr 11 16:50:26 2022 +0800
[cherry-pick-3.0.0-alpha] Cherry pick dev to 3.0.0 alpha (#9429)
* [Fix][UI Next][V1.0.0-Alpha]Add zh for dag execution policy (#9363)
* [Bug-9235][Alert]Fix wechat markdown message and change wechat form structure (#9367)
* fix wechat issues:
1. change table msg type to markdown.
2. change userId to not required and enrich hints
3. change 'app id' to 'app id and chat id'
* fix wechat issues:
1. revert table showtype and add markdown showtype.
2. enrich hints.
3. delete 'chatid', rename agentid to weChatAgentIdChatId.
4. modify code to send markdown message.
* fix wechat issues: Change the language pack of agentId to agentId/chatId.
* fix format
* fix param name
Co-authored-by: Amy <am...@163.com>
* [FIX-9355] Fix scheduleTime of start-process-instance api in api-doc (#9359)
* fix #9355
* fix #9355
* fix ut error
* fix ut error
* [CI] try to fix ci (#9366)
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* try to fix ci
* [optimization] [Service] Optimization ProcessService and add ProcessService interface (#9370)
* [task-spark][docs] Corrected notice section (#9375)
* [python] Migrate pythonGatewayServer into api server (#9372)
Currently the size of our distribute package is up to
800MB, this patch is migrate python gateway server into
api server
The distribute package size before and after this patch is:
```sh
# before
796M apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz
# after
647M apache-dolphinscheduler-2.0.4-SNAPSHOT-bin.tar.gz
```
* [Fix][UI Next][V1.0.0-Alpha] Add light color theme to echarts. (#9381)
* [Bug][API-9364]fix ProcessInstance wrong alert group id (#9383)
* fix ProcessInstance wrong alert group id
* change createComplementCommandList method to protected
* [BUG][WORKER-9349]fix param priority (#9379)
* fix param priority
* fix params priority code logic
* [Improvement] change method access (#9390)
* change method to protected
* change method access
* [Fix-9221] [alert-server] optimization and gracefully close (#9246)
* [Fix-9221] [alert-server] optimization and gracefully close
This closes #9221
* [Fix-9221] [alert-server] remove unused mock data
This closes #9221
* [Fix-9221] [alert-server] remove unused mock data
This closes #9221
* [Fix-9221] [alert-server] remove unnecessary Mockito stubbings
* [Fix-9221] [alert-server] init AlertPluginManager in AlertServer
* [Fix-9221] [alert-server] AlertServerTest add AlertPluginManager installPlugin
* [Fix-9221] [alert-server] replace @Eventlistener with @PostConstruct
* [Fix-9221] [alert-server] sonar check solution
* [Improvement-9221] [alert] update constructor injection and replace IStoppable with Closeable
Co-authored-by: guoshupei <gu...@lixiang.com>
* [Fix][UI Next][V1.0.0-Alpha] Fix the task instance forced success button multi-language support error. (#9392)
* [doc] Change get help from dev mail list to slack (#9377)
* Change all get help from dev mailing list to slack, because
we find out mailing list have many users ask for subscribe
and they maybe subscribe by accident.
* remove join dev mailing list in faq.md because we already
have it in https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html
* Add new code owner of docs module (#9388)
* [CI] Enable CI to remove unexpected files in /docs/img dir (#9393)
* [Bug][UI Next]Modify the display state logic of save buttons under workflow definition (#9403)
* Modifying site Configurations
* Modify the display state logic of save buttons under workflow definition
* [doc] Remove observability (#9402)
SkyWalking v9 is coming soon and there are without
DolphinScheduler menus anymore, So we should remove
the SW agent to avoid confusion.
close: #9242
* [DS-9387][refactor]Remove the lock in the start method of the MasterRegistryClient class (#9389)
* [Fix-9251] [WORKER] reslove the sql task about of add the udf resource failed (#9319)
* feat(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.extend the type of storage
3.add the s3utils
to manager resource
4.automatic inject the storage in addition to your
config
* fix(resource manager): update the dependency
* fix(resource manager): extend s3 to the storage of ds
fix the constant of hadooputils
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager):
merge the unitTest:
1.TenantServiceImpl
2.ResourceServiceImpl
3.UserServiceImpl
* fix(resource manager): extend s3 to the storage of ds
merge the resourceServiceTest
* fix(resource manager): test cancel the test method
createTenant verifyTenant
* fix(resource manager): merge the code follow the check-result of sonar
* fix(resource manager): extend s3 to the storage of ds
fit the spell question
* fix(resource manager): extend s3 to the storage of ds
revert the common.properties
* fix(resource manager): extend s3 to the storage of ds
update the storageConfig with None
* fix(resource manager): extend s3 to the storage of ds
fix the judge of resourceType
* fix(resource manager): extend s3 to the storage of ds
undo the compile-mysql
* fix(resource manager): extend s3 to the storage of ds
delete hadoop aws
* fix(resource manager): extend s3 to the storage of ds
update the know-dependencies to delete aws 1.7.4
update the e2e
file-manager common.properties
* fix(resource manager): extend s3 to the storage of ds
update the aws-region
* fix(resource manager): extend s3 to the storage of ds
fix the storageconfig init
* fix(resource manager): update e2e docker-compose
update e2e docker-compose
* fix(resource manager): extend s3 to the storage of ds
revent the e2e common.proprites
print the resource type in propertyUtil
* fix(resource manager): extend s3 to the storage of ds
1.println the properties
* fix(resource manager): println the s3 info
* fix(resource manager): extend s3 to the storage of ds
delete the info and upgrade the s3 info to e2e
* fix(resource manager): extend s3 to the storage of ds
add the bucket init
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager): extend s3 to the storage of ds
upgrade the s3 endpoint
* fix(resource manager): withPathStyleAccessEnabled(true)
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* fix(resource manager): upgrade the s3client builder
* fix(resource manager): correct the s3 point to s3client
* fix(resource manager): update the constant BUCKET_NAME
* fix(resource manager): e2e s3 endpoint -> s3:9000
* fix(resource manager): extend s3 to the storage of ds
1.fix some spell question
2.delete the import *
* style(resource manager): add info to createBucket
* style(resource manager): debug the log
* ci(resource manager): test
test s3
* ci(ci): add INSERT INTO dolphinscheduler.t_ds_tenant (id, tenant_code, description, queue_id, create_time, update_time) VALUES(1, 'root', NULL, 1, NULL, NULL); to h2.sql
* fix(resource manager): update the h2 sql
* fix(resource manager): solve to delete the tenant
* style(resource manager): merge the style end delete the unuse s3 config
* fix(resource manager): extend s3 to the storage of ds
UPDATE the rename resources when s3
* fix(resource manager): extend s3 to the storage of ds
1.fix the code style of QuartzImpl
* fix(resource manager): extend s3 to the storage of ds
1.impoort restore_type to CommonUtils
* fix(resource manager): update the work thread
* fix(resource manager): update the baseTaskProcessor
* fix(resource manager): upgrade dolphinscheduler-standalone-server.xml
* fix(resource manager): add user Info to dolphinscheduler_h2.sql
* fix(resource manager): merge the resourceType to NONE
* style(upgrade the log level to info):
* fix(resource manager): sysnc the h2.sql
* fix(resource manager): update the merge the user tenant
* fix(resource manager): merge the resourcesServiceImpl
* fix(resource manager):
when the storage is s3 ,that the directory can't be renamed
* fix(resource manager): in s3 ,the directory cannot be renamed
* fix(resource manager): delete the deleteRenameDirectory in E2E
* fix(resource manager): check the style and recoverd the test
* fix(resource manager): delete the log.print(LoginUser)
* fix(server): fix the udf serialize
* fix(master task): update the udfTest to update the json string
* fix(test): update the udfFuncTest
* fix(common): syn the common.properties
* fix(udfTest): upgrade the udfTest
* fix(common): revent the common.properties
* [Fix-9316] [Task] Configure DB2 data source SQL script execution report ResultSet has been closed exception in SQL task (#9317)
* fix db2 error in the sql task
* update limit in sql task
* [UI] Migrate NPM to PNPM in CI builds (#9431)
Co-authored-by: Devosend <de...@gmail.com>
Co-authored-by: Tq <ti...@gmail.com>
Co-authored-by: Amy <am...@163.com>
Co-authored-by: xiangzihao <46...@qq.com>
Co-authored-by: gaojun2048 <ga...@gmail.com>
Co-authored-by: mans2singh <ma...@users.noreply.github.com>
Co-authored-by: Jiajie Zhong <zh...@hotmail.com>
Co-authored-by: Amy0104 <97...@users.noreply.github.com>
Co-authored-by: guoshupei <15...@163.com>
Co-authored-by: guoshupei <gu...@lixiang.com>
Co-authored-by: songjianet <17...@qq.com>
Co-authored-by: Eric Gao <er...@gmail.com>
Co-authored-by: labbomb <73...@qq.com>
Co-authored-by: worry <70...@qq.com>
Co-authored-by: nobolity <no...@users.noreply.github.com>
Co-authored-by: Kerwin <37...@users.noreply.github.com>
Co-authored-by: kezhenxu94 <ke...@apache.org>
---
.github/CODEOWNERS | 1 +
.github/ISSUE_TEMPLATE/bug-report.yml | 7 +-
.github/workflows/backend.yml | 19 +-
.github/workflows/e2e.yml | 19 +-
.github/workflows/frontend.yml | 7 +-
.github/workflows/issue_robot.yml | 5 +-
.github/workflows/unit-test.yml | 19 +-
README.md | 2 +-
README_zh_CN.md | 4 +-
deploy/docker/docker-compose.yml | 23 -
deploy/docker/docker-stack.yml | 21 -
.../dolphinscheduler/templates/_helpers.tpl | 3 -
docs/configs/docsdev.js | 19 -
docs/docs/en/faq.md | 17 +-
docs/docs/en/guide/installation/pseudo-cluster.md | 4 +-
.../docs/en/guide/installation/skywalking-agent.md | 74 -
docs/docs/en/guide/installation/standalone.md | 4 +
docs/docs/en/guide/task/spark.md | 2 +-
docs/docs/en/guide/upgrade.md | 2 +-
docs/docs/zh/faq.md | 12 -
.../docs/zh/guide/installation/skywalking-agent.md | 74 -
docs/img/skywalking/import-dashboard-1.jpg | Bin 151863 -> 0 bytes
docs/img_utils.py | 35 +-
.../dolphinscheduler/alert/api/ShowType.java | 4 +-
.../alert/wechat/WeChatAlertChannelFactory.java | 10 +-
.../alert/wechat/WeChatAlertParamsConstants.java | 5 +-
.../plugin/alert/wechat/WeChatSender.java | 80 +-
.../plugin/alert/wechat/WechatAppChatMessage.java | 19 +-
.../plugin/alert/wechat/WechatAppMessage.java | 25 +-
.../dolphinscheduler/alert/AlertPluginManager.java | 24 +-
.../alert/AlertRequestProcessor.java | 8 +-
.../{AlertSender.java => AlertSenderService.java} | 69 +-
.../apache/dolphinscheduler/alert/AlertServer.java | 107 +-
.../dolphinscheduler/alert/AlertServerTest.java | 19 +-
.../alert/processor/AlertRequestProcessorTest.java | 21 +-
...SenderTest.java => AlertSenderServiceTest.java} | 35 +-
dolphinscheduler-api/pom.xml | 6 +
dolphinscheduler-api/src/main/docker/Dockerfile | 2 +-
.../configuration/PythonGatewayConfiguration.java | 16 +-
.../api/controller/ExecutorController.java | 80 +-
.../dolphinscheduler/api/python/PythonGateway.java | 54 +-
.../api/service/impl/ExecutorServiceImpl.java | 4 +-
.../service/impl/ProcessInstanceServiceImpl.java | 1 -
.../src/main/resources/application.yaml | 20 +
.../main/resources/i18n/messages_en_US.properties | 2 +-
.../main/resources/i18n/messages_zh_CN.properties | 2 +-
.../api/controller/ExecutorControllerTest.java | 3 +-
.../apache/dolphinscheduler/common/Constants.java | 1 +
.../dolphinscheduler/dao/entity/UdfFunc.java | 27 +-
.../dolphinscheduler/dao/entity/UdfFuncTest.java | 9 +-
.../src/main/assembly/dolphinscheduler-bin.xml | 5 -
.../master/registry/MasterRegistryClient.java | 5 -
dolphinscheduler-python/pom.xml | 75 -
.../pydolphinscheduler/UPDATING.md | 3 +
.../pydolphinscheduler/docs/source/start.rst | 28 +-
.../pydolphinscheduler/docs/source/tutorial.rst | 2 +-
.../tests/integration/__init__.py | 2 +-
.../tests/integration/test_submit_examples.py | 4 +-
.../dolphinscheduler-python-gateway-server.xml | 64 -
dolphinscheduler-python/src/main/bin/start.sh | 32 -
dolphinscheduler-python/src/main/docker/Dockerfile | 34 -
.../src/main/resources/application.yaml | 83 -
.../src/main/resources/logback-spring.xml | 57 -
.../command/alert/AlertSendRequestCommand.java | 10 +
.../service/process/ProcessService.java | 2985 ++------------------
...ProcessService.java => ProcessServiceImpl.java} | 240 +-
.../service/process/ProcessServiceTest.java | 33 +-
.../dolphinscheduler-standalone-server.xml | 4 -
.../src/main/dist-bin/start.sh | 2 +-
.../src/main/resources/application.yaml | 2 +
.../task/api/parameters/AbstractParameters.java | 10 +-
.../resource/AbstractResourceParameters.java | 3 +-
.../parameters/resource/DataSourceParameters.java | 12 +
.../api/parameters/resource/UdfFuncParameters.java | 13 +
.../plugin/task/api/parser/ParamUtils.java | 20 +-
.../dolphinscheduler/plugin/task/sql/SqlTask.java | 13 +-
dolphinscheduler-ui-next/pom.xml | 16 +-
.../src/components/chart/index.ts | 2 +-
.../src/locales/modules/en_US.ts | 2 +-
.../src/locales/modules/zh_CN.ts | 2 +-
pom.xml | 2 +-
script/dolphinscheduler-daemon.sh | 4 +-
script/env/install_env.sh | 5 -
script/scp-hosts.sh | 2 +-
script/start-all.sh | 7 -
script/status-all.sh | 8 -
script/stop-all.sh | 7 -
87 files changed, 921 insertions(+), 3903 deletions(-)
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
index 6458055d57..02ffa5b6e1 100644
--- a/.github/CODEOWNERS
+++ b/.github/CODEOWNERS
@@ -20,3 +20,4 @@ dolphinscheduler/dolphinscheduler-e2e @kezhenxu94
dolphinscheduler/dolphinscheduler-registry @kezhenxu94
dolphinscheduler/dolphinscheduler-standalone-server @kezhenxu94
dolphinscheduler/dolphinscheduler-python @zhongjiajie
+dolphinscheduler/docs @zhongjiajie @Tianqi-Dotes
diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml
index 8f12a37ebd..5edc2d071f 100644
--- a/.github/ISSUE_TEMPLATE/bug-report.yml
+++ b/.github/ISSUE_TEMPLATE/bug-report.yml
@@ -22,11 +22,12 @@ labels: [ "bug", "Waiting for reply" ]
body:
- type: markdown
attributes:
- value: |
+ value: >
Please make sure what you are reporting is indeed a bug with reproducible steps, if you want to ask questions
or share ideas, you can head to our
- [Discussions](https://github.com/apache/dolphinscheduler/discussions) tab, you can also [subscribe to our mailing list](mailto:dev-subscribe@dolphinscheduler.apache.org) and send
- emails to [our mailing list](mailto:dev@dolphinscheduler.apache.org)
+ [Discussions](https://github.com/apache/dolphinscheduler/discussions) tab, you can also
+ [join our slack](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw)
+ and send your question to channel `#troubleshooting`
For better global communication, Please write in English.
diff --git a/.github/workflows/backend.yml b/.github/workflows/backend.yml
index bb2b65c56d..c9fcc543b3 100644
--- a/.github/workflows/backend.yml
+++ b/.github/workflows/backend.yml
@@ -42,18 +42,18 @@ jobs:
name: Backend-Path-Filter
runs-on: ubuntu-latest
outputs:
- ignore: ${{ steps.filter.outputs.ignore }}
+ not-ignore: ${{ steps.filter.outputs.not-ignore }}
steps:
- uses: dorny/paths-filter@b2feaf19c27470162a626bd6fa8438ae5b263721
id: filter
with:
filters: |
- ignore:
- - '(docs/**|dolphinscheduler-ui/**|dolphinscheduler-ui-next/**)'
+ not-ignore:
+ - '!(docs/**|dolphinscheduler-ui/**|dolphinscheduler-ui-next/**)'
build:
name: Backend-Build
needs: paths-filter
- if: ${{ needs.paths-filter.outputs.ignore == 'false' }}
+ if: ${{ needs.paths-filter.outputs.not-ignore == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
@@ -81,13 +81,16 @@ jobs:
name: Build
runs-on: ubuntu-latest
timeout-minutes: 30
- needs: [ build ]
+ needs: [ build, paths-filter ]
if: always()
steps:
- name: Status
run: |
- if [[ ${{ needs.build.result }} == 'success' || ${{ needs.paths-filter.outputs.ignore == 'true' }} ]]; then
- echo "Passed!"
- else
+ if [[ ${{ needs.paths-filter.outputs.not-ignore }} == 'false' ]]; then
+ echo "Skip Build!"
+ exit 0
+ fi
+ if [[ ${{ needs.build.result }} != 'success' ]]; then
+ echo "Build Failed!"
exit -1
fi
diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml
index f35c8d13df..f34be88b17 100644
--- a/.github/workflows/e2e.yml
+++ b/.github/workflows/e2e.yml
@@ -33,18 +33,18 @@ jobs:
name: E2E-Path-Filter
runs-on: ubuntu-latest
outputs:
- ignore: ${{ steps.filter.outputs.ignore }}
+ not-ignore: ${{ steps.filter.outputs.not-ignore }}
steps:
- uses: dorny/paths-filter@b2feaf19c27470162a626bd6fa8438ae5b263721
id: filter
with:
filters: |
- ignore:
- - '(docs/**)'
+ not-ignore:
+ - '!(docs/**)'
build:
name: E2E-Build
needs: paths-filter
- if: ${{ needs.paths-filter.outputs.ignore == 'false' }}
+ if: ${{ needs.paths-filter.outputs.not-ignore == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
@@ -155,13 +155,16 @@ jobs:
name: E2E
runs-on: ubuntu-latest
timeout-minutes: 30
- needs: [ e2e ]
+ needs: [ e2e, paths-filter ]
if: always()
steps:
- name: Status
run: |
- if [[ ${{ needs.e2e.result }} == 'success' || ${{ needs.paths-filter.outputs.ignore == 'true' }} ]]; then
- echo "Passed!"
- else
+ if [[ ${{ needs.paths-filter.outputs.not-ignore }} == 'false' ]]; then
+ echo "Skip E2E!"
+ exit 0
+ fi
+ if [[ ${{ needs.e2e.result }} != 'success' ]]; then
+ echo "E2E Failed!"
exit -1
fi
diff --git a/.github/workflows/frontend.yml b/.github/workflows/frontend.yml
index b1125ce481..28487e19e3 100644
--- a/.github/workflows/frontend.yml
+++ b/.github/workflows/frontend.yml
@@ -58,6 +58,7 @@ jobs:
node-version: 16
- name: Compile and Build
run: |
- npm install
- npm run lint
- npm run build:prod
+ npm install pnpm -g
+ pnpm install
+ pnpm run lint
+ pnpm run build:prod
diff --git a/.github/workflows/issue_robot.yml b/.github/workflows/issue_robot.yml
index 3907773b85..63741a71a7 100644
--- a/.github/workflows/issue_robot.yml
+++ b/.github/workflows/issue_robot.yml
@@ -40,5 +40,8 @@ jobs:
- name: "Comment in issue"
uses: ./.github/actions/comment-on-issue
with:
- message: "Hi:\n* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.\n* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.\n* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email c [...]
+ message: |
+ Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
+ * In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
+ * If you haven't received a reply for a long time, you can [join our slack](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw) and send your question to channel `#troubleshooting`
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
diff --git a/.github/workflows/unit-test.yml b/.github/workflows/unit-test.yml
index b666741107..d7016ad991 100644
--- a/.github/workflows/unit-test.yml
+++ b/.github/workflows/unit-test.yml
@@ -40,18 +40,18 @@ jobs:
name: Unit-Test-Path-Filter
runs-on: ubuntu-latest
outputs:
- ignore: ${{ steps.filter.outputs.ignore }}
+ not-ignore: ${{ steps.filter.outputs.not-ignore }}
steps:
- uses: dorny/paths-filter@b2feaf19c27470162a626bd6fa8438ae5b263721
id: filter
with:
filters: |
- ignore:
- - '(docs/**)'
+ not-ignore:
+ - '!(docs/**)'
unit-test:
name: Unit-Test
needs: paths-filter
- if: ${{ needs.paths-filter.outputs.ignore == 'false' }}
+ if: ${{ needs.paths-filter.outputs.not-ignore == 'true' }}
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
@@ -114,13 +114,16 @@ jobs:
name: Unit Test
runs-on: ubuntu-latest
timeout-minutes: 30
- needs: [ unit-test ]
+ needs: [ unit-test, paths-filter ]
if: always()
steps:
- name: Status
run: |
- if [[ ${{ needs.unit-test.result }} == 'success' || ${{ needs.paths-filter.outputs.ignore == 'true' }} ]]; then
- echo "Passed!"
- else
+ if [[ ${{ needs.paths-filter.outputs.not-ignore }} == 'false' ]]; then
+ echo "Skip Unit Test!"
+ exit 0
+ fi
+ if [[ ${{ needs.unit-test.result }} != 'success' ]]; then
+ echo "Unit Test Failed!"
exit -1
fi
diff --git a/README.md b/README.md
index cc4c8f74b0..67b271872c 100644
--- a/README.md
+++ b/README.md
@@ -86,7 +86,7 @@ We would like to express our deep gratitude to all the open-source projects used
## Get Help
1. Submit an [issue](https://github.com/apache/dolphinscheduler/issues/new/choose)
-1. Subscribe to this mailing list: https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html, then email dev@dolphinscheduler.apache.org
+2. [Join our slack](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw) and send your question to channel `#troubleshooting`
## Community
diff --git a/README_zh_CN.md b/README_zh_CN.md
index 7f76129367..8decfe0348 100644
--- a/README_zh_CN.md
+++ b/README_zh_CN.md
@@ -87,8 +87,8 @@ Dolphin Scheduler使用了很多优秀的开源项目，比如google的guava、g
## 获得帮助
-1. 提交issue
-2. 先订阅邮件开发列表:[订阅邮件列表](https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html), 订阅成功后发送邮件到dev@dolphinscheduler.apache.org.
+1. 提交 [issue](https://github.com/apache/dolphinscheduler/issues/new/choose)
+2. [加入slack群](https://join.slack.com/t/asf-dolphinscheduler/shared_invite/zt-omtdhuio-_JISsxYhiVsltmC5h38yfw) 并在频道 `#troubleshooting` 中提问
## 社区
diff --git a/deploy/docker/docker-compose.yml b/deploy/docker/docker-compose.yml
index 83ca8a644a..2c557c4dd7 100644
--- a/deploy/docker/docker-compose.yml
+++ b/deploy/docker/docker-compose.yml
@@ -140,29 +140,6 @@ services:
networks:
- dolphinscheduler
- dolphinscheduler-python-gateway:
- image: ${HUB}/dolphinscheduler-python:${TAG}
- ports:
- - "54321:54321"
- - "25333:25333"
- env_file: .env
- healthcheck:
- test: [ "CMD", "curl", "http://localhost:54321/actuator/health" ]
- interval: 30s
- timeout: 5s
- retries: 3
- depends_on:
- dolphinscheduler-schema-initializer:
- condition: service_completed_successfully
- dolphinscheduler-zookeeper:
- condition: service_healthy
- volumes:
- - dolphinscheduler-logs:/opt/dolphinscheduler/logs
- - dolphinscheduler-shared-local:/opt/soft
- - dolphinscheduler-resource-local:/dolphinscheduler
- networks:
- - dolphinscheduler
-
networks:
dolphinscheduler:
driver: bridge
diff --git a/deploy/docker/docker-stack.yml b/deploy/docker/docker-stack.yml
index 8d0ae92553..02ee37be02 100644
--- a/deploy/docker/docker-stack.yml
+++ b/deploy/docker/docker-stack.yml
@@ -118,27 +118,6 @@ services:
mode: replicated
replicas: 1
- dolphinscheduler-python-gateway:
- image: apache/dolphinscheduler-python-gateway
- ports:
- - 54321:54321
- - 25333:25333
- env_file: .env
- healthcheck:
- test: [ "CMD", "curl", "http://localhost:54321/actuator/health" ]
- interval: 30s
- timeout: 5s
- retries: 3
- volumes:
- - dolphinscheduler-logs:/opt/dolphinscheduler/logs
- - dolphinscheduler-shared-local:/opt/soft
- - dolphinscheduler-resource-local:/dolphinscheduler
- networks:
- - dolphinscheduler
- deploy:
- mode: replicated
- replicas: 1
-
networks:
dolphinscheduler:
driver: overlay
diff --git a/deploy/kubernetes/dolphinscheduler/templates/_helpers.tpl b/deploy/kubernetes/dolphinscheduler/templates/_helpers.tpl
index f8fef1167a..c2684e029a 100644
--- a/deploy/kubernetes/dolphinscheduler/templates/_helpers.tpl
+++ b/deploy/kubernetes/dolphinscheduler/templates/_helpers.tpl
@@ -44,9 +44,6 @@ Create default docker images' fullname.
{{- define "dolphinscheduler.image.fullname.tools" -}}
{{- .Values.image.registry }}/dolphinscheduler-tools:{{ .Values.image.tag | default .Chart.AppVersion -}}
{{- end -}}
-{{- define "dolphinscheduler.image.fullname.python-gateway" -}}
-{{- .Values.image.registry }}/dolphinscheduler-python-gateway:{{ .Values.image.tag | default .Chart.AppVersion -}}
-{{- end -}}
{{/*
Create a default common labels.
diff --git a/docs/configs/docsdev.js b/docs/configs/docsdev.js
index 66673b2735..ad39afefb6 100644
--- a/docs/configs/docsdev.js
+++ b/docs/configs/docsdev.js
@@ -298,16 +298,6 @@ export default {
},
],
},
- {
- title: 'Observability',
- children: [
-
- {
- title: 'SkyWalking-Agent',
- link: '/en-us/docs/dev/user_doc/guide/installation/skywalking-agent.html',
- },
- ],
- },
{
title: 'FAQ',
children: [
@@ -609,15 +599,6 @@ export default {
},
],
},
- {
- title: '可观测性',
- children: [
- {
- title: 'SkyWalking-Agent',
- link: '/zh-cn/docs/dev/user_doc/guide/installation/skywalking-agent.html',
- },
- ],
- },
{
title: 'FAQ',
children: [
diff --git a/docs/docs/en/faq.md b/docs/docs/en/faq.md
index c920dfa327..05d8be7dc1 100644
--- a/docs/docs/en/faq.md
+++ b/docs/docs/en/faq.md
@@ -544,17 +544,6 @@ A: 1, edit nginx config file /etc/nginx/conf.d/escheduler.conf
---
-## Q : Welcome to subscribe the DolphinScheduler development mailing list
-A: In the process of using DolphinScheduler, if you have any questions or ideas, suggestions, you can participate in the DolphinScheduler community building through the Apache mailing list. Sending a subscription email is also very simple, the steps are as follows:
-
-1, Send an email to dev-subscribe@dolphinscheduler.apache.org with your own email address, subject and content.
-
-2, Receive confirmation email and reply. After completing step 1, you will receive a confirmation email from dev-help@dolphinscheduler.apache.org (if not received, please confirm whether the email is automatically classified as spam, promotion email, subscription email, etc.) . Then reply directly to the email, or click on the link in the email to reply quickly, the subject and content are arbitrary.
-
-3, Receive a welcome email. After completing the above steps, you will receive a welcome email with the subject WELCOME to dev@dolphinscheduler.apache.org, and you have successfully subscribed to the Apache DolphinScheduler mailing list.
-
----
-
## Q : Workflow Dependency
A: 1, It is currently judged according to natural days, at the end of last month: the judgment time is the workflow A start_time/scheduler_time between '2019-05-31 00:00:00' and '2019-05-31 23:59:59'. Last month: It is judged that there is an A instance completed every day from the 1st to the end of the month. Last week: There are completed A instances 7 days last week. The first two days: Judging yesterday and the day before yesterday, there must be a completed A instance for two days.
@@ -712,6 +701,12 @@ A：The repair can be completed by executing the following SQL in the database:
update t_ds_version set version='2.0.1';
```
+## Can not find python-gateway-server in distribute package
+
+After version 3.0.0-alpha, Python gateway server integrate into API server, and Python gateway service will start when you
+start API server. If you want disabled when Python gateway service you could change API server configuration in path
+`api-server/conf/application.yaml` and change attribute `python-gateway.enabled : false`.
+
---
## We will collect more FAQ later
\ No newline at end of file
diff --git a/docs/docs/en/guide/installation/pseudo-cluster.md b/docs/docs/en/guide/installation/pseudo-cluster.md
index b887030e4a..333aecc762 100644
--- a/docs/docs/en/guide/installation/pseudo-cluster.md
+++ b/docs/docs/en/guide/installation/pseudo-cluster.md
@@ -193,7 +193,9 @@ sh ./bin/dolphinscheduler-daemon.sh start alert-server
sh ./bin/dolphinscheduler-daemon.sh stop alert-server
```
-> **_Note:_**: Please refer to the section of "System Architecture Design" for service usage
+> **_Note:_**: Please refer to the section of "System Architecture Design" for service usage. Python gateway service is
+> started along with the api-server, and if you do not want to start Python gateway service please disabled it by changing
+> the yaml config `python-gateway.enabled : false` in api-server's configuration path `api-server/conf/application.yaml`
[jdk]: https://www.oracle.com/technetwork/java/javase/downloads/index.html
[zookeeper]: https://zookeeper.apache.org/releases.html
diff --git a/docs/docs/en/guide/installation/skywalking-agent.md b/docs/docs/en/guide/installation/skywalking-agent.md
deleted file mode 100644
index fedec673dc..0000000000
--- a/docs/docs/en/guide/installation/skywalking-agent.md
+++ /dev/null
@@ -1,74 +0,0 @@
-SkyWalking Agent Deployment
-=============================
-
-The `dolphinscheduler-skywalking` module provides [SkyWalking](https://skywalking.apache.org/) monitor agent for the DolphinScheduler project.
-
-This document describes how to enable SkyWalking version 8.4+ support with this module (recommend using SkyWalking 8.5.0).
-
-## Installation
-
-The following configuration is used to enable the SkyWalking agent.
-
-### Through Environment Variable Configuration (for Docker Compose)
-
-Modify SkyWalking environment variables in `docker/docker-swarm/config.env.sh`:
-
-```
-SKYWALKING_ENABLE=true
-SW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800
-SW_GRPC_LOG_SERVER_HOST=127.0.0.1
-SW_GRPC_LOG_SERVER_PORT=11800
-```
-
-And run:
-
-```shell
-$ docker-compose up -d
-```
-
-### Through Environment Variable Configuration (for Docker)
-
-```shell
-$ docker run -d --name dolphinscheduler \
--e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \
--e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
--e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
--e SKYWALKING_ENABLE="true" \
--e SW_AGENT_COLLECTOR_BACKEND_SERVICES="your.skywalking-oap-server.com:11800" \
--e SW_GRPC_LOG_SERVER_HOST="your.skywalking-log-reporter.com" \
--e SW_GRPC_LOG_SERVER_PORT="11800" \
--p 12345:12345 \
-apache/dolphinscheduler:1.3.8 all
-```
-
-### Through install_config.conf Configuration (for DolphinScheduler install.sh)
-
-Add the following configurations to `${workDir}/conf/config/install_config.conf`.
-
-```properties
-
-# SkyWalking config
-# note: enable SkyWalking tracking plugin
-enableSkywalking="true"
-# note: configure SkyWalking backend service address
-skywalkingServers="your.skywalking-oap-server.com:11800"
-# note: configure SkyWalking log reporter host
-skywalkingLogReporterHost="your.skywalking-log-reporter.com"
-# note: configure SkyWalking log reporter port
-skywalkingLogReporterPort="11800"
-
-```
-
-## Usage
-
-### Import Dashboard
-
-#### Import DolphinScheduler Dashboard to SkyWalking Server
-
-Copy the `${dolphinscheduler.home}/ext/skywalking-agent/dashboard/dolphinscheduler.yml` file into `${skywalking-oap-server.home}/config/ui-initialized-templates/` directory, and restart SkyWalking oap-server.
-
-#### View DolphinScheduler Dashboard
-
-If you have opened the SkyWalking dashboard with a browser before, you need to clear the browser cache.
-
-
diff --git a/docs/docs/en/guide/installation/standalone.md b/docs/docs/en/guide/installation/standalone.md
index a03944c17c..c9e37db7f1 100644
--- a/docs/docs/en/guide/installation/standalone.md
+++ b/docs/docs/en/guide/installation/standalone.md
@@ -39,4 +39,8 @@ sh ./bin/dolphinscheduler-daemon.sh start standalone-server
sh ./bin/dolphinscheduler-daemon.sh stop standalone-server
```
+> Note: Python gateway service is started along with the api-server, and if you do not want to start Python gateway
+> service please disabled it by changing the yaml config `python-gateway.enabled : false` in api-server's configuration
+> path `api-server/conf/application.yaml`
+
[jdk]: https://www.oracle.com/technetwork/java/javase/downloads/index.html
diff --git a/docs/docs/en/guide/task/spark.md b/docs/docs/en/guide/task/spark.md
index 7e025449c6..1e9f31524f 100644
--- a/docs/docs/en/guide/task/spark.md
+++ b/docs/docs/en/guide/task/spark.md
@@ -65,4 +65,4 @@ Configure the required content according to the parameter descriptions above.
## Notice
-JAVA and Scala only used for identification, there is no difference. If use Python to develop Flink, there is no class of the main function and the rest is the same.
+JAVA and Scala only used for identification, there is no difference. If you use Python to develop Spark application, there is no class of the main function and the rest is the same.
diff --git a/docs/docs/en/guide/upgrade.md b/docs/docs/en/guide/upgrade.md
index b401ce1354..38f80d8c73 100644
--- a/docs/docs/en/guide/upgrade.md
+++ b/docs/docs/en/guide/upgrade.md
@@ -6,7 +6,7 @@
`sh ./script/stop-all.sh`
-## Download the Newest Version Installation Package
+## Download the Latest Version Installation Package
- [download](/en-us/download/download.html) the latest version of the installation packages.
- The following upgrade operations need to be performed in the new version's directory.
diff --git a/docs/docs/zh/faq.md b/docs/docs/zh/faq.md
index f76853c8a3..8d152aae40 100644
--- a/docs/docs/zh/faq.md
+++ b/docs/docs/zh/faq.md
@@ -523,18 +523,6 @@ A：1，edit /etc/nginx/conf.d/escheduler.conf
---
-## Q：欢迎订阅 DolphinScheduler 开发邮件列表
-A：在使用 DolphinScheduler 的过程中，如果您有任何问题或者想法、建议，都可以通过 Apache 邮件列表参与到 DolphinScheduler 的社区建设中来。
- 发送订阅邮件也非常简单，步骤如下:
-
- 1，用自己的邮箱向 dev-subscribe@dolphinscheduler.apache.org 发送一封邮件，主题和内容任意。
-
- 2， 接收确认邮件并回复。 完成步骤1后，您将收到一封来自 dev-help@dolphinscheduler.apache.org 的确认邮件（如未收到，请确认邮件是否被自动归入垃圾邮件、推广邮件、订阅邮件等文件夹）。然后直接回复该邮件，或点击邮件里的链接快捷回复即可，主题和内容任意。
-
- 3， 接收欢迎邮件。 完成以上步骤后，您会收到一封主题为 WELCOME to dev@dolphinscheduler.apache.org 的欢迎邮件，至此您已成功订阅 Apache DolphinScheduler的邮件列表。
-
----
-
## Q：工作流依赖
A：1，目前是按照自然天来判断，上月末：判断时间是工作流 A start_time/scheduler_time between '2019-05-31 00:00:00' and '2019-05-31 23:59:59'。上月：是判断上个月从 1 号到月末每天都要有完成的A实例。上周： 上周 7 天都要有完成的 A 实例。前两天： 判断昨天和前天，两天都要有完成的 A 实例。
diff --git a/docs/docs/zh/guide/installation/skywalking-agent.md b/docs/docs/zh/guide/installation/skywalking-agent.md
deleted file mode 100644
index 6f36668473..0000000000
--- a/docs/docs/zh/guide/installation/skywalking-agent.md
+++ /dev/null
@@ -1,74 +0,0 @@
-SkyWalking Agent 部署
-=============================
-
-dolphinscheduler-skywalking 模块为 Dolphinscheduler 项目提供了 [Skywalking](https://skywalking.apache.org/) 监控代理。
-
-本文档介绍了如何通过此模块接入 SkyWalking 8.4+ (推荐使用8.5.0)。
-
-# 安装
-
-以下配置用于启用 Skywalking agent。
-
-### 通过配置环境变量 (使用 Docker Compose 部署时)
-
-修改 `docker/docker-swarm/config.env.sh` 文件中的 SKYWALKING 环境变量:
-
-```
-SKYWALKING_ENABLE=true
-SW_AGENT_COLLECTOR_BACKEND_SERVICES=127.0.0.1:11800
-SW_GRPC_LOG_SERVER_HOST=127.0.0.1
-SW_GRPC_LOG_SERVER_PORT=11800
-```
-
-并且运行
-
-```shell
-$ docker-compose up -d
-```
-
-### 通过配置环境变量 (使用 Docker 部署时)
-
-```shell
-$ docker run -d --name dolphinscheduler \
--e DATABASE_HOST="192.168.x.x" -e DATABASE_PORT="5432" -e DATABASE_DATABASE="dolphinscheduler" \
--e DATABASE_USERNAME="test" -e DATABASE_PASSWORD="test" \
--e ZOOKEEPER_QUORUM="192.168.x.x:2181" \
--e SKYWALKING_ENABLE="true" \
--e SW_AGENT_COLLECTOR_BACKEND_SERVICES="your.skywalking-oap-server.com:11800" \
--e SW_GRPC_LOG_SERVER_HOST="your.skywalking-log-reporter.com" \
--e SW_GRPC_LOG_SERVER_PORT="11800" \
--p 12345:12345 \
-apache/dolphinscheduler:1.3.8 all
-```
-
-### 通过配置 install_config.conf (使用 DolphinScheduler install.sh 部署时)
-
-添加以下配置到 `${workDir}/conf/config/install_config.conf`.
-
-```properties
-
-# skywalking config
-# note: enable skywalking tracking plugin
-enableSkywalking="true"
-# note: configure skywalking backend service address
-skywalkingServers="your.skywalking-oap-server.com:11800"
-# note: configure skywalking log reporter host
-skywalkingLogReporterHost="your.skywalking-log-reporter.com"
-# note: configure skywalking log reporter port
-skywalkingLogReporterPort="11800"
-
-```
-
-# 使用
-
-### 导入图表
-
-#### 导入图表到 Skywalking server
-
-复制 `${dolphinscheduler.home}/ext/skywalking-agent/dashboard/dolphinscheduler.yml` 文件到 `${skywalking-oap-server.home}/config/ui-initialized-templates/` 目录下，并重启 Skywalking oap-server。
-
-#### 查看 dolphinscheduler 图表
-
-如果之前已经使用浏览器打开过 Skywalking，则需要清空浏览器缓存。
-
-
diff --git a/docs/img/skywalking/import-dashboard-1.jpg b/docs/img/skywalking/import-dashboard-1.jpg
deleted file mode 100644
index db5da40c0f..0000000000
Binary files a/docs/img/skywalking/import-dashboard-1.jpg and /dev/null differ
diff --git a/docs/img_utils.py b/docs/img_utils.py
index 8a19557520..98ae07dcca 100644
--- a/docs/img_utils.py
+++ b/docs/img_utils.py
@@ -32,16 +32,6 @@ root_dir: Path = Path(__file__).parent
img_dir: Path = root_dir.joinpath("img")
doc_dir: Path = root_dir.joinpath("docs")
-expect_img_types: Set = {
- "jpg",
- "png",
-}
-
-
-def build_pattern() -> re.Pattern:
- """Build current document image regexp pattern."""
- return re.compile(f"(/img.*?\\.({'|'.join(expect_img_types)}))")
-
def get_files_recurse(path: Path) -> Set:
"""Get all files recursively from given :param:`path`."""
@@ -68,14 +58,15 @@ def get_paths_rel_path(paths: Set[Path], rel: Path) -> Set:
return {f"/{path.relative_to(rel)}" for path in paths}
-def get_docs_img_path(paths: Set[Path], pattern: re.Pattern) -> Set:
+def get_docs_img_path(paths: Set[Path]) -> Set:
"""Get all img syntax from given :param:`paths` using the regexp from :param:`pattern`."""
res = set()
+ pattern = re.compile(r"/img[\w./-]*")
for path in paths:
content = path.read_text()
find = pattern.findall(content)
if find:
- res |= {item[0] for item in find}
+ res |= {item for item in find}
return res
@@ -102,16 +93,6 @@ def diff_two_set(first: Set, second: Set) -> Tuple[set, set]:
return first.difference(second), second.difference(first)
-def check_diff_img_type() -> Tuple[set, set]:
- """Check images difference type.
-
- :return: Tuple[(actual - expect), (expect - actual)]
- """
- img = get_files_recurse(img_dir)
- img_suffix = get_paths_uniq_suffix(img)
- return diff_two_set(img_suffix, expect_img_types)
-
-
def check_diff_img() -> Tuple[set, set]:
"""Check images difference files.
@@ -120,20 +101,12 @@ def check_diff_img() -> Tuple[set, set]:
img = get_files_recurse(img_dir)
docs = get_files_recurse(doc_dir)
img_rel_path = get_paths_rel_path(img, root_dir)
- pat = build_pattern()
- docs_rel_path = get_docs_img_path(docs, pat)
+ docs_rel_path = get_docs_img_path(docs)
return diff_two_set(docs_rel_path, img_rel_path)
def check() -> None:
"""Runner for `check` sub command."""
- img_type_act, img_type_exp = check_diff_img_type()
- assert not img_type_act and not img_type_exp, (
- f"Images type assert failed: \n"
- f"* difference actual types to expect is: {img_type_act if img_type_act else 'None'}\n"
- f"* difference expect types to actual is: {img_type_exp if img_type_exp else 'None'}\n"
- )
-
img_docs, img_img = check_diff_img()
assert not img_docs and not img_img, (
f"Images assert failed: \n"
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-api/src/main/java/org/apache/dolphinscheduler/alert/api/ShowType.java b/dolphinscheduler-alert/dolphinscheduler-alert-api/src/main/java/org/apache/dolphinscheduler/alert/api/ShowType.java
index bb1b1b2afb..2f1b4f2b4b 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-api/src/main/java/org/apache/dolphinscheduler/alert/api/ShowType.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-api/src/main/java/org/apache/dolphinscheduler/alert/api/ShowType.java
@@ -25,11 +25,13 @@ public enum ShowType {
* 1 TEXT;
* 2 attachment;
* 3 TABLE+attachment;
+ * 4 MARKDOWN;
*/
TABLE(0, "table"),
TEXT(1, "text"),
ATTACHMENT(2, "attachment"),
- TABLE_ATTACHMENT(3, "table attachment");
+ TABLE_ATTACHMENT(3, "table attachment"),
+ MARKDOWN(4, "markdown"),;
private final int code;
private final String descp;
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WeChatAlertChannelFactory.java b/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WeChatAlertChannelFactory.java
index a77daf7e45..39a4a8159f 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WeChatAlertChannelFactory.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WeChatAlertChannelFactory.java
@@ -56,14 +56,14 @@ public final class WeChatAlertChannelFactory implements AlertChannelFactory {
.build();
InputParam usersParam = InputParam.newBuilder(WeChatAlertParamsConstants.NAME_ENTERPRISE_WE_CHAT_USERS, WeChatAlertParamsConstants.ENTERPRISE_WE_CHAT_USERS)
- .setPlaceholder("please input users ")
+ .setPlaceholder("use `|` to separate userIds and `@all` to everyone ")
.addValidate(Validate.newBuilder()
- .setRequired(true)
+ .setRequired(false)
.build())
.build();
InputParam agentIdParam = InputParam.newBuilder(WeChatAlertParamsConstants.NAME_ENTERPRISE_WE_CHAT_AGENT_ID, WeChatAlertParamsConstants.ENTERPRISE_WE_CHAT_AGENT_ID)
- .setPlaceholder("please input agent id ")
+ .setPlaceholder("please input agent id or chat id ")
.addValidate(Validate.newBuilder()
.setRequired(true)
.build())
@@ -77,9 +77,9 @@ public final class WeChatAlertChannelFactory implements AlertChannelFactory {
.build();
RadioParam showType = RadioParam.newBuilder(AlertConstants.NAME_SHOW_TYPE, AlertConstants.SHOW_TYPE)
- .addParamsOptions(new ParamsOptions(ShowType.TABLE.getDescp(), ShowType.TABLE.getDescp(), false))
+ .addParamsOptions(new ParamsOptions(ShowType.MARKDOWN.getDescp(), ShowType.MARKDOWN.getDescp(), false))
.addParamsOptions(new ParamsOptions(ShowType.TEXT.getDescp(), ShowType.TEXT.getDescp(), false))
- .setValue(ShowType.TABLE.getDescp())
+ .setValue(ShowType.MARKDOWN.getDescp())
.addValidate(Validate.newBuilder().setRequired(true).build())
.build();
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WeChatAlertParamsConstants.java b/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WeChatAlertParamsConstants.java
index db9813dab8..d8085b901f 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WeChatAlertParamsConstants.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WeChatAlertParamsConstants.java
@@ -24,9 +24,8 @@ public final class WeChatAlertParamsConstants {
static final String NAME_ENTERPRISE_WE_CHAT_SECRET = "secret";
static final String ENTERPRISE_WE_CHAT_TEAM_SEND_MSG = "$t('teamSendMsg')";
static final String NAME_ENTERPRISE_WE_CHAT_TEAM_SEND_MSG = "teamSendMsg";
- static final String ENTERPRISE_WE_CHAT_AGENT_ID = "$t('agentId')";
- static final String NAME_ENTERPRISE_WE_CHAT_AGENT_ID = "agentId";
- static final String NAME_ENTERPRISE_WE_CHAT_CHAT_ID = "chatId";
+ static final String ENTERPRISE_WE_CHAT_AGENT_ID = "$t('agentId/chatId')";
+ static final String NAME_ENTERPRISE_WE_CHAT_AGENT_ID = "agentId/chatId";
static final String ENTERPRISE_WE_CHAT_USERS = "$t('users')";
static final String NAME_ENTERPRISE_WE_CHAT_USERS = "users";
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WeChatSender.java b/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WeChatSender.java
index 8dfdf50986..b8ed73d587 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WeChatSender.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WeChatSender.java
@@ -17,15 +17,10 @@
package org.apache.dolphinscheduler.plugin.alert.wechat;
-import static java.util.Objects.requireNonNull;
-import static org.apache.dolphinscheduler.plugin.alert.wechat.WeChatAlertConstants.*;
-
import org.apache.dolphinscheduler.alert.api.AlertConstants;
import org.apache.dolphinscheduler.alert.api.AlertResult;
-import org.apache.dolphinscheduler.alert.api.ShowType;
import org.apache.dolphinscheduler.spi.utils.JSONUtils;
import org.apache.dolphinscheduler.spi.utils.StringUtils;
-
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
@@ -34,18 +29,20 @@ import org.apache.http.entity.StringEntity;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.io.IOException;
import java.util.HashMap;
-import java.util.Iterator;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
import java.util.Set;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
+
+import static java.util.Objects.requireNonNull;
+import static org.apache.dolphinscheduler.plugin.alert.wechat.WeChatAlertConstants.*;
public final class WeChatSender {
private static final Logger logger = LoggerFactory.getLogger(WeChatSender.class);
@@ -57,8 +54,7 @@ public final class WeChatSender {
private static final String CORP_ID_REGEX = "{corpId}";
private static final String SECRET_REGEX = "{secret}";
private static final String TOKEN_REGEX = "{token}";
- private final String weChatAgentId;
- private final String weChatChatId;
+ private final String weChatAgentIdChatId;
private final String weChatUsers;
private final String weChatTokenUrlReplace;
private final String weChatToken;
@@ -66,8 +62,7 @@ public final class WeChatSender {
private final String showType;
WeChatSender(Map<String, String> config) {
- weChatAgentId = config.get(WeChatAlertParamsConstants.NAME_ENTERPRISE_WE_CHAT_AGENT_ID);
- weChatChatId = config.get(WeChatAlertParamsConstants.NAME_ENTERPRISE_WE_CHAT_CHAT_ID);
+ weChatAgentIdChatId = config.get(WeChatAlertParamsConstants.NAME_ENTERPRISE_WE_CHAT_AGENT_ID);
weChatUsers = config.get(WeChatAlertParamsConstants.NAME_ENTERPRISE_WE_CHAT_USERS);
String weChatCorpId = config.get(WeChatAlertParamsConstants.NAME_ENTERPRISE_WE_CHAT_CORP_ID);
String weChatSecret = config.get(WeChatAlertParamsConstants.NAME_ENTERPRISE_WE_CHAT_SECRET);
@@ -76,8 +71,8 @@ public final class WeChatSender {
showType = config.get(AlertConstants.NAME_SHOW_TYPE);
requireNonNull(showType, AlertConstants.NAME_SHOW_TYPE + MUST_NOT_NULL);
weChatTokenUrlReplace = weChatTokenUrl
- .replace(CORP_ID_REGEX, weChatCorpId)
- .replace(SECRET_REGEX, weChatSecret);
+ .replace(CORP_ID_REGEX, weChatCorpId)
+ .replace(SECRET_REGEX, weChatSecret);
weChatToken = getToken();
}
@@ -100,42 +95,10 @@ public final class WeChatSender {
}
}
- /**
- * convert table to markdown style
- *
- * @param title the title
- * @param content the content
- * @return markdown table content
- */
- private static String markdownTable(String title, String content) {
- List<LinkedHashMap> mapItemsList = JSONUtils.toList(content, LinkedHashMap.class);
- if (null == mapItemsList || mapItemsList.isEmpty()) {
- logger.error("itemsList is null");
- throw new RuntimeException("itemsList is null");
- }
- StringBuilder contents = new StringBuilder(200);
- for (LinkedHashMap mapItems : mapItemsList) {
- Set<Entry<String, Object>> entries = mapItems.entrySet();
- Iterator<Entry<String, Object>> iterator = entries.iterator();
- StringBuilder t = new StringBuilder(String.format("`%s`%s", title, WeChatAlertConstants.MARKDOWN_ENTER));
-
- while (iterator.hasNext()) {
-
- Map.Entry<String, Object> entry = iterator.next();
- t.append(WeChatAlertConstants.MARKDOWN_QUOTE);
- t.append(entry.getKey()).append(":").append(entry.getValue());
- t.append(WeChatAlertConstants.MARKDOWN_ENTER);
- }
- contents.append(t);
- }
-
- return contents.toString();
- }
-
/**
* convert text to markdown style
*
- * @param title the title
+ * @param title the title
* @param content the content
* @return markdown text
*/
@@ -242,17 +205,17 @@ public final class WeChatSender {
return alertResult;
}
String enterpriseWeChatPushUrlReplace = "";
- Map<String,String> contentMap=new HashMap<>();
- contentMap.put(WeChatAlertConstants.WE_CHAT_CONTENT_KEY,data);
- String msgJson="";
+ Map<String, String> contentMap = new HashMap<>();
+ contentMap.put(WeChatAlertConstants.WE_CHAT_CONTENT_KEY, data);
+ String msgJson = "";
if (sendType.equals(WeChatType.APP.getDescp())) {
enterpriseWeChatPushUrlReplace = WeChatAlertConstants.WE_CHAT_PUSH_URL.replace(TOKEN_REGEX, weChatToken);
- WechatAppMessage wechatAppMessage=new WechatAppMessage(weChatUsers, WE_CHAT_MESSAGE_TYPE_TEXT, Integer.valueOf(weChatAgentId),contentMap, WE_CHAT_MESSAGE_SAFE_PUBLICITY, WE_CHAT_ENABLE_ID_TRANS, WE_CHAT_DUPLICATE_CHECK_INTERVAL_ZERO);
- msgJson=JSONUtils.toJsonString(wechatAppMessage);
+ WechatAppMessage wechatAppMessage = new WechatAppMessage(weChatUsers, showType, Integer.valueOf(weChatAgentIdChatId), contentMap, WE_CHAT_MESSAGE_SAFE_PUBLICITY, WE_CHAT_ENABLE_ID_TRANS, WE_CHAT_DUPLICATE_CHECK_INTERVAL_ZERO);
+ msgJson = JSONUtils.toJsonString(wechatAppMessage);
} else if (sendType.equals(WeChatType.APPCHAT.getDescp())) {
enterpriseWeChatPushUrlReplace = WeChatAlertConstants.WE_CHAT_APP_CHAT_PUSH_URL.replace(TOKEN_REGEX, weChatToken);
- WechatAppChatMessage wechatAppChatMessage=new WechatAppChatMessage(weChatChatId, WE_CHAT_MESSAGE_TYPE_TEXT, contentMap, WE_CHAT_MESSAGE_SAFE_PUBLICITY);
- msgJson=JSONUtils.toJsonString(wechatAppChatMessage);
+ WechatAppChatMessage wechatAppChatMessage = new WechatAppChatMessage(weChatAgentIdChatId, showType, contentMap, WE_CHAT_MESSAGE_SAFE_PUBLICITY);
+ msgJson = JSONUtils.toJsonString(wechatAppChatMessage);
}
try {
@@ -272,14 +235,7 @@ public final class WeChatSender {
* @return the markdown alert table/text
*/
private String markdownByAlert(String title, String content) {
- String result = "";
- if (showType.equals(ShowType.TABLE.getDescp())) {
- result = markdownTable(title, content);
- } else if (showType.equals(ShowType.TEXT.getDescp())) {
- result = markdownText(title, content);
- }
- return result;
-
+ return markdownText(title, content);
}
private String getToken() {
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WechatAppChatMessage.java b/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WechatAppChatMessage.java
index 5294972b3c..e04f7335b8 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WechatAppChatMessage.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WechatAppChatMessage.java
@@ -17,6 +17,8 @@
package org.apache.dolphinscheduler.plugin.alert.wechat;
+import org.apache.dolphinscheduler.alert.api.ShowType;
+
import java.util.Map;
public class WechatAppChatMessage {
@@ -24,6 +26,7 @@ public class WechatAppChatMessage {
private String chatid;
private String msgtype;
private Map<String,String> text;
+ private Map<String,String> markdown;
private Integer safe;
public String getChatid() {
@@ -58,13 +61,25 @@ public class WechatAppChatMessage {
this.safe = safe;
}
+ public Map<String, String> getMarkdown() {
+ return markdown;
+ }
+
+ public void setMarkdown(Map<String, String> markdown) {
+ this.markdown = markdown;
+ }
+
public WechatAppChatMessage() {
}
- public WechatAppChatMessage(String chatid, String msgtype, Map<String, String> text, Integer safe) {
+ public WechatAppChatMessage(String chatid, String msgtype, Map<String, String> contentMap, Integer safe) {
this.chatid = chatid;
this.msgtype = msgtype;
- this.text = text;
+ if (msgtype.equals(ShowType.MARKDOWN.getDescp())) {
+ this.markdown = contentMap;
+ } else {
+ this.text = contentMap;
+ }
this.safe = safe;
}
}
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WechatAppMessage.java b/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WechatAppMessage.java
index 4032e4856f..dbb74a6a93 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WechatAppMessage.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-plugins/dolphinscheduler-alert-wechat/src/main/java/org/apache/dolphinscheduler/plugin/alert/wechat/WechatAppMessage.java
@@ -17,6 +17,8 @@
package org.apache.dolphinscheduler.plugin.alert.wechat;
+import org.apache.dolphinscheduler.alert.api.ShowType;
+
import java.util.Map;
public class WechatAppMessage {
@@ -24,7 +26,8 @@ public class WechatAppMessage {
private String touser;
private String msgtype;
private Integer agentid;
- private Map<String,String> text;
+ private Map<String, String> text;
+ private Map<String, String> markdown;
private Integer safe;
private Integer enable_id_trans;
private Integer enable_duplicate_check;
@@ -85,16 +88,28 @@ public class WechatAppMessage {
this.enable_duplicate_check = enable_duplicate_check;
}
+ public Map<String, String> getMarkdown() {
+ return markdown;
+ }
+
+ public void setMarkdown(Map<String, String> markdown) {
+ this.markdown = markdown;
+ }
+
public WechatAppMessage() {
}
- public WechatAppMessage(String touser, String msgtype, Integer agentid, Map<String, String> text, Integer safe, Integer enable_id_trans, Integer enable_duplicate_check) {
+ public WechatAppMessage(String touser, String msgtype, Integer agentid, Map<String, String> contentMap, Integer safe, Integer enableIdTrans, Integer enableDuplicateCheck) {
this.touser = touser;
this.msgtype = msgtype;
this.agentid = agentid;
- this.text = text;
+ if (msgtype.equals(ShowType.MARKDOWN.getDescp())) {
+ this.markdown = contentMap;
+ } else {
+ this.text = contentMap;
+ }
this.safe = safe;
- this.enable_id_trans = enable_id_trans;
- this.enable_duplicate_check = enable_duplicate_check;
+ this.enable_id_trans = enableIdTrans;
+ this.enable_duplicate_check = enableDuplicateCheck;
}
}
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertPluginManager.java b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertPluginManager.java
index f590d31aee..cc84769304 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertPluginManager.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertPluginManager.java
@@ -41,8 +41,6 @@ import java.util.Optional;
import java.util.ServiceLoader;
import java.util.Set;
-import javax.annotation.PostConstruct;
-
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.context.event.ApplicationReadyEvent;
@@ -55,23 +53,23 @@ public final class AlertPluginManager {
private final PluginDao pluginDao;
- private final Map<Integer, AlertChannel> channelKeyedById = new HashMap<>();
-
- private final PluginParams warningTypeParams = getWarningTypeParams();
-
public AlertPluginManager(PluginDao pluginDao) {
this.pluginDao = pluginDao;
}
+ private final Map<Integer, AlertChannel> channelKeyedById = new HashMap<>();
+
+ private final PluginParams warningTypeParams = getWarningTypeParams();
+
public PluginParams getWarningTypeParams() {
return
- RadioParam.newBuilder(AlertConstants.NAME_WARNING_TYPE, AlertConstants.WARNING_TYPE)
- .addParamsOptions(new ParamsOptions(WarningType.SUCCESS.getDescp(), WarningType.SUCCESS.getDescp(), false))
- .addParamsOptions(new ParamsOptions(WarningType.FAILURE.getDescp(), WarningType.FAILURE.getDescp(), false))
- .addParamsOptions(new ParamsOptions(WarningType.ALL.getDescp(), WarningType.ALL.getDescp(), false))
- .setValue(WarningType.ALL.getDescp())
- .addValidate(Validate.newBuilder().setRequired(true).build())
- .build();
+ RadioParam.newBuilder(AlertConstants.NAME_WARNING_TYPE, AlertConstants.WARNING_TYPE)
+ .addParamsOptions(new ParamsOptions(WarningType.SUCCESS.getDescp(), WarningType.SUCCESS.getDescp(), false))
+ .addParamsOptions(new ParamsOptions(WarningType.FAILURE.getDescp(), WarningType.FAILURE.getDescp(), false))
+ .addParamsOptions(new ParamsOptions(WarningType.ALL.getDescp(), WarningType.ALL.getDescp(), false))
+ .setValue(WarningType.ALL.getDescp())
+ .addValidate(Validate.newBuilder().setRequired(true).build())
+ .build();
}
@EventListener
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertRequestProcessor.java b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertRequestProcessor.java
index c85292f725..6bb4aa0049 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertRequestProcessor.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertRequestProcessor.java
@@ -36,10 +36,10 @@ import io.netty.channel.Channel;
public final class AlertRequestProcessor implements NettyRequestProcessor {
private static final Logger logger = LoggerFactory.getLogger(AlertRequestProcessor.class);
- private final AlertSender alertSender;
+ private final AlertSenderService alertSenderService;
- public AlertRequestProcessor(AlertSender alertSender) {
- this.alertSender = alertSender;
+ public AlertRequestProcessor(AlertSenderService alertSenderService) {
+ this.alertSenderService = alertSenderService;
}
@Override
@@ -51,7 +51,7 @@ public final class AlertRequestProcessor implements NettyRequestProcessor {
logger.info("Received command : {}", alertSendRequestCommand);
- AlertSendResponseCommand alertSendResponseCommand = alertSender.syncHandler(
+ AlertSendResponseCommand alertSendResponseCommand = alertSenderService.syncHandler(
alertSendRequestCommand.getGroupId(),
alertSendRequestCommand.getTitle(),
alertSendRequestCommand.getContent(),
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertSender.java b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertSenderService.java
similarity index 84%
rename from dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertSender.java
rename to dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertSenderService.java
index 2f15fb65e3..c4dd1b5e87 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertSender.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertSenderService.java
@@ -17,43 +17,65 @@
package org.apache.dolphinscheduler.alert;
+import org.apache.commons.collections.CollectionUtils;
import org.apache.dolphinscheduler.alert.api.AlertChannel;
import org.apache.dolphinscheduler.alert.api.AlertConstants;
import org.apache.dolphinscheduler.alert.api.AlertData;
import org.apache.dolphinscheduler.alert.api.AlertInfo;
import org.apache.dolphinscheduler.alert.api.AlertResult;
+import org.apache.dolphinscheduler.common.Constants;
import org.apache.dolphinscheduler.common.enums.AlertStatus;
import org.apache.dolphinscheduler.common.enums.WarningType;
+import org.apache.dolphinscheduler.common.thread.Stopper;
+import org.apache.dolphinscheduler.common.thread.ThreadUtils;
import org.apache.dolphinscheduler.common.utils.JSONUtils;
import org.apache.dolphinscheduler.dao.AlertDao;
import org.apache.dolphinscheduler.dao.entity.Alert;
import org.apache.dolphinscheduler.dao.entity.AlertPluginInstance;
import org.apache.dolphinscheduler.remote.command.alert.AlertSendResponseCommand;
import org.apache.dolphinscheduler.remote.command.alert.AlertSendResponseResult;
-
-import org.apache.commons.collections.CollectionUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.stereotype.Service;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.Optional;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.stereotype.Component;
-
-@Component
-public final class AlertSender {
- private static final Logger logger = LoggerFactory.getLogger(AlertSender.class);
+@Service
+public final class AlertSenderService extends Thread {
+ private static final Logger logger = LoggerFactory.getLogger(AlertSenderService.class);
private final AlertDao alertDao;
private final AlertPluginManager alertPluginManager;
- public AlertSender(AlertDao alertDao, AlertPluginManager alertPluginManager) {
+ public AlertSenderService(AlertDao alertDao, AlertPluginManager alertPluginManager) {
this.alertDao = alertDao;
this.alertPluginManager = alertPluginManager;
}
+ @Override
+ public synchronized void start() {
+ super.setName("AlertSenderService");
+ super.start();
+ }
+
+ @Override
+ public void run() {
+ logger.info("alert sender started");
+ while (Stopper.isRunning()) {
+ try {
+ List<Alert> alerts = alertDao.listPendingAlerts();
+ this.send(alerts);
+ ThreadUtils.sleep(Constants.SLEEP_TIME_MILLIS * 5L);
+ } catch (Exception e) {
+ logger.error("alert sender thread error", e);
+ }
+ }
+ }
+
+
public void send(List<Alert> alerts) {
for (Alert alert : alerts) {
//get alert group from alert
@@ -66,11 +88,11 @@ public final class AlertSender {
}
AlertData alertData = new AlertData();
alertData.setId(alert.getId())
- .setContent(alert.getContent())
- .setLog(alert.getLog())
- .setTitle(alert.getTitle())
- .setTitle(alert.getTitle())
- .setWarnType(alert.getWarningType().getCode());
+ .setContent(alert.getContent())
+ .setLog(alert.getLog())
+ .setTitle(alert.getTitle())
+ .setTitle(alert.getTitle())
+ .setWarnType(alert.getWarningType().getCode());
for (AlertPluginInstance instance : alertInstanceList) {
@@ -81,23 +103,22 @@ public final class AlertSender {
}
}
}
-
}
/**
* sync send alert handler
*
* @param alertGroupId alertGroupId
- * @param title title
- * @param content content
+ * @param title title
+ * @param content content
* @return AlertSendResponseCommand
*/
- public AlertSendResponseCommand syncHandler(int alertGroupId, String title, String content , int warnType) {
+ public AlertSendResponseCommand syncHandler(int alertGroupId, String title, String content, int warnType) {
List<AlertPluginInstance> alertInstanceList = alertDao.listInstanceByAlertGroupId(alertGroupId);
AlertData alertData = new AlertData();
alertData.setContent(content)
- .setTitle(title)
- .setWarnType(warnType);
+ .setTitle(title)
+ .setWarnType(warnType);
boolean sendResponseStatus = true;
List<AlertSendResponseResult> sendResponseResults = new ArrayList<>();
@@ -116,7 +137,7 @@ public final class AlertSender {
AlertResult alertResult = this.alertResultHandler(instance, alertData);
if (alertResult != null) {
AlertSendResponseResult alertSendResponseResult = new AlertSendResponseResult(
- Boolean.parseBoolean(String.valueOf(alertResult.getStatus())), alertResult.getMessage());
+ Boolean.parseBoolean(String.valueOf(alertResult.getStatus())), alertResult.getMessage());
sendResponseStatus = sendResponseStatus && alertSendResponseResult.getStatus();
sendResponseResults.add(alertSendResponseResult);
}
@@ -128,7 +149,7 @@ public final class AlertSender {
/**
* alert result handler
*
- * @param instance instance
+ * @param instance instance
* @param alertData alertData
* @return AlertResult
*/
@@ -147,7 +168,7 @@ public final class AlertSender {
Map<String, String> paramsMap = JSONUtils.toMap(instance.getPluginInstanceParams());
String instanceWarnType = WarningType.ALL.getDescp();
- if(paramsMap != null){
+ if (paramsMap != null) {
instanceWarnType = paramsMap.getOrDefault(AlertConstants.NAME_WARNING_TYPE, WarningType.ALL.getDescp());
}
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertServer.java b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertServer.java
index 6b18befe39..60bd09e6e2 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertServer.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/main/java/org/apache/dolphinscheduler/alert/AlertServer.java
@@ -17,73 +17,95 @@
package org.apache.dolphinscheduler.alert;
+import org.apache.dolphinscheduler.common.Constants;
import org.apache.dolphinscheduler.common.thread.Stopper;
-import org.apache.dolphinscheduler.dao.AlertDao;
+import org.apache.dolphinscheduler.common.thread.ThreadUtils;
import org.apache.dolphinscheduler.dao.PluginDao;
-import org.apache.dolphinscheduler.dao.entity.Alert;
import org.apache.dolphinscheduler.remote.NettyRemotingServer;
import org.apache.dolphinscheduler.remote.command.CommandType;
import org.apache.dolphinscheduler.remote.config.NettyServerConfig;
-
-import java.io.Closeable;
-import java.util.List;
-import java.util.concurrent.Executors;
-import java.util.concurrent.TimeUnit;
-
-import javax.annotation.PreDestroy;
-
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.boot.SpringApplication;
+import org.springframework.boot.WebApplicationType;
import org.springframework.boot.autoconfigure.SpringBootApplication;
+import org.springframework.boot.builder.SpringApplicationBuilder;
import org.springframework.boot.context.event.ApplicationReadyEvent;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.event.EventListener;
+import javax.annotation.PreDestroy;
+import java.io.Closeable;
+
@SpringBootApplication
@ComponentScan("org.apache.dolphinscheduler")
public class AlertServer implements Closeable {
private static final Logger logger = LoggerFactory.getLogger(AlertServer.class);
private final PluginDao pluginDao;
- private final AlertDao alertDao;
- private final AlertPluginManager alertPluginManager;
- private final AlertSender alertSender;
+ private final AlertSenderService alertSenderService;
private final AlertRequestProcessor alertRequestProcessor;
+ private final AlertConfig alertConfig;
+ private NettyRemotingServer nettyRemotingServer;
- private NettyRemotingServer server;
-
- @Autowired
- private AlertConfig config;
-
- public AlertServer(PluginDao pluginDao, AlertDao alertDao, AlertPluginManager alertPluginManager, AlertSender alertSender, AlertRequestProcessor alertRequestProcessor) {
+ public AlertServer(PluginDao pluginDao, AlertSenderService alertSenderService, AlertRequestProcessor alertRequestProcessor, AlertConfig alertConfig) {
this.pluginDao = pluginDao;
- this.alertDao = alertDao;
- this.alertPluginManager = alertPluginManager;
- this.alertSender = alertSender;
+ this.alertSenderService = alertSenderService;
this.alertRequestProcessor = alertRequestProcessor;
+ this.alertConfig = alertConfig;
}
+ /**
+ * alert server startup, not use web service
+ *
+ * @param args arguments
+ */
public static void main(String[] args) {
- SpringApplication.run(AlertServer.class, args);
+ Thread.currentThread().setName(Constants.THREAD_NAME_ALERT_SERVER);
+ new SpringApplicationBuilder(AlertServer.class).web(WebApplicationType.NONE).run(args);
}
@EventListener
- public void start(ApplicationReadyEvent readyEvent) {
- logger.info("Starting Alert server");
+ public void run(ApplicationReadyEvent readyEvent) {
+ logger.info("alert server starting...");
checkTable();
startServer();
-
- Executors.newScheduledThreadPool(1)
- .scheduleAtFixedRate(new Sender(), 5, 5, TimeUnit.SECONDS);
+ alertSenderService.start();
}
@Override
@PreDestroy
public void close() {
- server.close();
+ destroy("alert server destroy");
+ }
+
+ /**
+ * gracefully stop
+ *
+ * @param cause stop cause
+ */
+ public void destroy(String cause) {
+
+ try {
+ // execute only once
+ if (Stopper.isStopped()) {
+ return;
+ }
+
+ logger.info("alert server is stopping ..., cause : {}", cause);
+
+ // set stop signal is true
+ Stopper.stop();
+
+ // thread sleep 3 seconds for thread quietly stop
+ ThreadUtils.sleep(3000L);
+
+ // close
+ this.nettyRemotingServer.close();
+
+ } catch (Exception e) {
+ logger.error("alert server stop exception ", e);
+ }
}
private void checkTable() {
@@ -95,26 +117,11 @@ public class AlertServer implements Closeable {
private void startServer() {
NettyServerConfig serverConfig = new NettyServerConfig();
- serverConfig.setListenPort(config.getPort());
+ serverConfig.setListenPort(alertConfig.getPort());
- server = new NettyRemotingServer(serverConfig);
- server.registerProcessor(CommandType.ALERT_SEND_REQUEST, alertRequestProcessor);
- server.start();
+ nettyRemotingServer = new NettyRemotingServer(serverConfig);
+ nettyRemotingServer.registerProcessor(CommandType.ALERT_SEND_REQUEST, alertRequestProcessor);
+ nettyRemotingServer.start();
}
- final class Sender implements Runnable {
- @Override
- public void run() {
- if (!Stopper.isRunning()) {
- return;
- }
-
- try {
- final List<Alert> alerts = alertDao.listPendingAlerts();
- alertSender.send(alerts);
- } catch (Exception e) {
- logger.error("Failed to send alert", e);
- }
- }
- }
}
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/AlertServerTest.java b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/AlertServerTest.java
index 911b912d76..c37cf6fe7a 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/AlertServerTest.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/AlertServerTest.java
@@ -18,7 +18,9 @@
package org.apache.dolphinscheduler.alert;
import junit.framework.TestCase;
+import org.apache.dolphinscheduler.dao.AlertDao;
import org.apache.dolphinscheduler.dao.PluginDao;
+import org.apache.dolphinscheduler.dao.entity.Alert;
import org.apache.dolphinscheduler.remote.NettyRemotingServer;
import org.apache.dolphinscheduler.remote.config.NettyServerConfig;
import org.junit.Assert;
@@ -27,9 +29,13 @@ import org.junit.runner.RunWith;
import org.mockito.InjectMocks;
import org.mockito.Mock;
import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
import org.mockito.junit.MockitoJUnitRunner;
import org.powermock.reflect.Whitebox;
+import java.util.ArrayList;
+import java.util.List;
+
@RunWith(MockitoJUnitRunner.class)
public class AlertServerTest extends TestCase {
@@ -42,19 +48,26 @@ public class AlertServerTest extends TestCase {
@Mock
private AlertConfig alertConfig;
+
+ @Mock
+ private AlertSenderService alertSenderService;
@Test
public void testStart() {
+
Mockito.when(pluginDao.checkPluginDefineTableExist()).thenReturn(true);
Mockito.when(alertConfig.getPort()).thenReturn(50053);
-
- alertServer.start(null);
+
+ Mockito.doNothing().when(alertSenderService).start();
+
+ alertServer.run(null);
- NettyRemotingServer nettyRemotingServer = Whitebox.getInternalState(alertServer, "server");
+ NettyRemotingServer nettyRemotingServer = Whitebox.getInternalState(alertServer, "nettyRemotingServer");
NettyServerConfig nettyServerConfig = Whitebox.getInternalState(nettyRemotingServer, "serverConfig");
Assert.assertEquals(50053, nettyServerConfig.getListenPort());
+
}
}
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/processor/AlertRequestProcessorTest.java b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/processor/AlertRequestProcessorTest.java
index 64e92a02ad..41277fad50 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/processor/AlertRequestProcessorTest.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/processor/AlertRequestProcessorTest.java
@@ -20,30 +20,35 @@ package org.apache.dolphinscheduler.alert.processor;
import static org.mockito.Mockito.mock;
import org.apache.dolphinscheduler.alert.AlertRequestProcessor;
-import org.apache.dolphinscheduler.alert.AlertSender;
+import org.apache.dolphinscheduler.alert.AlertSenderService;
import org.apache.dolphinscheduler.common.enums.WarningType;
-import org.apache.dolphinscheduler.dao.AlertDao;
import org.apache.dolphinscheduler.remote.command.Command;
import org.apache.dolphinscheduler.remote.command.CommandType;
import org.apache.dolphinscheduler.remote.command.alert.AlertSendRequestCommand;
+import org.apache.dolphinscheduler.remote.command.alert.AlertSendResponseCommand;
import org.junit.Assert;
-import org.junit.Before;
import org.junit.Test;
import io.netty.channel.Channel;
+import org.junit.runner.RunWith;
+import org.mockito.InjectMocks;
+import org.mockito.Mock;
+import org.mockito.Mockito;
+import org.mockito.junit.MockitoJUnitRunner;
+@RunWith(MockitoJUnitRunner.class)
public class AlertRequestProcessorTest {
+ @InjectMocks
private AlertRequestProcessor alertRequestProcessor;
- @Before
- public void before() {
- final AlertDao alertDao = mock(AlertDao.class);
- alertRequestProcessor = new AlertRequestProcessor(new AlertSender(alertDao, null));
- }
+ @Mock
+ private AlertSenderService alertSenderService;
+
@Test
public void testProcess() {
+ Mockito.when(alertSenderService.syncHandler(1, "title", "content", WarningType.FAILURE.getCode())).thenReturn(new AlertSendResponseCommand());
Channel channel = mock(Channel.class);
AlertSendRequestCommand alertSendRequestCommand = new AlertSendRequestCommand(1, "title", "content", WarningType.FAILURE.getCode());
Command reqCommand = alertSendRequestCommand.convert2Command();
diff --git a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/runner/AlertSenderTest.java b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/runner/AlertSenderServiceTest.java
similarity index 86%
rename from dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/runner/AlertSenderTest.java
rename to dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/runner/AlertSenderServiceTest.java
index 4060e46345..7b7853daf5 100644
--- a/dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/runner/AlertSenderTest.java
+++ b/dolphinscheduler-alert/dolphinscheduler-alert-server/src/test/java/org/apache/dolphinscheduler/alert/runner/AlertSenderServiceTest.java
@@ -21,7 +21,7 @@ import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import org.apache.dolphinscheduler.alert.AlertPluginManager;
-import org.apache.dolphinscheduler.alert.AlertSender;
+import org.apache.dolphinscheduler.alert.AlertSenderService;
import org.apache.dolphinscheduler.alert.api.AlertChannel;
import org.apache.dolphinscheduler.alert.api.AlertResult;
import org.apache.dolphinscheduler.common.enums.WarningType;
@@ -39,24 +39,29 @@ import java.util.Optional;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
+import org.mockito.InjectMocks;
+import org.mockito.Mock;
import org.mockito.Mockito;
+import org.mockito.MockitoAnnotations;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-public class AlertSenderTest {
- private static final Logger logger = LoggerFactory.getLogger(AlertSenderTest.class);
+public class AlertSenderServiceTest {
+ private static final Logger logger = LoggerFactory.getLogger(AlertSenderServiceTest.class);
+ @Mock
private AlertDao alertDao;
+ @Mock
private PluginDao pluginDao;
+ @Mock
private AlertPluginManager alertPluginManager;
- private AlertSender alertSender;
+ @InjectMocks
+ private AlertSenderService alertSenderService;
@Before
public void before() {
- alertDao = mock(AlertDao.class);
- pluginDao = mock(PluginDao.class);
- alertPluginManager = mock(AlertPluginManager.class);
+ MockitoAnnotations.initMocks(this);
}
@Test
@@ -65,12 +70,11 @@ public class AlertSenderTest {
int alertGroupId = 1;
String title = "alert mail test title";
String content = "alert mail test content";
- alertSender = new AlertSender(alertDao, alertPluginManager);
//1.alert instance does not exist
when(alertDao.listInstanceByAlertGroupId(alertGroupId)).thenReturn(null);
- AlertSendResponseCommand alertSendResponseCommand = alertSender.syncHandler(alertGroupId, title, content, WarningType.ALL.getCode());
+ AlertSendResponseCommand alertSendResponseCommand = alertSenderService.syncHandler(alertGroupId, title, content, WarningType.ALL.getCode());
Assert.assertFalse(alertSendResponseCommand.getResStatus());
alertSendResponseCommand.getResResults().forEach(result ->
logger.info("alert send response result, status:{}, message:{}", result.getStatus(), result.getMessage()));
@@ -89,7 +93,7 @@ public class AlertSenderTest {
PluginDefine pluginDefine = new PluginDefine(pluginName, "1", null);
when(pluginDao.getPluginDefineById(pluginDefineId)).thenReturn(pluginDefine);
- alertSendResponseCommand = alertSender.syncHandler(alertGroupId, title, content, WarningType.ALL.getCode());
+ alertSendResponseCommand = alertSenderService.syncHandler(alertGroupId, title, content, WarningType.ALL.getCode());
Assert.assertFalse(alertSendResponseCommand.getResStatus());
alertSendResponseCommand.getResResults().forEach(result ->
logger.info("alert send response result, status:{}, message:{}", result.getStatus(), result.getMessage()));
@@ -99,7 +103,7 @@ public class AlertSenderTest {
when(alertChannelMock.process(Mockito.any())).thenReturn(null);
when(alertPluginManager.getAlertChannel(1)).thenReturn(Optional.of(alertChannelMock));
- alertSendResponseCommand = alertSender.syncHandler(alertGroupId, title, content, WarningType.ALL.getCode());
+ alertSendResponseCommand = alertSenderService.syncHandler(alertGroupId, title, content, WarningType.ALL.getCode());
Assert.assertFalse(alertSendResponseCommand.getResStatus());
alertSendResponseCommand.getResResults().forEach(result ->
logger.info("alert send response result, status:{}, message:{}", result.getStatus(), result.getMessage()));
@@ -111,7 +115,7 @@ public class AlertSenderTest {
when(alertChannelMock.process(Mockito.any())).thenReturn(alertResult);
when(alertPluginManager.getAlertChannel(1)).thenReturn(Optional.of(alertChannelMock));
- alertSendResponseCommand = alertSender.syncHandler(alertGroupId, title, content, WarningType.ALL.getCode());
+ alertSendResponseCommand = alertSenderService.syncHandler(alertGroupId, title, content, WarningType.ALL.getCode());
Assert.assertFalse(alertSendResponseCommand.getResStatus());
alertSendResponseCommand.getResResults().forEach(result ->
logger.info("alert send response result, status:{}, message:{}", result.getStatus(), result.getMessage()));
@@ -123,7 +127,7 @@ public class AlertSenderTest {
when(alertChannelMock.process(Mockito.any())).thenReturn(alertResult);
when(alertPluginManager.getAlertChannel(1)).thenReturn(Optional.of(alertChannelMock));
- alertSendResponseCommand = alertSender.syncHandler(alertGroupId, title, content, WarningType.ALL.getCode());
+ alertSendResponseCommand = alertSenderService.syncHandler(alertGroupId, title, content, WarningType.ALL.getCode());
Assert.assertTrue(alertSendResponseCommand.getResStatus());
alertSendResponseCommand.getResResults().forEach(result ->
logger.info("alert send response result, status:{}, message:{}", result.getStatus(), result.getMessage()));
@@ -143,7 +147,7 @@ public class AlertSenderTest {
alert.setWarningType(WarningType.FAILURE);
alertList.add(alert);
- alertSender = new AlertSender(alertDao, alertPluginManager);
+// alertSenderService = new AlertSenderService();
int pluginDefineId = 1;
String pluginInstanceParams = "alert-instance-mail-params";
@@ -165,6 +169,7 @@ public class AlertSenderTest {
when(alertChannelMock.process(Mockito.any())).thenReturn(alertResult);
when(alertPluginManager.getAlertChannel(1)).thenReturn(Optional.of(alertChannelMock));
Assert.assertTrue(Boolean.parseBoolean(alertResult.getStatus()));
- alertSender.send(alertList);
+ when(alertDao.listInstanceByAlertGroupId(1)).thenReturn(new ArrayList<>());
+ alertSenderService.send(alertList);
}
}
diff --git a/dolphinscheduler-api/pom.xml b/dolphinscheduler-api/pom.xml
index d162239c7c..156a4d1160 100644
--- a/dolphinscheduler-api/pom.xml
+++ b/dolphinscheduler-api/pom.xml
@@ -275,6 +275,12 @@
</exclusions>
<scope>test</scope>
</dependency>
+
+ <!-- Python -->
+ <dependency>
+ <groupId>net.sf.py4j</groupId>
+ <artifactId>py4j</artifactId>
+ </dependency>
</dependencies>
<build>
diff --git a/dolphinscheduler-api/src/main/docker/Dockerfile b/dolphinscheduler-api/src/main/docker/Dockerfile
index 9a1a8c0ca1..946f2ff46a 100644
--- a/dolphinscheduler-api/src/main/docker/Dockerfile
+++ b/dolphinscheduler-api/src/main/docker/Dockerfile
@@ -29,6 +29,6 @@ WORKDIR $DOLPHINSCHEDULER_HOME
ADD ./target/api-server $DOLPHINSCHEDULER_HOME
-EXPOSE 12345
+EXPOSE 12345 25333
CMD [ "/bin/bash", "./bin/start.sh" ]
diff --git a/dolphinscheduler-python/src/main/java/org/apache/dolphinscheduler/server/config/PythonGatewayConfig.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/configuration/PythonGatewayConfiguration.java
similarity index 86%
rename from dolphinscheduler-python/src/main/java/org/apache/dolphinscheduler/server/config/PythonGatewayConfig.java
rename to dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/configuration/PythonGatewayConfiguration.java
index f06ea765b6..5735e27fd2 100644
--- a/dolphinscheduler-python/src/main/java/org/apache/dolphinscheduler/server/config/PythonGatewayConfig.java
+++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/configuration/PythonGatewayConfiguration.java
@@ -15,17 +15,17 @@
* limitations under the License.
*/
-package org.apache.dolphinscheduler.server.config;
+package org.apache.dolphinscheduler.api.configuration;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
-import org.springframework.context.annotation.Configuration;
import org.springframework.stereotype.Component;
@Component
@EnableConfigurationProperties
-@ConfigurationProperties("python-gateway")
-public class PythonGatewayConfig {
+@ConfigurationProperties(value = "python-gateway", ignoreUnknownFields = false)
+public class PythonGatewayConfiguration {
+ private boolean enabled;
private String gatewayServerAddress;
private int gatewayServerPort;
private String pythonAddress;
@@ -33,6 +33,14 @@ public class PythonGatewayConfig {
private int connectTimeout;
private int readTimeout;
+ public boolean getEnabled() {
+ return enabled;
+ }
+
+ public void setEnabled(boolean enabled) {
+ this.enabled = enabled;
+ }
+
public String getGatewayServerAddress() {
return gatewayServerAddress;
}
diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java
index 10c6177924..0a6fed5072 100644
--- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java
+++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/controller/ExecutorController.java
@@ -95,22 +95,22 @@ public class ExecutorController extends BaseController {
*/
@ApiOperation(value = "startProcessInstance", notes = "RUN_PROCESS_INSTANCE_NOTES")
@ApiImplicitParams({
- @ApiImplicitParam(name = "processDefinitionCode", value = "PROCESS_DEFINITION_CODE", required = true, dataType = "Long", example = "100"),
- @ApiImplicitParam(name = "scheduleTime", value = "SCHEDULE_TIME", dataType = "String"),
- @ApiImplicitParam(name = "failureStrategy", value = "FAILURE_STRATEGY", required = true, dataType = "FailureStrategy"),
- @ApiImplicitParam(name = "startNodeList", value = "START_NODE_LIST", dataType = "String"),
- @ApiImplicitParam(name = "taskDependType", value = "TASK_DEPEND_TYPE", dataType = "TaskDependType"),
- @ApiImplicitParam(name = "execType", value = "COMMAND_TYPE", dataType = "CommandType"),
- @ApiImplicitParam(name = "warningType", value = "WARNING_TYPE", required = true, dataType = "WarningType"),
- @ApiImplicitParam(name = "warningGroupId", value = "WARNING_GROUP_ID", dataType = "Int", example = "100"),
- @ApiImplicitParam(name = "runMode", value = "RUN_MODE", dataType = "RunMode"),
- @ApiImplicitParam(name = "processInstancePriority", value = "PROCESS_INSTANCE_PRIORITY", required = true, dataType = "Priority"),
- @ApiImplicitParam(name = "workerGroup", value = "WORKER_GROUP", dataType = "String", example = "default"),
- @ApiImplicitParam(name = "environmentCode", value = "ENVIRONMENT_CODE", dataType = "Long", example = "-1"),
- @ApiImplicitParam(name = "timeout", value = "TIMEOUT", dataType = "Int", example = "100"),
- @ApiImplicitParam(name = "expectedParallelismNumber", value = "EXPECTED_PARALLELISM_NUMBER", dataType = "Int" , example = "8"),
- @ApiImplicitParam(name = "dryRun", value = "DRY_RUN", dataType = "Int", example = "0"),
- @ApiImplicitParam(name = "complementDependentMode", value = "COMPLEMENT_DEPENDENT_MODE", dataType = "complementDependentMode")
+ @ApiImplicitParam(name = "processDefinitionCode", value = "PROCESS_DEFINITION_CODE", required = true, dataType = "Long", example = "100"),
+ @ApiImplicitParam(name = "scheduleTime", value = "SCHEDULE_TIME", required = true, dataType = "String", example = "2022-04-06 00:00:00,2022-04-06 00:00:00"),
+ @ApiImplicitParam(name = "failureStrategy", value = "FAILURE_STRATEGY", required = true, dataType = "FailureStrategy"),
+ @ApiImplicitParam(name = "startNodeList", value = "START_NODE_LIST", dataType = "String"),
+ @ApiImplicitParam(name = "taskDependType", value = "TASK_DEPEND_TYPE", dataType = "TaskDependType"),
+ @ApiImplicitParam(name = "execType", value = "COMMAND_TYPE", dataType = "CommandType"),
+ @ApiImplicitParam(name = "warningType", value = "WARNING_TYPE", required = true, dataType = "WarningType"),
+ @ApiImplicitParam(name = "warningGroupId", value = "WARNING_GROUP_ID", dataType = "Int", example = "100"),
+ @ApiImplicitParam(name = "runMode", value = "RUN_MODE", dataType = "RunMode"),
+ @ApiImplicitParam(name = "processInstancePriority", value = "PROCESS_INSTANCE_PRIORITY", required = true, dataType = "Priority"),
+ @ApiImplicitParam(name = "workerGroup", value = "WORKER_GROUP", dataType = "String", example = "default"),
+ @ApiImplicitParam(name = "environmentCode", value = "ENVIRONMENT_CODE", dataType = "Long", example = "-1"),
+ @ApiImplicitParam(name = "timeout", value = "TIMEOUT", dataType = "Int", example = "100"),
+ @ApiImplicitParam(name = "expectedParallelismNumber", value = "EXPECTED_PARALLELISM_NUMBER", dataType = "Int" , example = "8"),
+ @ApiImplicitParam(name = "dryRun", value = "DRY_RUN", dataType = "Int", example = "0"),
+ @ApiImplicitParam(name = "complementDependentMode", value = "COMPLEMENT_DEPENDENT_MODE", dataType = "complementDependentMode")
})
@PostMapping(value = "start-process-instance")
@ResponseStatus(HttpStatus.OK)
@@ -119,7 +119,7 @@ public class ExecutorController extends BaseController {
public Result startProcessInstance(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
@ApiParam(name = "projectCode", value = "PROJECT_CODE", required = true) @PathVariable long projectCode,
@RequestParam(value = "processDefinitionCode") long processDefinitionCode,
- @RequestParam(value = "scheduleTime", required = false) String scheduleTime,
+ @RequestParam(value = "scheduleTime") String scheduleTime,
@RequestParam(value = "failureStrategy") FailureStrategy failureStrategy,
@RequestParam(value = "startNodeList", required = false) String startNodeList,
@RequestParam(value = "taskDependType", required = false) TaskDependType taskDependType,
@@ -159,7 +159,7 @@ public class ExecutorController extends BaseController {
* batch execute process instance
* If any processDefinitionCode cannot be found, the failure information is returned and the status is set to
* failed. The successful task will run normally and will not stop
- *
+ *
* @param loginUser login user
* @param projectCode project code
* @param processDefinitionCodes process definition codes
@@ -180,7 +180,7 @@ public class ExecutorController extends BaseController {
@ApiOperation(value = "batchStartProcessInstance", notes = "BATCH_RUN_PROCESS_INSTANCE_NOTES")
@ApiImplicitParams({
@ApiImplicitParam(name = "processDefinitionCodes", value = "PROCESS_DEFINITION_CODES", required = true, dataType = "String", example = "1,2,3"),
- @ApiImplicitParam(name = "scheduleTime", value = "SCHEDULE_TIME", required = true, dataType = "String"),
+ @ApiImplicitParam(name = "scheduleTime", value = "SCHEDULE_TIME", required = true, dataType = "String", example = "2022-04-06 00:00:00,2022-04-06 00:00:00"),
@ApiImplicitParam(name = "failureStrategy", value = "FAILURE_STRATEGY", required = true, dataType = "FailureStrategy"),
@ApiImplicitParam(name = "startNodeList", value = "START_NODE_LIST", dataType = "String"),
@ApiImplicitParam(name = "taskDependType", value = "TASK_DEPEND_TYPE", dataType = "TaskDependType"),
@@ -201,24 +201,24 @@ public class ExecutorController extends BaseController {
@ApiException(START_PROCESS_INSTANCE_ERROR)
@AccessLogAnnotation(ignoreRequestArgs = "loginUser")
public Result batchStartProcessInstance(@ApiIgnore @RequestAttribute(value = Constants.SESSION_USER) User loginUser,
- @ApiParam(name = "projectCode", value = "PROJECT_CODE", required = true) @PathVariable long projectCode,
- @RequestParam(value = "processDefinitionCodes") String processDefinitionCodes,
- @RequestParam(value = "scheduleTime", required = false) String scheduleTime,
- @RequestParam(value = "failureStrategy") FailureStrategy failureStrategy,
- @RequestParam(value = "startNodeList", required = false) String startNodeList,
- @RequestParam(value = "taskDependType", required = false) TaskDependType taskDependType,
- @RequestParam(value = "execType", required = false) CommandType execType,
- @RequestParam(value = "warningType") WarningType warningType,
- @RequestParam(value = "warningGroupId", required = false) int warningGroupId,
- @RequestParam(value = "runMode", required = false) RunMode runMode,
- @RequestParam(value = "processInstancePriority", required = false) Priority processInstancePriority,
- @RequestParam(value = "workerGroup", required = false, defaultValue = "default") String workerGroup,
- @RequestParam(value = "environmentCode", required = false, defaultValue = "-1") Long environmentCode,
- @RequestParam(value = "timeout", required = false) Integer timeout,
- @RequestParam(value = "startParams", required = false) String startParams,
- @RequestParam(value = "expectedParallelismNumber", required = false) Integer expectedParallelismNumber,
- @RequestParam(value = "dryRun", defaultValue = "0", required = false) int dryRun,
- @RequestParam(value = "complementDependentMode", required = false) ComplementDependentMode complementDependentMode) {
+ @ApiParam(name = "projectCode", value = "PROJECT_CODE", required = true) @PathVariable long projectCode,
+ @RequestParam(value = "processDefinitionCodes") String processDefinitionCodes,
+ @RequestParam(value = "scheduleTime") String scheduleTime,
+ @RequestParam(value = "failureStrategy") FailureStrategy failureStrategy,
+ @RequestParam(value = "startNodeList", required = false) String startNodeList,
+ @RequestParam(value = "taskDependType", required = false) TaskDependType taskDependType,
+ @RequestParam(value = "execType", required = false) CommandType execType,
+ @RequestParam(value = "warningType") WarningType warningType,
+ @RequestParam(value = "warningGroupId", required = false) int warningGroupId,
+ @RequestParam(value = "runMode", required = false) RunMode runMode,
+ @RequestParam(value = "processInstancePriority", required = false) Priority processInstancePriority,
+ @RequestParam(value = "workerGroup", required = false, defaultValue = "default") String workerGroup,
+ @RequestParam(value = "environmentCode", required = false, defaultValue = "-1") Long environmentCode,
+ @RequestParam(value = "timeout", required = false) Integer timeout,
+ @RequestParam(value = "startParams", required = false) String startParams,
+ @RequestParam(value = "expectedParallelismNumber", required = false) Integer expectedParallelismNumber,
+ @RequestParam(value = "dryRun", defaultValue = "0", required = false) int dryRun,
+ @RequestParam(value = "complementDependentMode", required = false) ComplementDependentMode complementDependentMode) {
if (timeout == null) {
timeout = Constants.MAX_TASK_TIMEOUT;
@@ -269,8 +269,8 @@ public class ExecutorController extends BaseController {
*/
@ApiOperation(value = "execute", notes = "EXECUTE_ACTION_TO_PROCESS_INSTANCE_NOTES")
@ApiImplicitParams({
- @ApiImplicitParam(name = "processInstanceId", value = "PROCESS_INSTANCE_ID", required = true, dataType = "Int", example = "100"),
- @ApiImplicitParam(name = "executeType", value = "EXECUTE_TYPE", required = true, dataType = "ExecuteType")
+ @ApiImplicitParam(name = "processInstanceId", value = "PROCESS_INSTANCE_ID", required = true, dataType = "Int", example = "100"),
+ @ApiImplicitParam(name = "executeType", value = "EXECUTE_TYPE", required = true, dataType = "ExecuteType")
})
@PostMapping(value = "/execute")
@ResponseStatus(HttpStatus.OK)
@@ -293,7 +293,7 @@ public class ExecutorController extends BaseController {
*/
@ApiOperation(value = "startCheckProcessDefinition", notes = "START_CHECK_PROCESS_DEFINITION_NOTES")
@ApiImplicitParams({
- @ApiImplicitParam(name = "processDefinitionCode", value = "PROCESS_DEFINITION_CODE", required = true, dataType = "Long", example = "100")
+ @ApiImplicitParam(name = "processDefinitionCode", value = "PROCESS_DEFINITION_CODE", required = true, dataType = "Long", example = "100")
})
@PostMapping(value = "/start-check")
@ResponseStatus(HttpStatus.OK)
diff --git a/dolphinscheduler-python/src/main/java/org/apache/dolphinscheduler/server/PythonGatewayServer.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/python/PythonGateway.java
similarity index 95%
rename from dolphinscheduler-python/src/main/java/org/apache/dolphinscheduler/server/PythonGatewayServer.java
rename to dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/python/PythonGateway.java
index 2f7fc94da7..8aea47471b 100644
--- a/dolphinscheduler-python/src/main/java/org/apache/dolphinscheduler/server/PythonGatewayServer.java
+++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/python/PythonGateway.java
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.dolphinscheduler.server;
+package org.apache.dolphinscheduler.api.python;
import org.apache.dolphinscheduler.api.dto.resources.ResourceComponent;
import org.apache.dolphinscheduler.api.enums.Status;
@@ -56,7 +56,7 @@ import org.apache.dolphinscheduler.dao.mapper.ProjectMapper;
import org.apache.dolphinscheduler.dao.mapper.ProjectUserMapper;
import org.apache.dolphinscheduler.dao.mapper.ScheduleMapper;
import org.apache.dolphinscheduler.dao.mapper.TaskDefinitionMapper;
-import org.apache.dolphinscheduler.server.config.PythonGatewayConfig;
+import org.apache.dolphinscheduler.api.configuration.PythonGatewayConfiguration;
import org.apache.dolphinscheduler.spi.enums.ResourceType;
import org.apache.commons.collections.CollectionUtils;
@@ -75,17 +75,13 @@ import javax.annotation.PostConstruct;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.boot.SpringApplication;
-import org.springframework.boot.autoconfigure.SpringBootApplication;
-import org.springframework.boot.web.servlet.support.SpringBootServletInitializer;
-import org.springframework.context.annotation.ComponentScan;
+import org.springframework.stereotype.Component;
import py4j.GatewayServer;
-@SpringBootApplication
-@ComponentScan(value = "org.apache.dolphinscheduler")
-public class PythonGatewayServer extends SpringBootServletInitializer {
- private static final Logger logger = LoggerFactory.getLogger(PythonGatewayServer.class);
+@Component
+public class PythonGateway {
+ private static final Logger logger = LoggerFactory.getLogger(PythonGateway.class);
private static final WarningType DEFAULT_WARNING_TYPE = WarningType.NONE;
private static final int DEFAULT_WARNING_GROUP_ID = 0;
@@ -141,7 +137,7 @@ public class PythonGatewayServer extends SpringBootServletInitializer {
private DataSourceMapper dataSourceMapper;
@Autowired
- private PythonGatewayConfig pythonGatewayConfig;
+ private PythonGatewayConfiguration pythonGatewayConfiguration;
@Autowired
private ProjectUserMapper projectUserMapper;
@@ -546,30 +542,32 @@ public class PythonGatewayServer extends SpringBootServletInitializer {
}
@PostConstruct
- public void run() {
+ public void init() {
+ if (pythonGatewayConfiguration.getEnabled()) {
+ this.start();
+ }
+ }
+
+ private void start() {
GatewayServer server;
try {
- InetAddress gatewayHost = InetAddress.getByName(pythonGatewayConfig.getGatewayServerAddress());
- InetAddress pythonHost = InetAddress.getByName(pythonGatewayConfig.getPythonAddress());
+ InetAddress gatewayHost = InetAddress.getByName(pythonGatewayConfiguration.getGatewayServerAddress());
+ InetAddress pythonHost = InetAddress.getByName(pythonGatewayConfiguration.getPythonAddress());
server = new GatewayServer(
- this,
- pythonGatewayConfig.getGatewayServerPort(),
- pythonGatewayConfig.getPythonPort(),
- gatewayHost,
- pythonHost,
- pythonGatewayConfig.getConnectTimeout(),
- pythonGatewayConfig.getReadTimeout(),
- null
+ this,
+ pythonGatewayConfiguration.getGatewayServerPort(),
+ pythonGatewayConfiguration.getPythonPort(),
+ gatewayHost,
+ pythonHost,
+ pythonGatewayConfiguration.getConnectTimeout(),
+ pythonGatewayConfiguration.getReadTimeout(),
+ null
);
GatewayServer.turnLoggingOn();
- logger.info("PythonGatewayServer started on: " + gatewayHost.toString());
+ logger.info("PythonGatewayService started on: " + gatewayHost.toString());
server.start();
} catch (UnknownHostException e) {
- logger.error("exception occurred while constructing PythonGatewayServer().", e);
+ logger.error("exception occurred while constructing PythonGatewayService().", e);
}
}
-
- public static void main(String[] args) {
- SpringApplication.run(PythonGatewayServer.class, args);
- }
}
diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
index a14ac46124..4486824314 100644
--- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
+++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ExecutorServiceImpl.java
@@ -619,7 +619,7 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ
* @param runMode
* @return
*/
- private int createComplementCommandList(Date start, Date end, RunMode runMode, Command command,
+ protected int createComplementCommandList(Date start, Date end, RunMode runMode, Command command,
Integer expectedParallelismNumber, ComplementDependentMode complementDependentMode) {
int createCount = 0;
int dependentProcessDefinitionCreateCount = 0;
@@ -713,7 +713,7 @@ public class ExecutorServiceImpl extends BaseServiceImpl implements ExecutorServ
/**
* create complement dependent command
*/
- private int createComplementDependentCommand(List<Schedule> schedules, Command command) {
+ protected int createComplementDependentCommand(List<Schedule> schedules, Command command) {
int dependentProcessDefinitionCreateCount = 0;
Command dependentCommand;
diff --git a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessInstanceServiceImpl.java b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessInstanceServiceImpl.java
index 902b3a495c..c91bcfd217 100644
--- a/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessInstanceServiceImpl.java
+++ b/dolphinscheduler-api/src/main/java/org/apache/dolphinscheduler/api/service/impl/ProcessInstanceServiceImpl.java
@@ -214,7 +214,6 @@ public class ProcessInstanceServiceImpl extends BaseServiceImpl implements Proce
if (processDefinition == null || projectCode != processDefinition.getProjectCode()) {
putMsg(result, Status.PROCESS_DEFINE_NOT_EXIST, processId);
} else {
- processInstance.setWarningGroupId(processDefinition.getWarningGroupId());
processInstance.setLocations(processDefinition.getLocations());
processInstance.setDagData(processService.genDagData(processDefinition));
result.put(DATA_LIST, processInstance);
diff --git a/dolphinscheduler-api/src/main/resources/application.yaml b/dolphinscheduler-api/src/main/resources/application.yaml
index 3db21c3a04..23b93028af 100644
--- a/dolphinscheduler-api/src/main/resources/application.yaml
+++ b/dolphinscheduler-api/src/main/resources/application.yaml
@@ -108,6 +108,26 @@ audit:
metrics:
enabled: true
+python-gateway:
+ # Weather enable python gateway server or not. The default value is true.
+ enabled: true
+ # The address of Python gateway server start. Set its value to `0.0.0.0` if your Python API run in different
+ # between Python gateway server. It could be be specific to other address like `127.0.0.1` or `localhost`
+ gateway-server-address: 0.0.0.0
+ # The port of Python gateway server start. Define which port you could connect to Python gateway server from
+ # Python API side.
+ gateway-server-port: 25333
+ # The address of Python callback client.
+ python-address: 127.0.0.1
+ # The port of Python callback client.
+ python-port: 25334
+ # Close connection of socket server if no other request accept after x milliseconds. Define value is (0 = infinite),
+ # and socket server would never close even though no requests accept
+ connect-timeout: 0
+ # Close each active connection of socket server if python program not active after x milliseconds. Define value is
+ # (0 = infinite), and socket server would never close even though no requests accept
+ read-timeout: 0
+
# Override by profile
---
diff --git a/dolphinscheduler-api/src/main/resources/i18n/messages_en_US.properties b/dolphinscheduler-api/src/main/resources/i18n/messages_en_US.properties
index 7a4752f225..bdaba338d1 100644
--- a/dolphinscheduler-api/src/main/resources/i18n/messages_en_US.properties
+++ b/dolphinscheduler-api/src/main/resources/i18n/messages_en_US.properties
@@ -219,7 +219,7 @@ QUERY_ALL_DEFINITION_LIST_NOTES=query all definition list
PAGE_NO=page no
PROCESS_INSTANCE_ID=process instance id
PROCESS_INSTANCE_JSON=process instance info(json format)
-SCHEDULE_TIME=schedule time
+SCHEDULE_TIME=schedule time,empty string indicates the current day
SYNC_DEFINE=update the information of the process instance to the process definition
RECOVERY_PROCESS_INSTANCE_FLAG=whether to recovery process instance
PREVIEW_SCHEDULE_NOTES=preview schedule
diff --git a/dolphinscheduler-api/src/main/resources/i18n/messages_zh_CN.properties b/dolphinscheduler-api/src/main/resources/i18n/messages_zh_CN.properties
index bdcf2f5d37..748f2a662d 100644
--- a/dolphinscheduler-api/src/main/resources/i18n/messages_zh_CN.properties
+++ b/dolphinscheduler-api/src/main/resources/i18n/messages_zh_CN.properties
@@ -206,7 +206,7 @@ PROCESS_INSTANCE_ID=流程实例ID
PROCESS_INSTANCE_IDS=流程实例ID集合
PROCESS_INSTANCE_JSON=流程实例信息(json格式)
PREVIEW_SCHEDULE_NOTES=定时调度预览
-SCHEDULE_TIME=定时时间
+SCHEDULE_TIME=定时时间,空字符串表示当前天
SYNC_DEFINE=更新流程实例的信息是否同步到流程定义
RECOVERY_PROCESS_INSTANCE_FLAG=是否恢复流程实例
SEARCH_VAL=搜索值
diff --git a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ExecutorControllerTest.java b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ExecutorControllerTest.java
index 2d5c03a83a..7faed0bde0 100644
--- a/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ExecutorControllerTest.java
+++ b/dolphinscheduler-api/src/test/java/org/apache/dolphinscheduler/api/controller/ExecutorControllerTest.java
@@ -202,9 +202,10 @@ public class ExecutorControllerTest extends AbstractControllerTest {
paramsMap.add("processDefinitionCode", String.valueOf(processDefinitionCode));
paramsMap.add("failureStrategy", String.valueOf(failureStrategy));
paramsMap.add("warningType", String.valueOf(warningType));
+ paramsMap.add("scheduleTime", scheduleTime);
when(executorService.execProcessInstance(any(User.class), eq(projectCode), eq(processDefinitionCode),
- eq(null), eq(null), eq(failureStrategy), eq(null), eq(null), eq(warningType),
+ eq(scheduleTime), eq(null), eq(failureStrategy), eq(null), eq(null), eq(warningType),
eq(0), eq(null), eq(null), eq("default"), eq(-1L),
eq(Constants.MAX_TASK_TIMEOUT), eq(null), eq(null), eq(0),
eq(complementDependentMode))).thenReturn(executeServiceResult);
diff --git a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java
index 0572e8f9ec..4199627f71 100644
--- a/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java
+++ b/dolphinscheduler-common/src/main/java/org/apache/dolphinscheduler/common/Constants.java
@@ -327,6 +327,7 @@ public final class Constants {
public static final String NULL = "NULL";
public static final String THREAD_NAME_MASTER_SERVER = "Master-Server";
public static final String THREAD_NAME_WORKER_SERVER = "Worker-Server";
+ public static final String THREAD_NAME_ALERT_SERVER = "Alert-Server";
/**
* command parameter keys
diff --git a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/UdfFunc.java b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/UdfFunc.java
index 2a389b9bd5..949499d1e8 100644
--- a/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/UdfFunc.java
+++ b/dolphinscheduler-dao/src/main/java/org/apache/dolphinscheduler/dao/entity/UdfFunc.java
@@ -17,19 +17,18 @@
package org.apache.dolphinscheduler.dao.entity;
-import org.apache.dolphinscheduler.common.enums.UdfType;
-import org.apache.dolphinscheduler.common.utils.JSONUtils;
-
-import org.apache.commons.lang.StringUtils;
-
-import java.io.IOException;
-import java.util.Date;
-
import com.baomidou.mybatisplus.annotation.IdType;
+import com.baomidou.mybatisplus.annotation.TableField;
import com.baomidou.mybatisplus.annotation.TableId;
import com.baomidou.mybatisplus.annotation.TableName;
import com.fasterxml.jackson.databind.DeserializationContext;
import com.fasterxml.jackson.databind.KeyDeserializer;
+import org.apache.commons.lang.StringUtils;
+import org.apache.dolphinscheduler.common.enums.UdfType;
+import org.apache.dolphinscheduler.common.utils.JSONUtils;
+
+import java.io.IOException;
+import java.util.Date;
/**
* udf function
@@ -39,13 +38,23 @@ public class UdfFunc {
/**
* id
*/
- @TableId(value="id", type=IdType.AUTO)
+ @TableId(value = "id", type = IdType.AUTO)
private int id;
/**
* user id
*/
private int userId;
+ public String getResourceType() {
+ return resourceType;
+ }
+
+ public void setResourceType(String resourceType) {
+ this.resourceType = "UDF";
+ }
+
+ @TableField(exist = false)
+ private String resourceType = "UDF";
/**
* udf function name
*/
diff --git a/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/entity/UdfFuncTest.java b/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/entity/UdfFuncTest.java
index 5d0fbe86d4..4231ba35f6 100644
--- a/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/entity/UdfFuncTest.java
+++ b/dolphinscheduler-dao/src/test/java/org/apache/dolphinscheduler/dao/entity/UdfFuncTest.java
@@ -18,10 +18,11 @@
package org.apache.dolphinscheduler.dao.entity;
import org.apache.dolphinscheduler.dao.entity.UdfFunc.UdfFuncDeserializer;
-import java.io.IOException;
import org.junit.Assert;
import org.junit.Test;
+import java.io.IOException;
+
public class UdfFuncTest {
/**
@@ -35,9 +36,9 @@ public class UdfFuncTest {
udfFunc.setResourceId(2);
udfFunc.setClassName("org.apache.dolphinscheduler.test.mrUpdate");
- Assert.assertEquals("{\"id\":0,\"userId\":0,\"funcName\":null,\"className\":\"org.apache.dolphinscheduler.test.mrUpdate\",\"argTypes\":null,\"database\":null,"
- + "\"description\":null,\"resourceId\":2,\"resourceName\":\"dolphin_resource_update\",\"type\":null,\"createTime\":null,\"updateTime\":null}"
- , udfFunc.toString());
+ Assert.assertEquals("{\"id\":0,\"userId\":0,\"resourceType\":\"UDF\",\"funcName\":null,\"className\":\"org.apache.dolphinscheduler.test.mrUpdate\",\"argTypes\":null,\"database\":null,"
+ + "\"description\":null,\"resourceId\":2,\"resourceName\":\"dolphin_resource_update\",\"type\":null,\"createTime\":null,\"updateTime\":null}"
+ , udfFunc.toString());
}
/**
diff --git a/dolphinscheduler-dist/src/main/assembly/dolphinscheduler-bin.xml b/dolphinscheduler-dist/src/main/assembly/dolphinscheduler-bin.xml
index 3b113741c1..1049d2720a 100644
--- a/dolphinscheduler-dist/src/main/assembly/dolphinscheduler-bin.xml
+++ b/dolphinscheduler-dist/src/main/assembly/dolphinscheduler-bin.xml
@@ -50,11 +50,6 @@
<outputDirectory>logger-server</outputDirectory>
</fileSet>
- <fileSet>
- <directory>${basedir}/../dolphinscheduler-python/target/python-gateway-server</directory>
- <outputDirectory>python-gateway-server</outputDirectory>
- </fileSet>
-
<fileSet>
<directory>${basedir}/../dolphinscheduler-standalone-server/target/standalone-server</directory>
<outputDirectory>standalone-server</outputDirectory>
diff --git a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/registry/MasterRegistryClient.java b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/registry/MasterRegistryClient.java
index abf1734af4..e567b77464 100644
--- a/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/registry/MasterRegistryClient.java
+++ b/dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/registry/MasterRegistryClient.java
@@ -112,10 +112,7 @@ public class MasterRegistryClient {
}
public void start() {
- String nodeLock = Constants.REGISTRY_DOLPHINSCHEDULER_LOCK_FAILOVER_STARTUP_MASTERS;
try {
- // create distributed lock with the root node path of the lock space as /dolphinscheduler/lock/failover/startup-masters
- registryClient.getLock(nodeLock);
// master registry
registry();
@@ -123,8 +120,6 @@ public class MasterRegistryClient {
} catch (Exception e) {
logger.error("master start up exception", e);
throw new RuntimeException("master start up error", e);
- } finally {
- registryClient.releaseLock(nodeLock);
}
}
diff --git a/dolphinscheduler-python/pom.xml b/dolphinscheduler-python/pom.xml
index 00aaaf8f38..4a66459cae 100644
--- a/dolphinscheduler-python/pom.xml
+++ b/dolphinscheduler-python/pom.xml
@@ -28,82 +28,7 @@
<name>${project.artifactId}</name>
<packaging>jar</packaging>
- <dependencies>
- <!-- dolphinscheduler -->
- <dependency>
- <groupId>org.apache.dolphinscheduler</groupId>
- <artifactId>dolphinscheduler-api</artifactId>
- </dependency>
-
- <!--springboot-->
- <dependency>
- <groupId>org.springframework.boot</groupId>
- <artifactId>spring-boot-starter-web</artifactId>
- <exclusions>
- <exclusion>
- <groupId>org.springframework.boot</groupId>
- <artifactId>spring-boot-starter-tomcat</artifactId>
- </exclusion>
- <exclusion>
- <artifactId>log4j-to-slf4j</artifactId>
- <groupId>org.apache.logging.log4j</groupId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>net.sf.py4j</groupId>
- <artifactId>py4j</artifactId>
- </dependency>
-
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-jar-plugin</artifactId>
- <configuration>
- <excludes>
- <exclude>*.yaml</exclude>
- <exclude>*.xml</exclude>
- </excludes>
- </configuration>
- </plugin>
- <plugin>
- <artifactId>maven-assembly-plugin</artifactId>
- <executions>
- <execution>
- <id>dolphinscheduler-python-gateway-server</id>
- <phase>package</phase>
- <goals>
- <goal>single</goal>
- </goals>
- <configuration>
- <finalName>python-gateway-server</finalName>
- <descriptors>
- <descriptor>src/main/assembly/dolphinscheduler-python-gateway-server.xml</descriptor>
- </descriptors>
- <appendAssemblyId>false</appendAssemblyId>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
-
<profiles>
- <profile>
- <id>docker</id>
- <build>
- <plugins>
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>exec-maven-plugin</artifactId>
- </plugin>
- </plugins>
- </build>
- </profile>
<profile>
<id>python</id>
<build>
diff --git a/dolphinscheduler-python/pydolphinscheduler/UPDATING.md b/dolphinscheduler-python/pydolphinscheduler/UPDATING.md
index cf45c097fb..430e4b626f 100644
--- a/dolphinscheduler-python/pydolphinscheduler/UPDATING.md
+++ b/dolphinscheduler-python/pydolphinscheduler/UPDATING.md
@@ -24,6 +24,9 @@ It started after version 2.0.5 released
## dev
+* Integrate Python gateway server into Dolphinscheduler API server, and you could start Python gateway service by command
+ `./bin/dolphinscheduler-daemon.sh start api-server` instead of independent command
+ `./bin/dolphinscheduler-daemon.sh start python-gateway-server`.
* Remove parameter `queue` from class `ProcessDefinition` to avoid confuse user when it change but not work
* Change `yaml_parser.py` method `to_string` to magic method `__str__` make it more pythonic.
* Use package ``ruamel.yaml`` replace ``pyyaml`` for write yaml file with comment.
diff --git a/dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst b/dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst
index 0af90d5494..489f4e8eb3 100644
--- a/dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst
+++ b/dolphinscheduler-python/pydolphinscheduler/docs/source/start.rst
@@ -55,7 +55,7 @@ After Python is already installed on your machine following section
$ pip install apache-dolphinscheduler
The latest version of *PyDolphinScheduler* would be installed after you run above
-command in your terminal. You could go and `start Python Gateway Server`_ to finish
+command in your terminal. You could go and `start Python Gateway Service`_ to finish
the prepare, and then go to :doc:`tutorial` to make your hand dirty. But if you
want to install the unreleased version of *PyDolphinScheduler*, you could go and see
section `installing PyDolphinScheduler in dev`_ for more detail.
@@ -74,33 +74,39 @@ which we hold in GitHub
# Install PyDolphinScheduler in develop mode
$ cd dolphinscheduler-python/pydolphinscheduler && pip install -e .
-After you installed *PyDolphinScheduler*, please remember `start Python Gateway Server`_
+After you installed *PyDolphinScheduler*, please remember `start Python Gateway Service`_
which waiting for *PyDolphinScheduler*'s workflow definition require.
-Start Python Gateway Server
----------------------------
+Start Python Gateway Service
+----------------------------
Since **PyDolphinScheduler** is Python API for `Apache DolphinScheduler`_, it
could define workflow and tasks structure, but could not run it unless you
-`install Apache DolphinScheduler`_ and start Python gateway server. We only
-and some key steps here and you could go `install Apache DolphinScheduler`_
-for more detail
+`install Apache DolphinScheduler`_ and start its API server which including
+Python gateway service in it. We only and some key steps here and you could
+go `install Apache DolphinScheduler`_ for more detail
.. code-block:: bash
- # Start pythonGatewayServer
- $ ./bin/dolphinscheduler-daemon.sh start pythonGatewayServer
+ # Start DolphinScheduler api-server which including python gateway service
+ $ ./bin/dolphinscheduler-daemon.sh start api-server
To check whether the server is alive or not, you could run :code:`jps`. And
-the server is health if keyword `PythonGatewayServer` in the console.
+the server is health if keyword `ApiApplicationServer` in the console.
.. code-block:: bash
$ jps
....
- 201472 PythonGatewayServer
+ 201472 ApiApplicationServer
....
+.. note::
+
+ Please make sure you already enabled started Python gateway service along with `api-server`. The configuration is in
+ yaml config path `python-gateway.enabled : true` in api-server's configuration path in `api-server/conf/application.yaml`.
+ The default value is true and Python gateway service start when api server is been started.
+
What's More
-----------
diff --git a/dolphinscheduler-python/pydolphinscheduler/docs/source/tutorial.rst b/dolphinscheduler-python/pydolphinscheduler/docs/source/tutorial.rst
index 83c5746aba..e0f22fb816 100644
--- a/dolphinscheduler-python/pydolphinscheduler/docs/source/tutorial.rst
+++ b/dolphinscheduler-python/pydolphinscheduler/docs/source/tutorial.rst
@@ -130,7 +130,7 @@ Now, we could run the Python code like other Python script, for the basic usage
:end-before: [end submit_or_run]
If you not start your Apache DolphinScheduler server, you could find the way in
-:ref:`start:start Python gateway server` and it would have more detail about related server
+:ref:`start:start Python gateway service` and it would have more detail about related server
start. Beside attribute `run`, we have attribute `submit` for object `ProcessDefinition`
and it just submit workflow to the daemon but not setting the schedule information. For
more detail you could see :ref:`concept:process definition`.
diff --git a/dolphinscheduler-python/pydolphinscheduler/tests/integration/__init__.py b/dolphinscheduler-python/pydolphinscheduler/tests/integration/__init__.py
index 8079bfba1f..65625a9f04 100644
--- a/dolphinscheduler-python/pydolphinscheduler/tests/integration/__init__.py
+++ b/dolphinscheduler-python/pydolphinscheduler/tests/integration/__init__.py
@@ -15,4 +15,4 @@
# specific language governing permissions and limitations
# under the License.
-"""Test integration between Python API and PythonGatewayServer."""
+"""Test integration between Python API and PythonGatewayService."""
diff --git a/dolphinscheduler-python/pydolphinscheduler/tests/integration/test_submit_examples.py b/dolphinscheduler-python/pydolphinscheduler/tests/integration/test_submit_examples.py
index 0964e1b975..85e5e23e31 100644
--- a/dolphinscheduler-python/pydolphinscheduler/tests/integration/test_submit_examples.py
+++ b/dolphinscheduler-python/pydolphinscheduler/tests/integration/test_submit_examples.py
@@ -15,7 +15,7 @@
# specific language governing permissions and limitations
# under the License.
-"""Test whether success submit examples DAG to PythonGatewayServer."""
+"""Test whether success submit examples DAG to PythonGatewayService."""
from pathlib import Path
@@ -34,7 +34,7 @@ from tests.testing.path import path_example
],
)
def test_exec_white_list_example(example_path: Path):
- """Test execute examples and submit DAG to PythonGatewayServer."""
+ """Test execute examples and submit DAG to PythonGatewayService."""
try:
exec(example_path.read_text())
except Exception:
diff --git a/dolphinscheduler-python/src/main/assembly/dolphinscheduler-python-gateway-server.xml b/dolphinscheduler-python/src/main/assembly/dolphinscheduler-python-gateway-server.xml
deleted file mode 100644
index 49e03206ea..0000000000
--- a/dolphinscheduler-python/src/main/assembly/dolphinscheduler-python-gateway-server.xml
+++ /dev/null
@@ -1,64 +0,0 @@
-<!--
- ~ Licensed to the Apache Software Foundation (ASF) under one or more
- ~ contributor license agreements. See the NOTICE file distributed with
- ~ this work for additional information regarding copyright ownership.
- ~ The ASF licenses this file to You under the Apache License, Version 2.0
- ~ (the "License"); you may not use this file except in compliance with
- ~ the License. You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing, software
- ~ distributed under the License is distributed on an "AS IS" BASIS,
- ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- ~ See the License for the specific language governing permissions and
- ~ limitations under the License.
- -->
-
-<assembly xmlns="http://maven.apache.org/ASSEMBLY/2.1.0"
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
- xsi:schemaLocation="http://maven.apache.org/ASSEMBLY/2.1.0 http://maven.apache.org/xsd/assembly-2.1.0.xsd">
- <id>dolphinscheduler-python-gateway-server</id>
- <formats>
- <format>dir</format>
- </formats>
- <includeBaseDirectory>false</includeBaseDirectory>
- <baseDirectory>python-gateway-server</baseDirectory>
- <fileSets>
- <fileSet>
- <directory>${basedir}/src/main/resources</directory>
- <includes>
- <include>*.yaml</include>
- <include>*.xml</include>
- </includes>
- <outputDirectory>conf</outputDirectory>
- </fileSet>
- <fileSet>
- <directory>${basedir}/src/main/bin</directory>
- <outputDirectory>bin</outputDirectory>
- <fileMode>0755</fileMode>
- <directoryMode>0755</directoryMode>
- </fileSet>
- <fileSet>
- <directory>${basedir}/../script/env</directory>
- <outputDirectory>bin</outputDirectory>
- <includes>
- <include>dolphinscheduler_env.sh</include>
- </includes>
- <fileMode>0755</fileMode>
- <directoryMode>0755</directoryMode>
- </fileSet>
- <fileSet>
- <directory>${basedir}/../dolphinscheduler-common/src/main/resources</directory>
- <includes>
- <include>**/*.properties</include>
- </includes>
- <outputDirectory>conf</outputDirectory>
- </fileSet>
- </fileSets>
- <dependencySets>
- <dependencySet>
- <outputDirectory>libs</outputDirectory>
- </dependencySet>
- </dependencySets>
-</assembly>
diff --git a/dolphinscheduler-python/src/main/bin/start.sh b/dolphinscheduler-python/src/main/bin/start.sh
deleted file mode 100644
index d588a2b2d9..0000000000
--- a/dolphinscheduler-python/src/main/bin/start.sh
+++ /dev/null
@@ -1,32 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-BIN_DIR=$(dirname $0)
-DOLPHINSCHEDULER_HOME=${DOLPHINSCHEDULER_HOME:-$(cd $BIN_DIR/..; pwd)}
-
-source "$BIN_DIR/dolphinscheduler_env.sh"
-
-JAVA_OPTS=${JAVA_OPTS:-"-server -Duser.timezone=${SPRING_JACKSON_TIME_ZONE} -Xms1g -Xmx1g -Xmn512m -XX:+PrintGCDetails -Xloggc:gc.log -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=dump.hprof"}
-
-if [[ "$DOCKER" == "true" ]]; then
- JAVA_OPTS="${JAVA_OPTS} -XX:-UseContainerSupport"
-fi
-
-java $JAVA_OPTS \
- -cp "$DOLPHINSCHEDULER_HOME/conf":"$DOLPHINSCHEDULER_HOME/libs/*" \
- org.apache.dolphinscheduler.server.PythonGatewayServer
diff --git a/dolphinscheduler-python/src/main/docker/Dockerfile b/dolphinscheduler-python/src/main/docker/Dockerfile
deleted file mode 100644
index 7f50a8fc39..0000000000
--- a/dolphinscheduler-python/src/main/docker/Dockerfile
+++ /dev/null
@@ -1,34 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-FROM openjdk:8-jre-slim-buster
-
-ENV DOCKER true
-ENV TZ Asia/Shanghai
-ENV DOLPHINSCHEDULER_HOME /opt/dolphinscheduler
-
-RUN apt update ; \
- apt install -y curl sudo ; \
- rm -rf /var/lib/apt/lists/*
-
-WORKDIR $DOLPHINSCHEDULER_HOME
-
-ADD ./target/python-gateway-server $DOLPHINSCHEDULER_HOME
-
-EXPOSE 25333 54321
-
-CMD [ "/bin/bash", "./bin/start.sh" ]
diff --git a/dolphinscheduler-python/src/main/resources/application.yaml b/dolphinscheduler-python/src/main/resources/application.yaml
deleted file mode 100644
index b899c03638..0000000000
--- a/dolphinscheduler-python/src/main/resources/application.yaml
+++ /dev/null
@@ -1,83 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-spring:
- application:
- name: python-gateway-server
- main:
- banner-mode: off
- datasource:
- driver-class-name: org.h2.Driver
- url: jdbc:h2:mem:dolphinscheduler;MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=true
- username: sa
- password: ""
- jackson:
- time-zone: UTC
- date-format: "yyyy-MM-dd HH:mm:ss"
- servlet:
- multipart:
- max-file-size: 1024MB
- max-request-size: 1024MB
- messages:
- basename: i18n/messages
- jpa:
- hibernate:
- ddl-auto: none
-
-python-gateway:
- # The address of Python gateway server start. Set its value to `0.0.0.0` if your Python API run in different
- # between Python gateway server. It could be be specific to other address like `127.0.0.1` or `localhost`
- gateway-server-address: 0.0.0.0
- # The port of Python gateway server start. Define which port you could connect to Python gateway server from
- # Python API side.
- gateway-server-port: 25333
- # The address of Python callback client.
- python-address: 127.0.0.1
- # The port of Python callback client.
- python-port: 25334
- # Close connection of socket server if no other request accept after x milliseconds. Define value is (0 = infinite),
- # and socket server would never close even though no requests accept
- connect-timeout: 0
- # Close each active connection of socket server if python program not active after x milliseconds. Define value is
- # (0 = infinite), and socket server would never close even though no requests accept
- read-timeout: 0
-
-server:
- port: 54321
-
-management:
- endpoints:
- web:
- exposure:
- include: '*'
- metrics:
- tags:
- application: ${spring.application.name}
-
-metrics:
- enabled: true
-
-# Override by profile
----
-spring:
- config:
- activate:
- on-profile: postgresql
- quartz:
- properties:
- org.quartz.jobStore.driverDelegateClass: org.quartz.impl.jdbcjobstore.PostgreSQLDelegate
-
diff --git a/dolphinscheduler-python/src/main/resources/logback-spring.xml b/dolphinscheduler-python/src/main/resources/logback-spring.xml
deleted file mode 100644
index 4f06d160d6..0000000000
--- a/dolphinscheduler-python/src/main/resources/logback-spring.xml
+++ /dev/null
@@ -1,57 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
- ~ Licensed to the Apache Software Foundation (ASF) under one or more
- ~ contributor license agreements. See the NOTICE file distributed with
- ~ this work for additional information regarding copyright ownership.
- ~ The ASF licenses this file to You under the Apache License, Version 2.0
- ~ (the "License"); you may not use this file except in compliance with
- ~ the License. You may obtain a copy of the License at
- ~
- ~ http://www.apache.org/licenses/LICENSE-2.0
- ~
- ~ Unless required by applicable law or agreed to in writing, software
- ~ distributed under the License is distributed on an "AS IS" BASIS,
- ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- ~ See the License for the specific language governing permissions and
- ~ limitations under the License.
- -->
-
-<configuration scan="true" scanPeriod="120 seconds">
-<property name="log.base" value="logs"/>
-
-<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
- <encoder>
- <pattern>
- [%level] %date{yyyy-MM-dd HH:mm:ss.SSS} %logger{96}:[%line] - %msg%n
- </pattern>
- <charset>UTF-8</charset>
- </encoder>
-</appender>
-
-<appender name="PYTHONGATEWAYLOGFILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
- <file>${log.base}/dolphinscheduler-python-gateway.log</file>
- <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
- <level>INFO</level>
- </filter>
- <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
- <fileNamePattern>${log.base}/dolphinscheduler-python-gateway.%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
- <maxHistory>168</maxHistory>
- <maxFileSize>64MB</maxFileSize>
- </rollingPolicy>
- <encoder>
- <pattern>
- [%level] %date{yyyy-MM-dd HH:mm:ss.SSS} %logger{96}:[%line] - %msg%n
- </pattern>
- <charset>UTF-8</charset>
- </encoder>
-</appender>
-
-<root level="INFO">
- <if condition="${DOCKER:-false}">
- <then>
- <appender-ref ref="STDOUT"/>
- </then>
- </if>
- <appender-ref ref="PYTHONGATEWAYLOGFILE"/>
-</root>
-</configuration>
diff --git a/dolphinscheduler-remote/src/main/java/org/apache/dolphinscheduler/remote/command/alert/AlertSendRequestCommand.java b/dolphinscheduler-remote/src/main/java/org/apache/dolphinscheduler/remote/command/alert/AlertSendRequestCommand.java
index ba37e22d2a..83c81c9fbe 100644
--- a/dolphinscheduler-remote/src/main/java/org/apache/dolphinscheduler/remote/command/alert/AlertSendRequestCommand.java
+++ b/dolphinscheduler-remote/src/main/java/org/apache/dolphinscheduler/remote/command/alert/AlertSendRequestCommand.java
@@ -88,4 +88,14 @@ public class AlertSendRequestCommand implements Serializable {
command.setBody(body);
return command;
}
+
+ @Override
+ public String toString() {
+ return "AlertSendRequestCommand{" +
+ "groupId=" + groupId +
+ ", title='" + title + '\'' +
+ ", content='" + content + '\'' +
+ ", warnType=" + warnType +
+ '}';
+ }
}
diff --git a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
index 2255c0811b..395aa31668 100644
--- a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
+++ b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
@@ -17,38 +17,11 @@
package org.apache.dolphinscheduler.service.process;
-import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_END_DATE;
-import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_START_DATE;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_EMPTY_SUB_PROCESS;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_FATHER_PARAMS;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_RECOVER_PROCESS_ID_STRING;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_SUB_PROCESS;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_SUB_PROCESS_DEFINE_CODE;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_SUB_PROCESS_PARENT_INSTANCE_ID;
-import static org.apache.dolphinscheduler.common.Constants.LOCAL_PARAMS;
-import static org.apache.dolphinscheduler.plugin.task.api.utils.DataQualityConstants.TASK_INSTANCE_ID;
-
-import static java.util.stream.Collectors.toSet;
-
-import org.apache.dolphinscheduler.common.Constants;
import org.apache.dolphinscheduler.common.enums.AuthorizationType;
-import org.apache.dolphinscheduler.common.enums.CommandType;
-import org.apache.dolphinscheduler.common.enums.FailureStrategy;
-import org.apache.dolphinscheduler.common.enums.Flag;
-import org.apache.dolphinscheduler.common.enums.ReleaseState;
-import org.apache.dolphinscheduler.common.enums.TaskDependType;
import org.apache.dolphinscheduler.common.enums.TaskGroupQueueStatus;
-import org.apache.dolphinscheduler.common.enums.TimeoutFlag;
-import org.apache.dolphinscheduler.common.enums.WarningType;
import org.apache.dolphinscheduler.common.graph.DAG;
import org.apache.dolphinscheduler.common.model.TaskNode;
import org.apache.dolphinscheduler.common.model.TaskNodeRelation;
-import org.apache.dolphinscheduler.common.process.ProcessDag;
-import org.apache.dolphinscheduler.common.utils.CodeGenerateUtils;
-import org.apache.dolphinscheduler.common.utils.CodeGenerateUtils.CodeGenerateException;
-import org.apache.dolphinscheduler.common.utils.DateUtils;
-import org.apache.dolphinscheduler.common.utils.JSONUtils;
-import org.apache.dolphinscheduler.common.utils.ParameterUtils;
import org.apache.dolphinscheduler.dao.entity.Command;
import org.apache.dolphinscheduler.dao.entity.DagData;
import org.apache.dolphinscheduler.dao.entity.DataSource;
@@ -58,9 +31,7 @@ import org.apache.dolphinscheduler.dao.entity.DqExecuteResult;
import org.apache.dolphinscheduler.dao.entity.DqRule;
import org.apache.dolphinscheduler.dao.entity.DqRuleExecuteSql;
import org.apache.dolphinscheduler.dao.entity.DqRuleInputEntry;
-import org.apache.dolphinscheduler.dao.entity.DqTaskStatisticsValue;
import org.apache.dolphinscheduler.dao.entity.Environment;
-import org.apache.dolphinscheduler.dao.entity.ErrorCommand;
import org.apache.dolphinscheduler.dao.entity.ProcessDefinition;
import org.apache.dolphinscheduler.dao.entity.ProcessDefinitionLog;
import org.apache.dolphinscheduler.dao.entity.ProcessInstance;
@@ -73,2810 +44,256 @@ import org.apache.dolphinscheduler.dao.entity.Resource;
import org.apache.dolphinscheduler.dao.entity.Schedule;
import org.apache.dolphinscheduler.dao.entity.TaskDefinition;
import org.apache.dolphinscheduler.dao.entity.TaskDefinitionLog;
-import org.apache.dolphinscheduler.dao.entity.TaskGroup;
import org.apache.dolphinscheduler.dao.entity.TaskGroupQueue;
import org.apache.dolphinscheduler.dao.entity.TaskInstance;
import org.apache.dolphinscheduler.dao.entity.Tenant;
import org.apache.dolphinscheduler.dao.entity.UdfFunc;
import org.apache.dolphinscheduler.dao.entity.User;
-import org.apache.dolphinscheduler.dao.mapper.CommandMapper;
-import org.apache.dolphinscheduler.dao.mapper.DataSourceMapper;
-import org.apache.dolphinscheduler.dao.mapper.DqComparisonTypeMapper;
-import org.apache.dolphinscheduler.dao.mapper.DqExecuteResultMapper;
-import org.apache.dolphinscheduler.dao.mapper.DqRuleExecuteSqlMapper;
-import org.apache.dolphinscheduler.dao.mapper.DqRuleInputEntryMapper;
-import org.apache.dolphinscheduler.dao.mapper.DqRuleMapper;
-import org.apache.dolphinscheduler.dao.mapper.DqTaskStatisticsValueMapper;
-import org.apache.dolphinscheduler.dao.mapper.EnvironmentMapper;
-import org.apache.dolphinscheduler.dao.mapper.ErrorCommandMapper;
-import org.apache.dolphinscheduler.dao.mapper.ProcessDefinitionLogMapper;
-import org.apache.dolphinscheduler.dao.mapper.ProcessDefinitionMapper;
-import org.apache.dolphinscheduler.dao.mapper.ProcessInstanceMapMapper;
-import org.apache.dolphinscheduler.dao.mapper.ProcessInstanceMapper;
-import org.apache.dolphinscheduler.dao.mapper.ProcessTaskRelationLogMapper;
-import org.apache.dolphinscheduler.dao.mapper.ProcessTaskRelationMapper;
-import org.apache.dolphinscheduler.dao.mapper.ProjectMapper;
-import org.apache.dolphinscheduler.dao.mapper.ResourceMapper;
-import org.apache.dolphinscheduler.dao.mapper.ResourceUserMapper;
-import org.apache.dolphinscheduler.dao.mapper.ScheduleMapper;
-import org.apache.dolphinscheduler.dao.mapper.TaskDefinitionLogMapper;
-import org.apache.dolphinscheduler.dao.mapper.TaskDefinitionMapper;
-import org.apache.dolphinscheduler.dao.mapper.TaskGroupMapper;
-import org.apache.dolphinscheduler.dao.mapper.TaskGroupQueueMapper;
-import org.apache.dolphinscheduler.dao.mapper.TaskInstanceMapper;
-import org.apache.dolphinscheduler.dao.mapper.TenantMapper;
-import org.apache.dolphinscheduler.dao.mapper.UdfFuncMapper;
-import org.apache.dolphinscheduler.dao.mapper.UserMapper;
-import org.apache.dolphinscheduler.dao.mapper.WorkFlowLineageMapper;
-import org.apache.dolphinscheduler.dao.utils.DagHelper;
-import org.apache.dolphinscheduler.dao.utils.DqRuleUtils;
-import org.apache.dolphinscheduler.plugin.task.api.enums.Direct;
import org.apache.dolphinscheduler.plugin.task.api.enums.ExecutionStatus;
-import org.apache.dolphinscheduler.plugin.task.api.enums.dp.DqTaskState;
import org.apache.dolphinscheduler.plugin.task.api.model.DateInterval;
-import org.apache.dolphinscheduler.plugin.task.api.model.Property;
-import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
-import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
-import org.apache.dolphinscheduler.plugin.task.api.parameters.ParametersNode;
-import org.apache.dolphinscheduler.plugin.task.api.parameters.SubProcessParameters;
-import org.apache.dolphinscheduler.plugin.task.api.parameters.TaskTimeoutParameter;
-import org.apache.dolphinscheduler.remote.command.StateEventChangeCommand;
-import org.apache.dolphinscheduler.remote.command.TaskEventChangeCommand;
-import org.apache.dolphinscheduler.remote.processor.StateEventCallbackService;
-import org.apache.dolphinscheduler.remote.utils.Host;
-import org.apache.dolphinscheduler.service.bean.SpringApplicationContext;
-import org.apache.dolphinscheduler.service.exceptions.ServiceException;
-import org.apache.dolphinscheduler.service.log.LogClientService;
-import org.apache.dolphinscheduler.service.quartz.cron.CronUtils;
-import org.apache.dolphinscheduler.service.task.TaskPluginManager;
import org.apache.dolphinscheduler.spi.enums.ResourceType;
+import org.slf4j.Logger;
+import org.springframework.transaction.annotation.Transactional;
-import org.apache.commons.collections.CollectionUtils;
-import org.apache.commons.lang.StringUtils;
-import org.apache.commons.lang.math.NumberUtils;
-
-import java.util.ArrayList;
-import java.util.Arrays;
import java.util.Date;
-import java.util.EnumMap;
-import java.util.HashMap;
-import java.util.HashSet;
import java.util.List;
import java.util.Map;
-import java.util.Map.Entry;
-import java.util.Objects;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Component;
-import org.springframework.transaction.annotation.Transactional;
+public interface ProcessService {
+ @Transactional
+ ProcessInstance handleCommand(Logger logger, String host, Command command);
-import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
-import com.fasterxml.jackson.core.type.TypeReference;
-import com.fasterxml.jackson.databind.node.ObjectNode;
-import com.google.common.collect.Lists;
+ void moveToErrorCommand(Command command, String message);
-/**
- * process relative dao that some mappers in this.
- */
-@Component
-public class ProcessService {
+ int createCommand(Command command);
- private final Logger logger = LoggerFactory.getLogger(getClass());
+ List<Command> findCommandPage(int pageSize, int pageNumber);
- private final int[] stateArray = new int[]{ExecutionStatus.SUBMITTED_SUCCESS.ordinal(),
- ExecutionStatus.DISPATCH.ordinal(),
- ExecutionStatus.RUNNING_EXECUTION.ordinal(),
- ExecutionStatus.DELAY_EXECUTION.ordinal(),
- ExecutionStatus.READY_PAUSE.ordinal(),
- ExecutionStatus.READY_STOP.ordinal()};
+ List<Command> findCommandPageBySlot(int pageSize, int pageNumber, int masterCount, int thisMasterSlot);
- @Autowired
- private UserMapper userMapper;
+ boolean verifyIsNeedCreateCommand(Command command);
- @Autowired
- private ProcessDefinitionMapper processDefineMapper;
+ ProcessInstance findProcessInstanceDetailById(int processId);
- @Autowired
- private ProcessDefinitionLogMapper processDefineLogMapper;
+ List<TaskDefinition> getTaskNodeListByDefinition(long defineCode);
- @Autowired
- private ProcessInstanceMapper processInstanceMapper;
+ ProcessInstance findProcessInstanceById(int processId);
- @Autowired
- private DataSourceMapper dataSourceMapper;
+ ProcessDefinition findProcessDefineById(int processDefinitionId);
- @Autowired
- private ProcessInstanceMapMapper processInstanceMapMapper;
+ ProcessDefinition findProcessDefinition(Long processDefinitionCode, int version);
- @Autowired
- private TaskInstanceMapper taskInstanceMapper;
+ ProcessDefinition findProcessDefinitionByCode(Long processDefinitionCode);
- @Autowired
- private CommandMapper commandMapper;
+ int deleteWorkProcessInstanceById(int processInstanceId);
- @Autowired
- private ScheduleMapper scheduleMapper;
+ int deleteAllSubWorkProcessByParentId(int processInstanceId);
- @Autowired
- private UdfFuncMapper udfFuncMapper;
+ void removeTaskLogFile(Integer processInstanceId);
- @Autowired
- private ResourceMapper resourceMapper;
+ void deleteWorkTaskInstanceByProcessInstanceId(int processInstanceId);
- @Autowired
- private ResourceUserMapper resourceUserMapper;
+ void recurseFindSubProcess(long parentCode, List<Long> ids);
- @Autowired
- private ErrorCommandMapper errorCommandMapper;
+ void createRecoveryWaitingThreadCommand(Command originCommand, ProcessInstance processInstance);
- @Autowired
- private TenantMapper tenantMapper;
+ Tenant getTenantForProcess(int tenantId, int userId);
- @Autowired
- private ProjectMapper projectMapper;
+ Environment findEnvironmentByCode(Long environmentCode);
- @Autowired
- private DqExecuteResultMapper dqExecuteResultMapper;
+ void setSubProcessParam(ProcessInstance subProcessInstance);
- @Autowired
- private DqRuleMapper dqRuleMapper;
+ TaskInstance submitTaskWithRetry(ProcessInstance processInstance, TaskInstance taskInstance, int commitRetryTimes, int commitInterval);
- @Autowired
- private DqRuleInputEntryMapper dqRuleInputEntryMapper;
+ @Transactional(rollbackFor = Exception.class)
+ TaskInstance submitTask(ProcessInstance processInstance, TaskInstance taskInstance);
- @Autowired
- private DqRuleExecuteSqlMapper dqRuleExecuteSqlMapper;
+ void createSubWorkProcess(ProcessInstance parentProcessInstance, TaskInstance task);
- @Autowired
- private DqComparisonTypeMapper dqComparisonTypeMapper;
+ Map<String, String> getGlobalParamMap(String globalParams);
- @Autowired
- private DqTaskStatisticsValueMapper dqTaskStatisticsValueMapper;
+ Command createSubProcessCommand(ProcessInstance parentProcessInstance,
+ ProcessInstance childInstance,
+ ProcessInstanceMap instanceMap,
+ TaskInstance task);
- @Autowired
- private TaskDefinitionMapper taskDefinitionMapper;
+ TaskInstance submitTaskInstanceToDB(TaskInstance taskInstance, ProcessInstance processInstance);
- @Autowired
- private TaskDefinitionLogMapper taskDefinitionLogMapper;
+ ExecutionStatus getSubmitTaskState(TaskInstance taskInstance, ProcessInstance processInstance);
- @Autowired
- private ProcessTaskRelationMapper processTaskRelationMapper;
+ void saveProcessInstance(ProcessInstance processInstance);
- @Autowired
- private ProcessTaskRelationLogMapper processTaskRelationLogMapper;
+ int saveCommand(Command command);
+ boolean saveTaskInstance(TaskInstance taskInstance);
- @Autowired
- StateEventCallbackService stateEventCallbackService;
+ boolean createTaskInstance(TaskInstance taskInstance);
- @Autowired
- private EnvironmentMapper environmentMapper;
+ boolean updateTaskInstance(TaskInstance taskInstance);
- @Autowired
- private TaskGroupQueueMapper taskGroupQueueMapper;
+ TaskInstance findTaskInstanceById(Integer taskId);
- @Autowired
- private TaskGroupMapper taskGroupMapper;
+ List<TaskInstance> findTaskInstanceByIdList(List<Integer> idList);
- @Autowired
- private WorkFlowLineageMapper workFlowLineageMapper;
+ void packageTaskInstance(TaskInstance taskInstance, ProcessInstance processInstance);
- @Autowired
- private TaskPluginManager taskPluginManager;
+ void updateTaskDefinitionResources(TaskDefinition taskDefinition);
+
+ List<Integer> findTaskIdByInstanceState(int instanceId, ExecutionStatus state);
+
+ List<TaskInstance> findValidTaskListByProcessId(Integer processInstanceId);
+
+ List<TaskInstance> findPreviousTaskListByWorkProcessId(Integer processInstanceId);
+
+ int updateWorkProcessInstanceMap(ProcessInstanceMap processInstanceMap);
+
+ int createWorkProcessInstanceMap(ProcessInstanceMap processInstanceMap);
+
+ ProcessInstanceMap findWorkProcessMapByParent(Integer parentWorkProcessId, Integer parentTaskId);
+
+ int deleteWorkProcessMapByParentId(int parentWorkProcessId);
+
+ ProcessInstance findSubProcessInstance(Integer parentProcessId, Integer parentTaskId);
+
+ ProcessInstance findParentProcessInstance(Integer subProcessId);
+
+ int updateProcessInstance(ProcessInstance processInstance);
+
+ void changeOutParam(TaskInstance taskInstance);
+
+ List<String> convertIntListToString(List<Integer> intList);
+
+ Schedule querySchedule(int id);
+
+ List<Schedule> queryReleaseSchedulerListByProcessDefinitionCode(long processDefinitionCode);
+
+ Map<Long, String> queryWorkerGroupByProcessDefinitionCodes(List<Long> processDefinitionCodeList);
+
+ List<DependentProcessDefinition> queryDependentProcessDefinitionByProcessDefinitionCode(long processDefinitionCode);
+
+ List<ProcessInstance> queryNeedFailoverProcessInstances(String host);
+
+ List<String> queryNeedFailoverProcessInstanceHost();
- /**
- * handle Command (construct ProcessInstance from Command) , wrapped in transaction
- *
- * @param logger logger
- * @param host host
- * @param command found command
- * @return process instance
- */
- @Transactional
- public ProcessInstance handleCommand(Logger logger, String host, Command command) {
- ProcessInstance processInstance = constructProcessInstance(command, host);
- // cannot construct process instance, return null
- if (processInstance == null) {
- logger.error("scan command, command parameter is error: {}", command);
- moveToErrorCommand(command, "process instance is null");
- return null;
- }
- processInstance.setCommandType(command.getCommandType());
- processInstance.addHistoryCmd(command.getCommandType());
- //if the processDefinition is serial
- ProcessDefinition processDefinition = this.findProcessDefinition(processInstance.getProcessDefinitionCode(), processInstance.getProcessDefinitionVersion());
- if (processDefinition.getExecutionType().typeIsSerial()) {
- saveSerialProcess(processInstance, processDefinition);
- if (processInstance.getState() != ExecutionStatus.SUBMITTED_SUCCESS) {
- setSubProcessParam(processInstance);
- deleteCommandWithCheck(command.getId());
- return null;
- }
- } else {
- saveProcessInstance(processInstance);
- }
- setSubProcessParam(processInstance);
- deleteCommandWithCheck(command.getId());
- return processInstance;
- }
-
- protected void saveSerialProcess(ProcessInstance processInstance, ProcessDefinition processDefinition) {
- processInstance.setState(ExecutionStatus.SERIAL_WAIT);
- saveProcessInstance(processInstance);
- //serial wait
- //when we get the running instance(or waiting instance) only get the priority instance(by id)
- if (processDefinition.getExecutionType().typeIsSerialWait()) {
- while (true) {
- List<ProcessInstance> runningProcessInstances = this.processInstanceMapper.queryByProcessDefineCodeAndStatusAndNextId(processInstance.getProcessDefinitionCode(),
- Constants.RUNNING_PROCESS_STATE, processInstance.getId());
- if (CollectionUtils.isEmpty(runningProcessInstances)) {
- processInstance.setState(ExecutionStatus.SUBMITTED_SUCCESS);
- saveProcessInstance(processInstance);
- return;
- }
- ProcessInstance runningProcess = runningProcessInstances.get(0);
- if (this.processInstanceMapper.updateNextProcessIdById(processInstance.getId(), runningProcess.getId())) {
- return;
- }
- }
- } else if (processDefinition.getExecutionType().typeIsSerialDiscard()) {
- List<ProcessInstance> runningProcessInstances = this.processInstanceMapper.queryByProcessDefineCodeAndStatusAndNextId(processInstance.getProcessDefinitionCode(),
- Constants.RUNNING_PROCESS_STATE, processInstance.getId());
- if (CollectionUtils.isEmpty(runningProcessInstances)) {
- processInstance.setState(ExecutionStatus.STOP);
- saveProcessInstance(processInstance);
- }
- } else if (processDefinition.getExecutionType().typeIsSerialPriority()) {
- List<ProcessInstance> runningProcessInstances = this.processInstanceMapper.queryByProcessDefineCodeAndStatusAndNextId(processInstance.getProcessDefinitionCode(),
- Constants.RUNNING_PROCESS_STATE, processInstance.getId());
- if (CollectionUtils.isNotEmpty(runningProcessInstances)) {
- for (ProcessInstance info : runningProcessInstances) {
- info.setCommandType(CommandType.STOP);
- info.addHistoryCmd(CommandType.STOP);
- info.setState(ExecutionStatus.READY_STOP);
- int update = updateProcessInstance(info);
- // determine whether the process is normal
- if (update > 0) {
- String host = info.getHost();
- String address = host.split(":")[0];
- int port = Integer.parseInt(host.split(":")[1]);
- StateEventChangeCommand stateEventChangeCommand = new StateEventChangeCommand(
- info.getId(), 0, info.getState(), info.getId(), 0
- );
- try {
- stateEventCallbackService.sendResult(address, port, stateEventChangeCommand.convert2Command());
- } catch (Exception e) {
- logger.error("sendResultError");
- }
- }
- }
- }
- }
- }
-
- /**
- * save error command, and delete original command
- *
- * @param command command
- * @param message message
- */
- public void moveToErrorCommand(Command command, String message) {
- ErrorCommand errorCommand = new ErrorCommand(command, message);
- this.errorCommandMapper.insert(errorCommand);
- this.commandMapper.deleteById(command.getId());
- }
-
- /**
- * set process waiting thread
- *
- * @param command command
- * @param processInstance processInstance
- * @return process instance
- */
- private ProcessInstance setWaitingThreadProcess(Command command, ProcessInstance processInstance) {
- processInstance.setState(ExecutionStatus.WAITING_THREAD);
- if (command.getCommandType() != CommandType.RECOVER_WAITING_THREAD) {
- processInstance.addHistoryCmd(command.getCommandType());
- }
- saveProcessInstance(processInstance);
- this.setSubProcessParam(processInstance);
- createRecoveryWaitingThreadCommand(command, processInstance);
- return null;
- }
-
- /**
- * insert one command
- *
- * @param command command
- * @return create result
- */
- public int createCommand(Command command) {
- int result = 0;
- if (command != null) {
- result = commandMapper.insert(command);
- }
- return result;
- }
-
- /**
- * get command page
- */
- public List<Command> findCommandPage(int pageSize, int pageNumber) {
- return commandMapper.queryCommandPage(pageSize, pageNumber * pageSize);
- }
-
- /**
- * get command page
- */
- public List<Command> findCommandPageBySlot(int pageSize, int pageNumber, int masterCount, int thisMasterSlot) {
- if (masterCount <= 0) {
- return Lists.newArrayList();
- }
- return commandMapper.queryCommandPageBySlot(pageSize, pageNumber * pageSize, masterCount, thisMasterSlot);
- }
-
- /**
- * check the input command exists in queue list
- *
- * @param command command
- * @return create command result
- */
- public boolean verifyIsNeedCreateCommand(Command command) {
- boolean isNeedCreate = true;
- EnumMap<CommandType, Integer> cmdTypeMap = new EnumMap<>(CommandType.class);
- cmdTypeMap.put(CommandType.REPEAT_RUNNING, 1);
- cmdTypeMap.put(CommandType.RECOVER_SUSPENDED_PROCESS, 1);
- cmdTypeMap.put(CommandType.START_FAILURE_TASK_PROCESS, 1);
- CommandType commandType = command.getCommandType();
-
- if (cmdTypeMap.containsKey(commandType)) {
- ObjectNode cmdParamObj = JSONUtils.parseObject(command.getCommandParam());
- int processInstanceId = cmdParamObj.path(CMD_PARAM_RECOVER_PROCESS_ID_STRING).asInt();
-
- List<Command> commands = commandMapper.selectList(null);
- // for all commands
- for (Command tmpCommand : commands) {
- if (cmdTypeMap.containsKey(tmpCommand.getCommandType())) {
- ObjectNode tempObj = JSONUtils.parseObject(tmpCommand.getCommandParam());
- if (tempObj != null && processInstanceId == tempObj.path(CMD_PARAM_RECOVER_PROCESS_ID_STRING).asInt()) {
- isNeedCreate = false;
- break;
- }
- }
- }
- }
- return isNeedCreate;
- }
-
- /**
- * find process instance detail by id
- *
- * @param processId processId
- * @return process instance
- */
- public ProcessInstance findProcessInstanceDetailById(int processId) {
- return processInstanceMapper.queryDetailById(processId);
- }
-
- /**
- * get task node list by definitionId
- */
- public List<TaskDefinition> getTaskNodeListByDefinition(long defineCode) {
- ProcessDefinition processDefinition = processDefineMapper.queryByCode(defineCode);
- if (processDefinition == null) {
- logger.error("process define not exists");
- return Lists.newArrayList();
- }
- List<ProcessTaskRelationLog> processTaskRelations = processTaskRelationLogMapper.queryByProcessCodeAndVersion(processDefinition.getCode(), processDefinition.getVersion());
- Set<TaskDefinition> taskDefinitionSet = new HashSet<>();
- for (ProcessTaskRelationLog processTaskRelation : processTaskRelations) {
- if (processTaskRelation.getPostTaskCode() > 0) {
- taskDefinitionSet.add(new TaskDefinition(processTaskRelation.getPostTaskCode(), processTaskRelation.getPostTaskVersion()));
- }
- }
- if (taskDefinitionSet.isEmpty()) {
- return Lists.newArrayList();
- }
- List<TaskDefinitionLog> taskDefinitionLogs = taskDefinitionLogMapper.queryByTaskDefinitions(taskDefinitionSet);
- return Lists.newArrayList(taskDefinitionLogs);
- }
-
- /**
- * find process instance by id
- *
- * @param processId processId
- * @return process instance
- */
- public ProcessInstance findProcessInstanceById(int processId) {
- return processInstanceMapper.selectById(processId);
- }
-
- /**
- * find process define by id.
- *
- * @param processDefinitionId processDefinitionId
- * @return process definition
- */
- public ProcessDefinition findProcessDefineById(int processDefinitionId) {
- return processDefineMapper.selectById(processDefinitionId);
- }
-
- /**
- * find process define by code and version.
- *
- * @param processDefinitionCode processDefinitionCode
- * @return process definition
- */
- public ProcessDefinition findProcessDefinition(Long processDefinitionCode, int version) {
- ProcessDefinition processDefinition = processDefineMapper.queryByCode(processDefinitionCode);
- if (processDefinition == null || processDefinition.getVersion() != version) {
- processDefinition = processDefineLogMapper.queryByDefinitionCodeAndVersion(processDefinitionCode, version);
- if (processDefinition != null) {
- processDefinition.setId(0);
- }
- }
- return processDefinition;
- }
-
- /**
- * find process define by code.
- *
- * @param processDefinitionCode processDefinitionCode
- * @return process definition
- */
- public ProcessDefinition findProcessDefinitionByCode(Long processDefinitionCode) {
- return processDefineMapper.queryByCode(processDefinitionCode);
- }
-
- /**
- * delete work process instance by id
- *
- * @param processInstanceId processInstanceId
- * @return delete process instance result
- */
- public int deleteWorkProcessInstanceById(int processInstanceId) {
- return processInstanceMapper.deleteById(processInstanceId);
- }
-
- /**
- * delete all sub process by parent instance id
- *
- * @param processInstanceId processInstanceId
- * @return delete all sub process instance result
- */
- public int deleteAllSubWorkProcessByParentId(int processInstanceId) {
-
- List<Integer> subProcessIdList = processInstanceMapMapper.querySubIdListByParentId(processInstanceId);
-
- for (Integer subId : subProcessIdList) {
- deleteAllSubWorkProcessByParentId(subId);
- deleteWorkProcessMapByParentId(subId);
- removeTaskLogFile(subId);
- deleteWorkProcessInstanceById(subId);
- }
- return 1;
- }
-
- /**
- * remove task log file
- *
- * @param processInstanceId processInstanceId
- */
- public void removeTaskLogFile(Integer processInstanceId) {
- List<TaskInstance> taskInstanceList = findValidTaskListByProcessId(processInstanceId);
- if (CollectionUtils.isEmpty(taskInstanceList)) {
- return;
- }
- try (LogClientService logClient = new LogClientService()) {
- for (TaskInstance taskInstance : taskInstanceList) {
- String taskLogPath = taskInstance.getLogPath();
- if (StringUtils.isEmpty(taskInstance.getHost())) {
- continue;
- }
- Host host = Host.of(taskInstance.getHost());
- // remove task log from loggerserver
- logClient.removeTaskLog(host.getIp(), host.getPort(), taskLogPath);
- }
- }
- }
-
- /**
- * recursive delete all task instance by process instance id
- */
- public void deleteWorkTaskInstanceByProcessInstanceId(int processInstanceId) {
- List<TaskInstance> taskInstanceList = findValidTaskListByProcessId(processInstanceId);
- if (CollectionUtils.isEmpty(taskInstanceList)) {
- return;
- }
-
- List<Integer> taskInstanceIdList = new ArrayList<>();
-
- for (TaskInstance taskInstance : taskInstanceList) {
- taskInstanceIdList.add(taskInstance.getId());
- }
-
- taskInstanceMapper.deleteBatchIds(taskInstanceIdList);
- }
-
- /**
- * recursive query sub process definition id by parent id.
- *
- * @param parentCode parentCode
- * @param ids ids
- */
- public void recurseFindSubProcess(long parentCode, List<Long> ids) {
- List<TaskDefinition> taskNodeList = this.getTaskNodeListByDefinition(parentCode);
-
- if (taskNodeList != null && !taskNodeList.isEmpty()) {
-
- for (TaskDefinition taskNode : taskNodeList) {
- String parameter = taskNode.getTaskParams();
- ObjectNode parameterJson = JSONUtils.parseObject(parameter);
- if (parameterJson.get(CMD_PARAM_SUB_PROCESS_DEFINE_CODE) != null) {
- SubProcessParameters subProcessParam = JSONUtils.parseObject(parameter, SubProcessParameters.class);
- ids.add(subProcessParam.getProcessDefinitionCode());
- recurseFindSubProcess(subProcessParam.getProcessDefinitionCode(), ids);
- }
- }
- }
- }
-
- /**
- * create recovery waiting thread command when thread pool is not enough for the process instance.
- * sub work process instance need not to create recovery command.
- * create recovery waiting thread command and delete origin command at the same time.
- * if the recovery command is exists, only update the field update_time
- *
- * @param originCommand originCommand
- * @param processInstance processInstance
- */
- public void createRecoveryWaitingThreadCommand(Command originCommand, ProcessInstance processInstance) {
-
- // sub process doesnot need to create wait command
- if (processInstance.getIsSubProcess() == Flag.YES) {
- if (originCommand != null) {
- commandMapper.deleteById(originCommand.getId());
- }
- return;
- }
- Map<String, String> cmdParam = new HashMap<>();
- cmdParam.put(Constants.CMD_PARAM_RECOVERY_WAITING_THREAD, String.valueOf(processInstance.getId()));
- // process instance quit by "waiting thread" state
- if (originCommand == null) {
- Command command = new Command(
- CommandType.RECOVER_WAITING_THREAD,
- processInstance.getTaskDependType(),
- processInstance.getFailureStrategy(),
- processInstance.getExecutorId(),
- processInstance.getProcessDefinition().getCode(),
- JSONUtils.toJsonString(cmdParam),
- processInstance.getWarningType(),
- processInstance.getWarningGroupId(),
- processInstance.getScheduleTime(),
- processInstance.getWorkerGroup(),
- processInstance.getEnvironmentCode(),
- processInstance.getProcessInstancePriority(),
- processInstance.getDryRun(),
- processInstance.getId(),
- processInstance.getProcessDefinitionVersion()
- );
- saveCommand(command);
- return;
- }
-
- // update the command time if current command if recover from waiting
- if (originCommand.getCommandType() == CommandType.RECOVER_WAITING_THREAD) {
- originCommand.setUpdateTime(new Date());
- saveCommand(originCommand);
- } else {
- // delete old command and create new waiting thread command
- commandMapper.deleteById(originCommand.getId());
- originCommand.setId(0);
- originCommand.setCommandType(CommandType.RECOVER_WAITING_THREAD);
- originCommand.setUpdateTime(new Date());
- originCommand.setCommandParam(JSONUtils.toJsonString(cmdParam));
- originCommand.setProcessInstancePriority(processInstance.getProcessInstancePriority());
- saveCommand(originCommand);
- }
- }
-
- /**
- * get schedule time from command
- *
- * @param command command
- * @param cmdParam cmdParam map
- * @return date
- */
- private Date getScheduleTime(Command command, Map<String, String> cmdParam) {
- Date scheduleTime = command.getScheduleTime();
- if (scheduleTime == null
- && cmdParam != null
- && cmdParam.containsKey(CMDPARAM_COMPLEMENT_DATA_START_DATE)) {
-
- Date start = DateUtils.stringToDate(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_START_DATE));
- Date end = DateUtils.stringToDate(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE));
- List<Schedule> schedules = queryReleaseSchedulerListByProcessDefinitionCode(command.getProcessDefinitionCode());
- List<Date> complementDateList = CronUtils.getSelfFireDateList(start, end, schedules);
-
- if (complementDateList.size() > 0) {
- scheduleTime = complementDateList.get(0);
- } else {
- logger.error("set scheduler time error: complement date list is empty, command: {}",
- command.toString());
- }
- }
- return scheduleTime;
- }
-
- /**
- * generate a new work process instance from command.
- *
- * @param processDefinition processDefinition
- * @param command command
- * @param cmdParam cmdParam map
- * @return process instance
- */
- private ProcessInstance generateNewProcessInstance(ProcessDefinition processDefinition,
- Command command,
- Map<String, String> cmdParam) {
- ProcessInstance processInstance = new ProcessInstance(processDefinition);
- processInstance.setProcessDefinitionCode(processDefinition.getCode());
- processInstance.setProcessDefinitionVersion(processDefinition.getVersion());
- processInstance.setState(ExecutionStatus.RUNNING_EXECUTION);
- processInstance.setRecovery(Flag.NO);
- processInstance.setStartTime(new Date());
- processInstance.setRestartTime(processInstance.getStartTime());
- processInstance.setRunTimes(1);
- processInstance.setMaxTryTimes(0);
- processInstance.setCommandParam(command.getCommandParam());
- processInstance.setCommandType(command.getCommandType());
- processInstance.setIsSubProcess(Flag.NO);
- processInstance.setTaskDependType(command.getTaskDependType());
- processInstance.setFailureStrategy(command.getFailureStrategy());
- processInstance.setExecutorId(command.getExecutorId());
- WarningType warningType = command.getWarningType() == null ? WarningType.NONE : command.getWarningType();
- processInstance.setWarningType(warningType);
- Integer warningGroupId = command.getWarningGroupId() == null ? 0 : command.getWarningGroupId();
- processInstance.setWarningGroupId(warningGroupId);
- processInstance.setDryRun(command.getDryRun());
-
- if (command.getScheduleTime() != null) {
- processInstance.setScheduleTime(command.getScheduleTime());
- }
- processInstance.setCommandStartTime(command.getStartTime());
- processInstance.setLocations(processDefinition.getLocations());
-
- // reset global params while there are start parameters
- setGlobalParamIfCommanded(processDefinition, cmdParam);
-
- // curing global params
- processInstance.setGlobalParams(ParameterUtils.curingGlobalParams(
- processDefinition.getGlobalParamMap(),
- processDefinition.getGlobalParamList(),
- getCommandTypeIfComplement(processInstance, command),
- processInstance.getScheduleTime()));
-
- // set process instance priority
- processInstance.setProcessInstancePriority(command.getProcessInstancePriority());
- String workerGroup = StringUtils.isBlank(command.getWorkerGroup()) ? Constants.DEFAULT_WORKER_GROUP : command.getWorkerGroup();
- processInstance.setWorkerGroup(workerGroup);
- processInstance.setEnvironmentCode(Objects.isNull(command.getEnvironmentCode()) ? -1 : command.getEnvironmentCode());
- processInstance.setTimeout(processDefinition.getTimeout());
- processInstance.setTenantId(processDefinition.getTenantId());
- return processInstance;
- }
-
- private void setGlobalParamIfCommanded(ProcessDefinition processDefinition, Map<String, String> cmdParam) {
- // get start params from command param
- Map<String, String> startParamMap = new HashMap<>();
- if (cmdParam != null && cmdParam.containsKey(Constants.CMD_PARAM_START_PARAMS)) {
- String startParamJson = cmdParam.get(Constants.CMD_PARAM_START_PARAMS);
- startParamMap = JSONUtils.toMap(startParamJson);
- }
- Map<String, String> fatherParamMap = new HashMap<>();
- if (cmdParam != null && cmdParam.containsKey(Constants.CMD_PARAM_FATHER_PARAMS)) {
- String fatherParamJson = cmdParam.get(Constants.CMD_PARAM_FATHER_PARAMS);
- fatherParamMap = JSONUtils.toMap(fatherParamJson);
- }
- startParamMap.putAll(fatherParamMap);
- // set start param into global params
- if (startParamMap.size() > 0
- && processDefinition.getGlobalParamMap() != null) {
- for (Map.Entry<String, String> param : processDefinition.getGlobalParamMap().entrySet()) {
- String val = startParamMap.get(param.getKey());
- if (val != null) {
- param.setValue(val);
- }
- }
- }
- }
-
- /**
- * get process tenant
- * there is tenant id in definition, use the tenant of the definition.
- * if there is not tenant id in the definiton or the tenant not exist
- * use definition creator's tenant.
- *
- * @param tenantId tenantId
- * @param userId userId
- * @return tenant
- */
- public Tenant getTenantForProcess(int tenantId, int userId) {
- Tenant tenant = null;
- if (tenantId >= 0) {
- tenant = tenantMapper.queryById(tenantId);
- }
-
- if (userId == 0) {
- return null;
- }
-
- if (tenant == null) {
- User user = userMapper.selectById(userId);
- tenant = tenantMapper.queryById(user.getTenantId());
- }
- return tenant;
- }
-
- /**
- * get an environment
- * use the code of the environment to find a environment.
- *
- * @param environmentCode environmentCode
- * @return Environment
- */
- public Environment findEnvironmentByCode(Long environmentCode) {
- Environment environment = null;
- if (environmentCode >= 0) {
- environment = environmentMapper.queryByEnvironmentCode(environmentCode);
- }
- return environment;
- }
-
- /**
- * check command parameters is valid
- *
- * @param command command
- * @param cmdParam cmdParam map
- * @return whether command param is valid
- */
- private Boolean checkCmdParam(Command command, Map<String, String> cmdParam) {
- if (command.getTaskDependType() == TaskDependType.TASK_ONLY || command.getTaskDependType() == TaskDependType.TASK_PRE) {
- if (cmdParam == null
- || !cmdParam.containsKey(Constants.CMD_PARAM_START_NODES)
- || cmdParam.get(Constants.CMD_PARAM_START_NODES).isEmpty()) {
- logger.error("command node depend type is {}, but start nodes is null ", command.getTaskDependType());
- return false;
- }
- }
- return true;
- }
-
- /**
- * construct process instance according to one command.
- *
- * @param command command
- * @param host host
- * @return process instance
- */
- protected ProcessInstance constructProcessInstance(Command command, String host) {
- ProcessInstance processInstance;
- ProcessDefinition processDefinition;
- CommandType commandType = command.getCommandType();
-
- processDefinition = this.findProcessDefinition(command.getProcessDefinitionCode(), command.getProcessDefinitionVersion());
- if (processDefinition == null) {
- logger.error("cannot find the work process define! define code : {}", command.getProcessDefinitionCode());
- return null;
- }
- Map<String, String> cmdParam = JSONUtils.toMap(command.getCommandParam());
- int processInstanceId = command.getProcessInstanceId();
- if (processInstanceId == 0) {
- processInstance = generateNewProcessInstance(processDefinition, command, cmdParam);
- } else {
- processInstance = this.findProcessInstanceDetailById(processInstanceId);
- if (processInstance == null) {
- return processInstance;
- }
- }
- if (cmdParam != null) {
- CommandType commandTypeIfComplement = getCommandTypeIfComplement(processInstance, command);
- // reset global params while repeat running is needed by cmdParam
- if (commandTypeIfComplement == CommandType.REPEAT_RUNNING) {
- setGlobalParamIfCommanded(processDefinition, cmdParam);
- }
-
- // Recalculate global parameters after rerun.
- processInstance.setGlobalParams(ParameterUtils.curingGlobalParams(
- processDefinition.getGlobalParamMap(),
- processDefinition.getGlobalParamList(),
- commandTypeIfComplement,
- processInstance.getScheduleTime()));
- processInstance.setProcessDefinition(processDefinition);
- }
- //reset command parameter
- if (processInstance.getCommandParam() != null) {
- Map<String, String> processCmdParam = JSONUtils.toMap(processInstance.getCommandParam());
- processCmdParam.forEach((key, value) -> {
- if (!cmdParam.containsKey(key)) {
- cmdParam.put(key, value);
- }
- });
- }
- // reset command parameter if sub process
- if (cmdParam != null && cmdParam.containsKey(Constants.CMD_PARAM_SUB_PROCESS)) {
- processInstance.setCommandParam(command.getCommandParam());
- }
- if (Boolean.FALSE.equals(checkCmdParam(command, cmdParam))) {
- logger.error("command parameter check failed!");
- return null;
- }
- if (command.getScheduleTime() != null) {
- processInstance.setScheduleTime(command.getScheduleTime());
- }
- processInstance.setHost(host);
- processInstance.setRestartTime(new Date());
- ExecutionStatus runStatus = ExecutionStatus.RUNNING_EXECUTION;
- int runTime = processInstance.getRunTimes();
- switch (commandType) {
- case START_PROCESS:
- break;
- case START_FAILURE_TASK_PROCESS:
- // find failed tasks and init these tasks
- List<Integer> failedList = this.findTaskIdByInstanceState(processInstance.getId(), ExecutionStatus.FAILURE);
- List<Integer> toleranceList = this.findTaskIdByInstanceState(processInstance.getId(), ExecutionStatus.NEED_FAULT_TOLERANCE);
- List<Integer> killedList = this.findTaskIdByInstanceState(processInstance.getId(), ExecutionStatus.KILL);
- cmdParam.remove(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING);
-
- failedList.addAll(killedList);
- failedList.addAll(toleranceList);
- for (Integer taskId : failedList) {
- initTaskInstance(this.findTaskInstanceById(taskId));
- }
- cmdParam.put(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING,
- String.join(Constants.COMMA, convertIntListToString(failedList)));
- processInstance.setCommandParam(JSONUtils.toJsonString(cmdParam));
- processInstance.setRunTimes(runTime + 1);
- break;
- case START_CURRENT_TASK_PROCESS:
- break;
- case RECOVER_WAITING_THREAD:
- break;
- case RECOVER_SUSPENDED_PROCESS:
- // find pause tasks and init task's state
- cmdParam.remove(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING);
- List<Integer> suspendedNodeList = this.findTaskIdByInstanceState(processInstance.getId(), ExecutionStatus.PAUSE);
- List<Integer> stopNodeList = findTaskIdByInstanceState(processInstance.getId(),
- ExecutionStatus.KILL);
- suspendedNodeList.addAll(stopNodeList);
- for (Integer taskId : suspendedNodeList) {
- // initialize the pause state
- initTaskInstance(this.findTaskInstanceById(taskId));
- }
- cmdParam.put(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING, String.join(",", convertIntListToString(suspendedNodeList)));
- processInstance.setCommandParam(JSONUtils.toJsonString(cmdParam));
- processInstance.setRunTimes(runTime + 1);
- break;
- case RECOVER_TOLERANCE_FAULT_PROCESS:
- // recover tolerance fault process
- processInstance.setRecovery(Flag.YES);
- runStatus = processInstance.getState();
- break;
- case COMPLEMENT_DATA:
- // delete all the valid tasks when complement data if id is not null
- if (processInstance.getId() != 0) {
- List<TaskInstance> taskInstanceList = this.findValidTaskListByProcessId(processInstance.getId());
- for (TaskInstance taskInstance : taskInstanceList) {
- taskInstance.setFlag(Flag.NO);
- this.updateTaskInstance(taskInstance);
- }
- }
- break;
- case REPEAT_RUNNING:
- // delete the recover task names from command parameter
- if (cmdParam.containsKey(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING)) {
- cmdParam.remove(Constants.CMD_PARAM_RECOVERY_START_NODE_STRING);
- processInstance.setCommandParam(JSONUtils.toJsonString(cmdParam));
- }
- // delete all the valid tasks when repeat running
- List<TaskInstance> validTaskList = findValidTaskListByProcessId(processInstance.getId());
- for (TaskInstance taskInstance : validTaskList) {
- taskInstance.setFlag(Flag.NO);
- updateTaskInstance(taskInstance);
- }
- processInstance.setStartTime(new Date());
- processInstance.setRestartTime(processInstance.getStartTime());
- processInstance.setEndTime(null);
- processInstance.setRunTimes(runTime + 1);
- initComplementDataParam(processDefinition, processInstance, cmdParam);
- break;
- case SCHEDULER:
- break;
- default:
- break;
- }
- processInstance.setState(runStatus);
- return processInstance;
- }
-
- /**
- * get process definition by command
- * If it is a fault-tolerant command, get the specified version of ProcessDefinition through ProcessInstance
- * Otherwise, get the latest version of ProcessDefinition
- *
- * @return ProcessDefinition
- */
- private ProcessDefinition getProcessDefinitionByCommand(long processDefinitionCode, Map<String, String> cmdParam) {
- if (cmdParam != null) {
- int processInstanceId = 0;
- if (cmdParam.containsKey(Constants.CMD_PARAM_RECOVER_PROCESS_ID_STRING)) {
- processInstanceId = Integer.parseInt(cmdParam.get(Constants.CMD_PARAM_RECOVER_PROCESS_ID_STRING));
- } else if (cmdParam.containsKey(Constants.CMD_PARAM_SUB_PROCESS)) {
- processInstanceId = Integer.parseInt(cmdParam.get(Constants.CMD_PARAM_SUB_PROCESS));
- } else if (cmdParam.containsKey(Constants.CMD_PARAM_RECOVERY_WAITING_THREAD)) {
- processInstanceId = Integer.parseInt(cmdParam.get(Constants.CMD_PARAM_RECOVERY_WAITING_THREAD));
- }
-
- if (processInstanceId != 0) {
- ProcessInstance processInstance = this.findProcessInstanceDetailById(processInstanceId);
- if (processInstance == null) {
- return null;
- }
-
- return processDefineLogMapper.queryByDefinitionCodeAndVersion(
- processInstance.getProcessDefinitionCode(), processInstance.getProcessDefinitionVersion());
- }
- }
-
- return processDefineMapper.queryByCode(processDefinitionCode);
- }
-
- /**
- * return complement data if the process start with complement data
- *
- * @param processInstance processInstance
- * @param command command
- * @return command type
- */
- private CommandType getCommandTypeIfComplement(ProcessInstance processInstance, Command command) {
- if (CommandType.COMPLEMENT_DATA == processInstance.getCmdTypeIfComplement()) {
- return CommandType.COMPLEMENT_DATA;
- } else {
- return command.getCommandType();
- }
- }
-
- /**
- * initialize complement data parameters
- *
- * @param processDefinition processDefinition
- * @param processInstance processInstance
- * @param cmdParam cmdParam
- */
- private void initComplementDataParam(ProcessDefinition processDefinition,
- ProcessInstance processInstance,
- Map<String, String> cmdParam) {
- if (!processInstance.isComplementData()) {
- return;
- }
-
- Date start = DateUtils.stringToDate(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_START_DATE));
- Date end = DateUtils.stringToDate(cmdParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE));
- List<Schedule> listSchedules = queryReleaseSchedulerListByProcessDefinitionCode(processInstance.getProcessDefinitionCode());
- List<Date> complementDate = CronUtils.getSelfFireDateList(start, end, listSchedules);
-
- if (complementDate.size() > 0
- && Flag.NO == processInstance.getIsSubProcess()) {
- processInstance.setScheduleTime(complementDate.get(0));
- }
- processInstance.setGlobalParams(ParameterUtils.curingGlobalParams(
- processDefinition.getGlobalParamMap(),
- processDefinition.getGlobalParamList(),
- CommandType.COMPLEMENT_DATA, processInstance.getScheduleTime()));
- }
-
- /**
- * set sub work process parameters.
- * handle sub work process instance, update relation table and command parameters
- * set sub work process flag, extends parent work process command parameters
- *
- * @param subProcessInstance subProcessInstance
- */
- public void setSubProcessParam(ProcessInstance subProcessInstance) {
- String cmdParam = subProcessInstance.getCommandParam();
- if (StringUtils.isEmpty(cmdParam)) {
- return;
- }
- Map<String, String> paramMap = JSONUtils.toMap(cmdParam);
- // write sub process id into cmd param.
- if (paramMap.containsKey(CMD_PARAM_SUB_PROCESS)
- && CMD_PARAM_EMPTY_SUB_PROCESS.equals(paramMap.get(CMD_PARAM_SUB_PROCESS))) {
- paramMap.remove(CMD_PARAM_SUB_PROCESS);
- paramMap.put(CMD_PARAM_SUB_PROCESS, String.valueOf(subProcessInstance.getId()));
- subProcessInstance.setCommandParam(JSONUtils.toJsonString(paramMap));
- subProcessInstance.setIsSubProcess(Flag.YES);
- this.saveProcessInstance(subProcessInstance);
- }
- // copy parent instance user def params to sub process..
- String parentInstanceId = paramMap.get(CMD_PARAM_SUB_PROCESS_PARENT_INSTANCE_ID);
- if (StringUtils.isNotEmpty(parentInstanceId)) {
- ProcessInstance parentInstance = findProcessInstanceDetailById(Integer.parseInt(parentInstanceId));
- if (parentInstance != null) {
- subProcessInstance.setGlobalParams(
- joinGlobalParams(parentInstance.getGlobalParams(), subProcessInstance.getGlobalParams()));
- this.saveProcessInstance(subProcessInstance);
- } else {
- logger.error("sub process command params error, cannot find parent instance: {} ", cmdParam);
- }
- }
- ProcessInstanceMap processInstanceMap = JSONUtils.parseObject(cmdParam, ProcessInstanceMap.class);
- if (processInstanceMap == null || processInstanceMap.getParentProcessInstanceId() == 0) {
- return;
- }
- // update sub process id to process map table
- processInstanceMap.setProcessInstanceId(subProcessInstance.getId());
-
- this.updateWorkProcessInstanceMap(processInstanceMap);
- }
-
- /**
- * join parent global params into sub process.
- * only the keys doesn't in sub process global would be joined.
- *
- * @param parentGlobalParams parentGlobalParams
- * @param subGlobalParams subGlobalParams
- * @return global params join
- */
- private String joinGlobalParams(String parentGlobalParams, String subGlobalParams) {
-
- // Since JSONUtils.toList return unmodified list, we need to creat a new List here.
- List<Property> parentParams = Lists.newArrayList(JSONUtils.toList(parentGlobalParams, Property.class));
- List<Property> subParams = JSONUtils.toList(subGlobalParams, Property.class);
-
- Set<String> parentParamKeys = parentParams.stream().map(Property::getProp).collect(toSet());
-
- // We will combine the params of parent workflow and sub workflow
- // If the params are defined in both, we will use parent's params to override the sub workflow(ISSUE-7962)
- // todo: Do we need to consider the other attribute of Property?
- // e.g. the subProp's type is not equals with parent, or subProp's direct is not equals with parent
- // It's suggested to add node name in property, this kind of problem can be solved.
- List<Property> extraSubParams = subParams.stream()
- .filter(subProp -> !parentParamKeys.contains(subProp.getProp())).collect(Collectors.toList());
- parentParams.addAll(extraSubParams);
- return JSONUtils.toJsonString(parentParams);
- }
-
- /**
- * initialize task instance
- *
- * @param taskInstance taskInstance
- */
- private void initTaskInstance(TaskInstance taskInstance) {
-
- if (!taskInstance.isSubProcess()
- && (taskInstance.getState().typeIsCancel() || taskInstance.getState().typeIsFailure())) {
- taskInstance.setFlag(Flag.NO);
- updateTaskInstance(taskInstance);
- return;
- }
- taskInstance.setState(ExecutionStatus.SUBMITTED_SUCCESS);
- updateTaskInstance(taskInstance);
- }
-
- /**
- * retry submit task to db
- */
- public TaskInstance submitTaskWithRetry(ProcessInstance processInstance, TaskInstance taskInstance, int commitRetryTimes, int commitInterval) {
- int retryTimes = 1;
- TaskInstance task = null;
- while (retryTimes <= commitRetryTimes) {
- try {
- // submit task to db
- task = SpringApplicationContext.getBean(ProcessService.class).submitTask(processInstance, taskInstance);
- if (task != null && task.getId() != 0) {
- break;
- }
- logger.error("task commit to db failed , taskId {} has already retry {} times, please check the database", taskInstance.getId(), retryTimes);
- Thread.sleep(commitInterval);
- } catch (Exception e) {
- logger.error("task commit to db failed", e);
- }
- retryTimes += 1;
- }
- return task;
- }
-
- /**
- * submit task to db
- * submit sub process to command
- *
- * @param processInstance processInstance
- * @param taskInstance taskInstance
- * @return task instance
- */
- @Transactional(rollbackFor = Exception.class)
- public TaskInstance submitTask(ProcessInstance processInstance, TaskInstance taskInstance) {
- logger.info("start submit task : {}, instance id:{}, state: {}",
- taskInstance.getName(), taskInstance.getProcessInstanceId(), processInstance.getState());
- //submit to db
- TaskInstance task = submitTaskInstanceToDB(taskInstance, processInstance);
- if (task == null) {
- logger.error("end submit task to db error, task name:{}, process id:{} state: {} ",
- taskInstance.getName(), taskInstance.getProcessInstance(), processInstance.getState());
- return null;
- }
-
- if (!task.getState().typeIsFinished()) {
- createSubWorkProcess(processInstance, task);
- }
-
- logger.info("end submit task to db successfully:{} {} state:{} complete, instance id:{} state: {} ",
- taskInstance.getId(), taskInstance.getName(), task.getState(), processInstance.getId(), processInstance.getState());
- return task;
- }
-
- /**
- * set work process instance map
- * consider o
- * repeat running does not generate new sub process instance
- * set map {parent instance id, task instance id, 0(child instance id)}
- *
- * @param parentInstance parentInstance
- * @param parentTask parentTask
- * @return process instance map
- */
- private ProcessInstanceMap setProcessInstanceMap(ProcessInstance parentInstance, TaskInstance parentTask) {
- ProcessInstanceMap processMap = findWorkProcessMapByParent(parentInstance.getId(), parentTask.getId());
- if (processMap != null) {
- return processMap;
- }
- if (parentInstance.getCommandType() == CommandType.REPEAT_RUNNING) {
- // update current task id to map
- processMap = findPreviousTaskProcessMap(parentInstance, parentTask);
- if (processMap != null) {
- processMap.setParentTaskInstanceId(parentTask.getId());
- updateWorkProcessInstanceMap(processMap);
- return processMap;
- }
- }
- // new task
- processMap = new ProcessInstanceMap();
- processMap.setParentProcessInstanceId(parentInstance.getId());
- processMap.setParentTaskInstanceId(parentTask.getId());
- createWorkProcessInstanceMap(processMap);
- return processMap;
- }
-
- /**
- * find previous task work process map.
- *
- * @param parentProcessInstance parentProcessInstance
- * @param parentTask parentTask
- * @return process instance map
- */
- private ProcessInstanceMap findPreviousTaskProcessMap(ProcessInstance parentProcessInstance,
- TaskInstance parentTask) {
-
- Integer preTaskId = 0;
- List<TaskInstance> preTaskList = this.findPreviousTaskListByWorkProcessId(parentProcessInstance.getId());
- for (TaskInstance task : preTaskList) {
- if (task.getName().equals(parentTask.getName())) {
- preTaskId = task.getId();
- ProcessInstanceMap map = findWorkProcessMapByParent(parentProcessInstance.getId(), preTaskId);
- if (map != null) {
- return map;
- }
- }
- }
- logger.info("sub process instance is not found,parent task:{},parent instance:{}",
- parentTask.getId(), parentProcessInstance.getId());
- return null;
- }
-
- /**
- * create sub work process command
- *
- * @param parentProcessInstance parentProcessInstance
- * @param task task
- */
- public void createSubWorkProcess(ProcessInstance parentProcessInstance, TaskInstance task) {
- if (!task.isSubProcess()) {
- return;
- }
- //check create sub work flow firstly
- ProcessInstanceMap instanceMap = findWorkProcessMapByParent(parentProcessInstance.getId(), task.getId());
- if (null != instanceMap && CommandType.RECOVER_TOLERANCE_FAULT_PROCESS == parentProcessInstance.getCommandType()) {
- // recover failover tolerance would not create a new command when the sub command already have been created
- return;
- }
- instanceMap = setProcessInstanceMap(parentProcessInstance, task);
- ProcessInstance childInstance = null;
- if (instanceMap.getProcessInstanceId() != 0) {
- childInstance = findProcessInstanceById(instanceMap.getProcessInstanceId());
- }
- Command subProcessCommand = createSubProcessCommand(parentProcessInstance, childInstance, instanceMap, task);
- updateSubProcessDefinitionByParent(parentProcessInstance, subProcessCommand.getProcessDefinitionCode());
- initSubInstanceState(childInstance);
- createCommand(subProcessCommand);
- logger.info("sub process command created: {} ", subProcessCommand);
- }
-
- /**
- * complement data needs transform parent parameter to child.
- */
- protected String getSubWorkFlowParam(ProcessInstanceMap instanceMap, ProcessInstance parentProcessInstance, Map<String, String> fatherParams) {
- // set sub work process command
- String processMapStr = JSONUtils.toJsonString(instanceMap);
- Map<String, String> cmdParam = JSONUtils.toMap(processMapStr);
- if (parentProcessInstance.isComplementData()) {
- Map<String, String> parentParam = JSONUtils.toMap(parentProcessInstance.getCommandParam());
- String endTime = parentParam.get(CMDPARAM_COMPLEMENT_DATA_END_DATE);
- String startTime = parentParam.get(CMDPARAM_COMPLEMENT_DATA_START_DATE);
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_END_DATE, endTime);
- cmdParam.put(CMDPARAM_COMPLEMENT_DATA_START_DATE, startTime);
- processMapStr = JSONUtils.toJsonString(cmdParam);
- }
- if (fatherParams.size() != 0) {
- cmdParam.put(CMD_PARAM_FATHER_PARAMS, JSONUtils.toJsonString(fatherParams));
- processMapStr = JSONUtils.toJsonString(cmdParam);
- }
- return processMapStr;
- }
-
- public Map<String, String> getGlobalParamMap(String globalParams) {
- List<Property> propList;
- Map<String, String> globalParamMap = new HashMap<>();
- if (StringUtils.isNotEmpty(globalParams)) {
- propList = JSONUtils.toList(globalParams, Property.class);
- globalParamMap = propList.stream().collect(Collectors.toMap(Property::getProp, Property::getValue));
- }
-
- return globalParamMap;
- }
-
- /**
- * create sub work process command
- */
- public Command createSubProcessCommand(ProcessInstance parentProcessInstance,
- ProcessInstance childInstance,
- ProcessInstanceMap instanceMap,
- TaskInstance task) {
- CommandType commandType = getSubCommandType(parentProcessInstance, childInstance);
- Map<String, Object> subProcessParam = JSONUtils.toMap(task.getTaskParams(), String.class, Object.class);
- long childDefineCode = 0L;
- if (subProcessParam.containsKey(Constants.CMD_PARAM_SUB_PROCESS_DEFINE_CODE)) {
- childDefineCode = NumberUtils.toLong(String.valueOf(subProcessParam.get(Constants.CMD_PARAM_SUB_PROCESS_DEFINE_CODE)));
- }
- ProcessDefinition subProcessDefinition = processDefineMapper.queryByCode(childDefineCode);
-
- Object localParams = subProcessParam.get(Constants.LOCAL_PARAMS);
- List<Property> allParam = JSONUtils.toList(JSONUtils.toJsonString(localParams), Property.class);
- Map<String, String> globalMap = this.getGlobalParamMap(task.getVarPool());
- Map<String, String> fatherParams = new HashMap<>();
- if (CollectionUtils.isNotEmpty(allParam)) {
- for (Property info : allParam) {
- if (Direct.OUT == info.getDirect()) {
- continue;
- }
- fatherParams.put(info.getProp(), globalMap.get(info.getProp()));
- }
- }
- String processParam = getSubWorkFlowParam(instanceMap, parentProcessInstance, fatherParams);
- int subProcessInstanceId = childInstance == null ? 0 : childInstance.getId();
- return new Command(
- commandType,
- TaskDependType.TASK_POST,
- parentProcessInstance.getFailureStrategy(),
- parentProcessInstance.getExecutorId(),
- subProcessDefinition.getCode(),
- processParam,
- parentProcessInstance.getWarningType(),
- parentProcessInstance.getWarningGroupId(),
- parentProcessInstance.getScheduleTime(),
- task.getWorkerGroup(),
- task.getEnvironmentCode(),
- parentProcessInstance.getProcessInstancePriority(),
- parentProcessInstance.getDryRun(),
- subProcessInstanceId,
- subProcessDefinition.getVersion()
- );
- }
-
- /**
- * initialize sub work flow state
- * child instance state would be initialized when 'recovery from pause/stop/failure'
- */
- private void initSubInstanceState(ProcessInstance childInstance) {
- if (childInstance != null) {
- childInstance.setState(ExecutionStatus.RUNNING_EXECUTION);
- updateProcessInstance(childInstance);
- }
- }
-
- /**
- * get sub work flow command type
- * child instance exist: child command = fatherCommand
- * child instance not exists: child command = fatherCommand[0]
- */
- private CommandType getSubCommandType(ProcessInstance parentProcessInstance, ProcessInstance childInstance) {
- CommandType commandType = parentProcessInstance.getCommandType();
- if (childInstance == null) {
- String fatherHistoryCommand = parentProcessInstance.getHistoryCmd();
- commandType = CommandType.valueOf(fatherHistoryCommand.split(Constants.COMMA)[0]);
- }
- return commandType;
- }
-
- /**
- * update sub process definition
- *
- * @param parentProcessInstance parentProcessInstance
- * @param childDefinitionCode childDefinitionId
- */
- private void updateSubProcessDefinitionByParent(ProcessInstance parentProcessInstance, long childDefinitionCode) {
- ProcessDefinition fatherDefinition = this.findProcessDefinition(parentProcessInstance.getProcessDefinitionCode(),
- parentProcessInstance.getProcessDefinitionVersion());
- ProcessDefinition childDefinition = this.findProcessDefinitionByCode(childDefinitionCode);
- if (childDefinition != null && fatherDefinition != null) {
- childDefinition.setWarningGroupId(fatherDefinition.getWarningGroupId());
- processDefineMapper.updateById(childDefinition);
- }
- }
-
- /**
- * submit task to mysql
- *
- * @param taskInstance taskInstance
- * @param processInstance processInstance
- * @return task instance
- */
- public TaskInstance submitTaskInstanceToDB(TaskInstance taskInstance, ProcessInstance processInstance) {
- ExecutionStatus processInstanceState = processInstance.getState();
- if (processInstanceState.typeIsFinished()
- || processInstanceState == ExecutionStatus.READY_PAUSE
- || processInstanceState == ExecutionStatus.READY_STOP) {
- logger.warn("processInstance {} was {}, skip submit task", processInstance.getProcessDefinitionCode(), processInstanceState);
- return null;
- }
- taskInstance.setExecutorId(processInstance.getExecutorId());
- taskInstance.setProcessInstancePriority(processInstance.getProcessInstancePriority());
- taskInstance.setState(getSubmitTaskState(taskInstance, processInstance));
- if (taskInstance.getSubmitTime() == null) {
- taskInstance.setSubmitTime(new Date());
- }
- if (taskInstance.getFirstSubmitTime() == null) {
- taskInstance.setFirstSubmitTime(taskInstance.getSubmitTime());
- }
- boolean saveResult = saveTaskInstance(taskInstance);
- if (!saveResult) {
- return null;
- }
- return taskInstance;
- }
-
- /**
- * get submit task instance state by the work process state
- * cannot modify the task state when running/kill/submit success, or this
- * task instance is already exists in task queue .
- * return pause if work process state is ready pause
- * return stop if work process state is ready stop
- * if all of above are not satisfied, return submit success
- *
- * @param taskInstance taskInstance
- * @param processInstance processInstance
- * @return process instance state
- */
- public ExecutionStatus getSubmitTaskState(TaskInstance taskInstance, ProcessInstance processInstance) {
- ExecutionStatus state = taskInstance.getState();
- // running, delayed or killed
- // the task already exists in task queue
- // return state
- if (
- state == ExecutionStatus.RUNNING_EXECUTION
- || state == ExecutionStatus.DELAY_EXECUTION
- || state == ExecutionStatus.KILL
- || state == ExecutionStatus.DISPATCH
- ) {
- return state;
- }
- //return pasue /stop if process instance state is ready pause / stop
- // or return submit success
- if (processInstance.getState() == ExecutionStatus.READY_PAUSE) {
- state = ExecutionStatus.PAUSE;
- } else if (processInstance.getState() == ExecutionStatus.READY_STOP
- || !checkProcessStrategy(taskInstance, processInstance)) {
- state = ExecutionStatus.KILL;
- } else {
- state = ExecutionStatus.SUBMITTED_SUCCESS;
- }
- return state;
- }
-
- /**
- * check process instance strategy
- *
- * @param taskInstance taskInstance
- * @return check strategy result
- */
- private boolean checkProcessStrategy(TaskInstance taskInstance, ProcessInstance processInstance) {
- FailureStrategy failureStrategy = processInstance.getFailureStrategy();
- if (failureStrategy == FailureStrategy.CONTINUE) {
- return true;
- }
- List<TaskInstance> taskInstances = this.findValidTaskListByProcessId(taskInstance.getProcessInstanceId());
-
- for (TaskInstance task : taskInstances) {
- if (task.getState() == ExecutionStatus.FAILURE
- && task.getRetryTimes() >= task.getMaxRetryTimes()) {
- return false;
- }
- }
- return true;
- }
-
- /**
- * insert or update work process instance to data base
- *
- * @param processInstance processInstance
- */
- public void saveProcessInstance(ProcessInstance processInstance) {
- if (processInstance == null) {
- logger.error("save error, process instance is null!");
- return;
- }
- if (processInstance.getId() != 0) {
- processInstanceMapper.updateById(processInstance);
- } else {
- processInstanceMapper.insert(processInstance);
- }
- }
-
- /**
- * insert or update command
- *
- * @param command command
- * @return save command result
- */
- public int saveCommand(Command command) {
- if (command.getId() != 0) {
- return commandMapper.updateById(command);
- } else {
- return commandMapper.insert(command);
- }
- }
-
- /**
- * insert or update task instance
- *
- * @param taskInstance taskInstance
- * @return save task instance result
- */
- public boolean saveTaskInstance(TaskInstance taskInstance) {
- if (taskInstance.getId() != 0) {
- return updateTaskInstance(taskInstance);
- } else {
- return createTaskInstance(taskInstance);
- }
- }
-
- /**
- * insert task instance
- *
- * @param taskInstance taskInstance
- * @return create task instance result
- */
- public boolean createTaskInstance(TaskInstance taskInstance) {
- int count = taskInstanceMapper.insert(taskInstance);
- return count > 0;
- }
-
- /**
- * update task instance
- *
- * @param taskInstance taskInstance
- * @return update task instance result
- */
- public boolean updateTaskInstance(TaskInstance taskInstance) {
- int count = taskInstanceMapper.updateById(taskInstance);
- return count > 0;
- }
-
- /**
- * find task instance by id
- *
- * @param taskId task id
- * @return task instance
- */
- public TaskInstance findTaskInstanceById(Integer taskId) {
- return taskInstanceMapper.selectById(taskId);
- }
-
- /**
- * find task instance list by id list
- *
- * @param idList task id list
- * @return task instance list
- */
- public List<TaskInstance> findTaskInstanceByIdList(List<Integer> idList) {
- if (CollectionUtils.isEmpty(idList)) {
- return new ArrayList<>();
- }
- return taskInstanceMapper.selectBatchIds(idList);
- }
-
- /**
- * package task instance
- */
- public void packageTaskInstance(TaskInstance taskInstance, ProcessInstance processInstance) {
- taskInstance.setProcessInstance(processInstance);
- taskInstance.setProcessDefine(processInstance.getProcessDefinition());
- TaskDefinition taskDefinition = this.findTaskDefinition(
- taskInstance.getTaskCode(),
- taskInstance.getTaskDefinitionVersion());
- this.updateTaskDefinitionResources(taskDefinition);
- taskInstance.setTaskDefine(taskDefinition);
- }
-
- /**
- * Update {@link ResourceInfo} information in {@link TaskDefinition}
- *
- * @param taskDefinition the given {@link TaskDefinition}
- */
- public void updateTaskDefinitionResources(TaskDefinition taskDefinition) {
- Map<String, Object> taskParameters = JSONUtils.parseObject(
- taskDefinition.getTaskParams(),
- new TypeReference<Map<String, Object>>() {
- });
- if (taskParameters != null) {
- // if contains mainJar field, query resource from database
- // Flink, Spark, MR
- if (taskParameters.containsKey("mainJar")) {
- Object mainJarObj = taskParameters.get("mainJar");
- ResourceInfo mainJar = JSONUtils.parseObject(
- JSONUtils.toJsonString(mainJarObj),
- ResourceInfo.class);
- ResourceInfo resourceInfo = updateResourceInfo(mainJar);
- if (resourceInfo != null) {
- taskParameters.put("mainJar", resourceInfo);
- }
- }
- // update resourceList information
- if (taskParameters.containsKey("resourceList")) {
- String resourceListStr = JSONUtils.toJsonString(taskParameters.get("resourceList"));
- List<ResourceInfo> resourceInfos = JSONUtils.toList(resourceListStr, ResourceInfo.class);
- List<ResourceInfo> updatedResourceInfos = resourceInfos
- .stream()
- .map(this::updateResourceInfo)
- .filter(Objects::nonNull)
- .collect(Collectors.toList());
- taskParameters.put("resourceList", updatedResourceInfos);
- }
- // set task parameters
- taskDefinition.setTaskParams(JSONUtils.toJsonString(taskParameters));
- }
- }
-
- /**
- * update {@link ResourceInfo} by given original ResourceInfo
- *
- * @param res origin resource info
- * @return {@link ResourceInfo}
- */
- private ResourceInfo updateResourceInfo(ResourceInfo res) {
- ResourceInfo resourceInfo = null;
- // only if mainJar is not null and does not contains "resourceName" field
- if (res != null) {
- int resourceId = res.getId();
- if (resourceId <= 0) {
- logger.error("invalid resourceId, {}", resourceId);
- return null;
- }
- resourceInfo = new ResourceInfo();
- // get resource from database, only one resource should be returned
- Resource resource = getResourceById(resourceId);
- resourceInfo.setId(resourceId);
- resourceInfo.setRes(resource.getFileName());
- resourceInfo.setResourceName(resource.getFullName());
- if (logger.isInfoEnabled()) {
- logger.info("updated resource info {}",
- JSONUtils.toJsonString(resourceInfo));
- }
- }
- return resourceInfo;
- }
-
- /**
- * get id list by task state
- *
- * @param instanceId instanceId
- * @param state state
- * @return task instance states
- */
- public List<Integer> findTaskIdByInstanceState(int instanceId, ExecutionStatus state) {
- return taskInstanceMapper.queryTaskByProcessIdAndState(instanceId, state.ordinal());
- }
-
- /**
- * find valid task list by process definition id
- *
- * @param processInstanceId processInstanceId
- * @return task instance list
- */
- public List<TaskInstance> findValidTaskListByProcessId(Integer processInstanceId) {
- return taskInstanceMapper.findValidTaskListByProcessId(processInstanceId, Flag.YES);
- }
-
- /**
- * find previous task list by work process id
- *
- * @param processInstanceId processInstanceId
- * @return task instance list
- */
- public List<TaskInstance> findPreviousTaskListByWorkProcessId(Integer processInstanceId) {
- return taskInstanceMapper.findValidTaskListByProcessId(processInstanceId, Flag.NO);
- }
-
- /**
- * update work process instance map
- *
- * @param processInstanceMap processInstanceMap
- * @return update process instance result
- */
- public int updateWorkProcessInstanceMap(ProcessInstanceMap processInstanceMap) {
- return processInstanceMapMapper.updateById(processInstanceMap);
- }
-
- /**
- * create work process instance map
- *
- * @param processInstanceMap processInstanceMap
- * @return create process instance result
- */
- public int createWorkProcessInstanceMap(ProcessInstanceMap processInstanceMap) {
- int count = 0;
- if (processInstanceMap != null) {
- return processInstanceMapMapper.insert(processInstanceMap);
- }
- return count;
- }
-
- /**
- * find work process map by parent process id and parent task id.
- *
- * @param parentWorkProcessId parentWorkProcessId
- * @param parentTaskId parentTaskId
- * @return process instance map
- */
- public ProcessInstanceMap findWorkProcessMapByParent(Integer parentWorkProcessId, Integer parentTaskId) {
- return processInstanceMapMapper.queryByParentId(parentWorkProcessId, parentTaskId);
- }
-
- /**
- * delete work process map by parent process id
- *
- * @param parentWorkProcessId parentWorkProcessId
- * @return delete process map result
- */
- public int deleteWorkProcessMapByParentId(int parentWorkProcessId) {
- return processInstanceMapMapper.deleteByParentProcessId(parentWorkProcessId);
-
- }
-
- /**
- * find sub process instance
- *
- * @param parentProcessId parentProcessId
- * @param parentTaskId parentTaskId
- * @return process instance
- */
- public ProcessInstance findSubProcessInstance(Integer parentProcessId, Integer parentTaskId) {
- ProcessInstance processInstance = null;
- ProcessInstanceMap processInstanceMap = processInstanceMapMapper.queryByParentId(parentProcessId, parentTaskId);
- if (processInstanceMap == null || processInstanceMap.getProcessInstanceId() == 0) {
- return processInstance;
- }
- processInstance = findProcessInstanceById(processInstanceMap.getProcessInstanceId());
- return processInstance;
- }
-
- /**
- * find parent process instance
- *
- * @param subProcessId subProcessId
- * @return process instance
- */
- public ProcessInstance findParentProcessInstance(Integer subProcessId) {
- ProcessInstance processInstance = null;
- ProcessInstanceMap processInstanceMap = processInstanceMapMapper.queryBySubProcessId(subProcessId);
- if (processInstanceMap == null || processInstanceMap.getProcessInstanceId() == 0) {
- return processInstance;
- }
- processInstance = findProcessInstanceById(processInstanceMap.getParentProcessInstanceId());
- return processInstance;
- }
-
- /**
- * update process instance
- *
- * @param processInstance processInstance
- * @return update process instance result
- */
- public int updateProcessInstance(ProcessInstance processInstance) {
- return processInstanceMapper.updateById(processInstance);
- }
-
- /**
- * for show in page of taskInstance
- */
- public void changeOutParam(TaskInstance taskInstance) {
- if (StringUtils.isEmpty(taskInstance.getVarPool())) {
- return;
- }
- List<Property> properties = JSONUtils.toList(taskInstance.getVarPool(), Property.class);
- if (CollectionUtils.isEmpty(properties)) {
- return;
- }
- //if the result more than one line,just get the first .
- Map<String, Object> taskParams = JSONUtils.parseObject(taskInstance.getTaskParams(), new TypeReference<Map<String, Object>>() {
- });
- Object localParams = taskParams.get(LOCAL_PARAMS);
- if (localParams == null) {
- return;
- }
- List<Property> allParam = JSONUtils.toList(JSONUtils.toJsonString(localParams), Property.class);
- Map<String, String> outProperty = new HashMap<>();
- for (Property info : properties) {
- if (info.getDirect() == Direct.OUT) {
- outProperty.put(info.getProp(), info.getValue());
- }
- }
- for (Property info : allParam) {
- if (info.getDirect() == Direct.OUT) {
- String paramName = info.getProp();
- info.setValue(outProperty.get(paramName));
- }
- }
- taskParams.put(LOCAL_PARAMS, allParam);
- taskInstance.setTaskParams(JSONUtils.toJsonString(taskParams));
- }
-
- /**
- * convert integer list to string list
- *
- * @param intList intList
- * @return string list
- */
- public List<String> convertIntListToString(List<Integer> intList) {
- if (intList == null) {
- return new ArrayList<>();
- }
- List<String> result = new ArrayList<>(intList.size());
- for (Integer intVar : intList) {
- result.add(String.valueOf(intVar));
- }
- return result;
- }
-
- /**
- * query schedule by id
- *
- * @param id id
- * @return schedule
- */
- public Schedule querySchedule(int id) {
- return scheduleMapper.selectById(id);
- }
-
- /**
- * query Schedule by processDefinitionCode
- *
- * @param processDefinitionCode processDefinitionCode
- * @see Schedule
- */
- public List<Schedule> queryReleaseSchedulerListByProcessDefinitionCode(long processDefinitionCode) {
- return scheduleMapper.queryReleaseSchedulerListByProcessDefinitionCode(processDefinitionCode);
- }
-
- /**
- * query Schedule by processDefinitionCode
- *
- * @param processDefinitionCodeList processDefinitionCodeList
- * @see Schedule
- */
- public Map<Long, String> queryWorkerGroupByProcessDefinitionCodes(List<Long> processDefinitionCodeList) {
- List<Schedule> processDefinitionScheduleList = scheduleMapper.querySchedulesByProcessDefinitionCodes(processDefinitionCodeList);
- return processDefinitionScheduleList.stream().collect(Collectors.toMap(Schedule::getProcessDefinitionCode,
- Schedule::getWorkerGroup));
- }
-
- /**
- * query dependent process definition by process definition code
- *
- * @param processDefinitionCode processDefinitionCode
- * @see DependentProcessDefinition
- */
- public List<DependentProcessDefinition> queryDependentProcessDefinitionByProcessDefinitionCode(long processDefinitionCode) {
- return workFlowLineageMapper.queryDependentProcessDefinitionByProcessDefinitionCode(processDefinitionCode);
- }
-
- /**
- * query need failover process instance
- *
- * @param host host
- * @return process instance list
- */
- public List<ProcessInstance> queryNeedFailoverProcessInstances(String host) {
- return processInstanceMapper.queryByHostAndStatus(host, stateArray);
- }
-
- public List<String> queryNeedFailoverProcessInstanceHost() {
- return processInstanceMapper.queryNeedFailoverProcessInstanceHost(stateArray);
- }
-
- /**
- * process need failover process instance
- *
- * @param processInstance processInstance
- */
@Transactional(rollbackFor = RuntimeException.class)
- public void processNeedFailoverProcessInstances(ProcessInstance processInstance) {
- //1 update processInstance host is null
- processInstance.setHost(Constants.NULL);
- processInstanceMapper.updateById(processInstance);
-
- ProcessDefinition processDefinition = findProcessDefinition(processInstance.getProcessDefinitionCode(), processInstance.getProcessDefinitionVersion());
-
- //2 insert into recover command
- Command cmd = new Command();
- cmd.setProcessDefinitionCode(processDefinition.getCode());
- cmd.setProcessDefinitionVersion(processDefinition.getVersion());
- cmd.setProcessInstanceId(processInstance.getId());
- cmd.setCommandParam(String.format("{\"%s\":%d}", Constants.CMD_PARAM_RECOVER_PROCESS_ID_STRING, processInstance.getId()));
- cmd.setExecutorId(processInstance.getExecutorId());
- cmd.setCommandType(CommandType.RECOVER_TOLERANCE_FAULT_PROCESS);
- createCommand(cmd);
- }
-
- /**
- * query all need failover task instances by host
- *
- * @param host host
- * @return task instance list
- */
- public List<TaskInstance> queryNeedFailoverTaskInstances(String host) {
- return taskInstanceMapper.queryByHostAndStatus(host,
- stateArray);
- }
-
- /**
- * find data source by id
- *
- * @param id id
- * @return datasource
- */
- public DataSource findDataSourceById(int id) {
- return dataSourceMapper.selectById(id);
- }
-
- /**
- * update process instance state by id
- *
- * @param processInstanceId processInstanceId
- * @param executionStatus executionStatus
- * @return update process result
- */
- public int updateProcessInstanceState(Integer processInstanceId, ExecutionStatus executionStatus) {
- ProcessInstance instance = processInstanceMapper.selectById(processInstanceId);
- instance.setState(executionStatus);
- return processInstanceMapper.updateById(instance);
- }
-
- /**
- * find process instance by the task id
- *
- * @param taskId taskId
- * @return process instance
- */
- public ProcessInstance findProcessInstanceByTaskId(int taskId) {
- TaskInstance taskInstance = taskInstanceMapper.selectById(taskId);
- if (taskInstance != null) {
- return processInstanceMapper.selectById(taskInstance.getProcessInstanceId());
- }
- return null;
- }
-
- /**
- * find udf function list by id list string
- *
- * @param ids ids
- * @return udf function list
- */
- public List<UdfFunc> queryUdfFunListByIds(Integer[] ids) {
- return udfFuncMapper.queryUdfByIdStr(ids, null);
- }
-
- /**
- * find tenant code by resource name
- *
- * @param resName resource name
- * @param resourceType resource type
- * @return tenant code
- */
- public String queryTenantCodeByResName(String resName, ResourceType resourceType) {
- // in order to query tenant code successful although the version is older
- String fullName = resName.startsWith("/") ? resName : String.format("/%s", resName);
-
- List<Resource> resourceList = resourceMapper.queryResource(fullName, resourceType.ordinal());
- if (CollectionUtils.isEmpty(resourceList)) {
- return StringUtils.EMPTY;
- }
- int userId = resourceList.get(0).getUserId();
- User user = userMapper.selectById(userId);
- if (Objects.isNull(user)) {
- return StringUtils.EMPTY;
- }
- Tenant tenant = tenantMapper.queryById(user.getTenantId());
- if (Objects.isNull(tenant)) {
- return StringUtils.EMPTY;
- }
- return tenant.getTenantCode();
- }
-
- /**
- * find schedule list by process define codes.
- *
- * @param codes codes
- * @return schedule list
- */
- public List<Schedule> selectAllByProcessDefineCode(long[] codes) {
- return scheduleMapper.selectAllByProcessDefineArray(codes);
- }
-
- /**
- * find last scheduler process instance in the date interval
- *
- * @param definitionCode definitionCode
- * @param dateInterval dateInterval
- * @return process instance
- */
- public ProcessInstance findLastSchedulerProcessInterval(Long definitionCode, DateInterval dateInterval) {
- return processInstanceMapper.queryLastSchedulerProcess(definitionCode,
- dateInterval.getStartTime(),
- dateInterval.getEndTime());
- }
-
- /**
- * find last manual process instance interval
- *
- * @param definitionCode process definition code
- * @param dateInterval dateInterval
- * @return process instance
- */
- public ProcessInstance findLastManualProcessInterval(Long definitionCode, DateInterval dateInterval) {
- return processInstanceMapper.queryLastManualProcess(definitionCode,
- dateInterval.getStartTime(),
- dateInterval.getEndTime());
- }
-
- /**
- * find last running process instance
- *
- * @param definitionCode process definition code
- * @param startTime start time
- * @param endTime end time
- * @return process instance
- */
- public ProcessInstance findLastRunningProcess(Long definitionCode, Date startTime, Date endTime) {
- return processInstanceMapper.queryLastRunningProcess(definitionCode,
- startTime,
- endTime,
- stateArray);
- }
-
- /**
- * query user queue by process instance
- *
- * @param processInstance processInstance
- * @return queue
- */
- public String queryUserQueueByProcessInstance(ProcessInstance processInstance) {
-
- String queue = "";
- if (processInstance == null) {
- return queue;
- }
- User executor = userMapper.selectById(processInstance.getExecutorId());
- if (executor != null) {
- queue = executor.getQueue();
- }
- return queue;
- }
-
- /**
- * query project name and user name by processInstanceId.
- *
- * @param processInstanceId processInstanceId
- * @return projectName and userName
- */
- public ProjectUser queryProjectWithUserByProcessInstanceId(int processInstanceId) {
- return projectMapper.queryProjectWithUserByProcessInstanceId(processInstanceId);
- }
-
- /**
- * get task worker group
- *
- * @param taskInstance taskInstance
- * @return workerGroupId
- */
- public String getTaskWorkerGroup(TaskInstance taskInstance) {
- String workerGroup = taskInstance.getWorkerGroup();
-
- if (StringUtils.isNotBlank(workerGroup)) {
- return workerGroup;
- }
- int processInstanceId = taskInstance.getProcessInstanceId();
- ProcessInstance processInstance = findProcessInstanceById(processInstanceId);
-
- if (processInstance != null) {
- return processInstance.getWorkerGroup();
- }
- logger.info("task : {} will use default worker group", taskInstance.getId());
- return Constants.DEFAULT_WORKER_GROUP;
- }
-
- /**
- * get have perm project list
- *
- * @param userId userId
- * @return project list
- */
- public List<Project> getProjectListHavePerm(int userId) {
- List<Project> createProjects = projectMapper.queryProjectCreatedByUser(userId);
- List<Project> authedProjects = projectMapper.queryAuthedProjectListByUserId(userId);
-
- if (createProjects == null) {
- createProjects = new ArrayList<>();
- }
-
- if (authedProjects != null) {
- createProjects.addAll(authedProjects);
- }
- return createProjects;
- }
-
- /**
- * list unauthorized udf function
- *
- * @param userId user id
- * @param needChecks data source id array
- * @return unauthorized udf function list
- */
- public <T> List<T> listUnauthorized(int userId, T[] needChecks, AuthorizationType authorizationType) {
- List<T> resultList = new ArrayList<>();
-
- if (Objects.nonNull(needChecks) && needChecks.length > 0) {
- Set<T> originResSet = new HashSet<>(Arrays.asList(needChecks));
-
- switch (authorizationType) {
- case RESOURCE_FILE_ID:
- case UDF_FILE:
- List<Resource> ownUdfResources = resourceMapper.listAuthorizedResourceById(userId, needChecks);
- addAuthorizedResources(ownUdfResources, userId);
- Set<Integer> authorizedResourceFiles = ownUdfResources.stream().map(Resource::getId).collect(toSet());
- originResSet.removeAll(authorizedResourceFiles);
- break;
- case RESOURCE_FILE_NAME:
- List<Resource> ownResources = resourceMapper.listAuthorizedResource(userId, needChecks);
- addAuthorizedResources(ownResources, userId);
- Set<String> authorizedResources = ownResources.stream().map(Resource::getFullName).collect(toSet());
- originResSet.removeAll(authorizedResources);
- break;
- case DATASOURCE:
- Set<Integer> authorizedDatasources = dataSourceMapper.listAuthorizedDataSource(userId, needChecks).stream().map(DataSource::getId).collect(toSet());
- originResSet.removeAll(authorizedDatasources);
- break;
- case UDF:
- Set<Integer> authorizedUdfs = udfFuncMapper.listAuthorizedUdfFunc(userId, needChecks).stream().map(UdfFunc::getId).collect(toSet());
- originResSet.removeAll(authorizedUdfs);
- break;
- default:
- break;
- }
-
- resultList.addAll(originResSet);
- }
-
- return resultList;
- }
-
- /**
- * get user by user id
- *
- * @param userId user id
- * @return User
- */
- public User getUserById(int userId) {
- return userMapper.selectById(userId);
- }
-
- /**
- * get resource by resource id
- *
- * @param resourceId resource id
- * @return Resource
- */
- public Resource getResourceById(int resourceId) {
- return resourceMapper.selectById(resourceId);
- }
-
- /**
- * list resources by ids
- *
- * @param resIds resIds
- * @return resource list
- */
- public List<Resource> listResourceByIds(Integer[] resIds) {
- return resourceMapper.listResourceByIds(resIds);
- }
-
- /**
- * format task app id in task instance
- */
- public String formatTaskAppId(TaskInstance taskInstance) {
- ProcessInstance processInstance = findProcessInstanceById(taskInstance.getProcessInstanceId());
- if (processInstance == null) {
- return "";
- }
- ProcessDefinition definition = findProcessDefinition(processInstance.getProcessDefinitionCode(), processInstance.getProcessDefinitionVersion());
- if (definition == null) {
- return "";
- }
- return String.format("%s_%s_%s", definition.getId(), processInstance.getId(), taskInstance.getId());
- }
-
- /**
- * switch process definition version to process definition log version
- */
- public int switchVersion(ProcessDefinition processDefinition, ProcessDefinitionLog processDefinitionLog) {
- if (null == processDefinition || null == processDefinitionLog) {
- return Constants.DEFINITION_FAILURE;
- }
- processDefinitionLog.setId(processDefinition.getId());
- processDefinitionLog.setReleaseState(ReleaseState.OFFLINE);
- processDefinitionLog.setFlag(Flag.YES);
-
- int result = processDefineMapper.updateById(processDefinitionLog);
- if (result > 0) {
- result = switchProcessTaskRelationVersion(processDefinitionLog);
- if (result <= 0) {
- return Constants.EXIT_CODE_FAILURE;
- }
- }
- return result;
- }
-
- public int switchProcessTaskRelationVersion(ProcessDefinition processDefinition) {
- List<ProcessTaskRelation> processTaskRelationList = processTaskRelationMapper.queryByProcessCode(processDefinition.getProjectCode(), processDefinition.getCode());
- if (!processTaskRelationList.isEmpty()) {
- processTaskRelationMapper.deleteByCode(processDefinition.getProjectCode(), processDefinition.getCode());
- }
- List<ProcessTaskRelationLog> processTaskRelationLogList = processTaskRelationLogMapper.queryByProcessCodeAndVersion(processDefinition.getCode(), processDefinition.getVersion());
- int batchInsert = processTaskRelationMapper.batchInsert(processTaskRelationLogList);
- if (batchInsert == 0) {
- return Constants.EXIT_CODE_FAILURE;
- } else {
- int result = 0;
- for (ProcessTaskRelationLog taskRelationLog : processTaskRelationLogList) {
- int switchResult = switchTaskDefinitionVersion(taskRelationLog.getPostTaskCode(), taskRelationLog.getPostTaskVersion());
- if (switchResult != Constants.EXIT_CODE_FAILURE) {
- result++;
- }
- }
- return result;
- }
- }
-
- public int switchTaskDefinitionVersion(long taskCode, int taskVersion) {
- TaskDefinition taskDefinition = taskDefinitionMapper.queryByCode(taskCode);
- if (taskDefinition == null) {
- return Constants.EXIT_CODE_FAILURE;
- }
- if (taskDefinition.getVersion() == taskVersion) {
- return Constants.EXIT_CODE_SUCCESS;
- }
- TaskDefinitionLog taskDefinitionUpdate = taskDefinitionLogMapper.queryByDefinitionCodeAndVersion(taskCode, taskVersion);
- if (taskDefinitionUpdate == null) {
- return Constants.EXIT_CODE_FAILURE;
- }
- taskDefinitionUpdate.setUpdateTime(new Date());
- taskDefinitionUpdate.setId(taskDefinition.getId());
- return taskDefinitionMapper.updateById(taskDefinitionUpdate);
- }
-
- /**
- * get resource ids
- *
- * @param taskDefinition taskDefinition
- * @return resource ids
- */
- public String getResourceIds(TaskDefinition taskDefinition) {
- Set<Integer> resourceIds = null;
- AbstractParameters params = taskPluginManager.getParameters(ParametersNode.builder().taskType(taskDefinition.getTaskType()).taskParams(taskDefinition.getTaskParams()).build());
-
- if (params != null && CollectionUtils.isNotEmpty(params.getResourceFilesList())) {
- resourceIds = params.getResourceFilesList().
- stream()
- .filter(t -> t.getId() != 0)
- .map(ResourceInfo::getId)
- .collect(Collectors.toSet());
- }
- if (CollectionUtils.isEmpty(resourceIds)) {
- return StringUtils.EMPTY;
- }
- return StringUtils.join(resourceIds, ",");
- }
-
- public int saveTaskDefine(User operator, long projectCode, List<TaskDefinitionLog> taskDefinitionLogs, Boolean syncDefine) {
- Date now = new Date();
- List<TaskDefinitionLog> newTaskDefinitionLogs = new ArrayList<>();
- List<TaskDefinitionLog> updateTaskDefinitionLogs = new ArrayList<>();
- for (TaskDefinitionLog taskDefinitionLog : taskDefinitionLogs) {
- taskDefinitionLog.setProjectCode(projectCode);
- taskDefinitionLog.setUpdateTime(now);
- taskDefinitionLog.setOperateTime(now);
- taskDefinitionLog.setOperator(operator.getId());
- taskDefinitionLog.setResourceIds(getResourceIds(taskDefinitionLog));
- if (taskDefinitionLog.getCode() == 0) {
- try {
- taskDefinitionLog.setCode(CodeGenerateUtils.getInstance().genCode());
- } catch (CodeGenerateException e) {
- logger.error("Task code get error, ", e);
- return Constants.DEFINITION_FAILURE;
- }
- }
- if (taskDefinitionLog.getVersion() == 0) {
- // init first version
- taskDefinitionLog.setVersion(Constants.VERSION_FIRST);
- }
-
- TaskDefinitionLog definitionCodeAndVersion = taskDefinitionLogMapper
- .queryByDefinitionCodeAndVersion(taskDefinitionLog.getCode(), taskDefinitionLog.getVersion());
- if (definitionCodeAndVersion == null) {
- taskDefinitionLog.setUserId(operator.getId());
- taskDefinitionLog.setCreateTime(now);
- newTaskDefinitionLogs.add(taskDefinitionLog);
- continue;
- }
- if (taskDefinitionLog.equals(definitionCodeAndVersion)) {
- // do nothing if equals
- continue;
- }
- taskDefinitionLog.setUserId(definitionCodeAndVersion.getUserId());
- Integer version = taskDefinitionLogMapper.queryMaxVersionForDefinition(taskDefinitionLog.getCode());
- taskDefinitionLog.setVersion(version + 1);
- taskDefinitionLog.setCreateTime(definitionCodeAndVersion.getCreateTime());
- updateTaskDefinitionLogs.add(taskDefinitionLog);
- }
- int insertResult = 0;
- int updateResult = 0;
- for (TaskDefinitionLog taskDefinitionToUpdate : updateTaskDefinitionLogs) {
- TaskDefinition task = taskDefinitionMapper.queryByCode(taskDefinitionToUpdate.getCode());
- if (task == null) {
- newTaskDefinitionLogs.add(taskDefinitionToUpdate);
- } else {
- insertResult += taskDefinitionLogMapper.insert(taskDefinitionToUpdate);
- if (Boolean.TRUE.equals(syncDefine)) {
- taskDefinitionToUpdate.setId(task.getId());
- updateResult += taskDefinitionMapper.updateById(taskDefinitionToUpdate);
- } else {
- updateResult++;
- }
- }
- }
- if (!newTaskDefinitionLogs.isEmpty()) {
- insertResult += taskDefinitionLogMapper.batchInsert(newTaskDefinitionLogs);
- if (Boolean.TRUE.equals(syncDefine)) {
- updateResult += taskDefinitionMapper.batchInsert(newTaskDefinitionLogs);
- } else {
- updateResult += newTaskDefinitionLogs.size();
- }
- }
- return (insertResult & updateResult) > 0 ? 1 : Constants.EXIT_CODE_SUCCESS;
- }
-
- /**
- * save processDefinition (including create or update processDefinition)
- */
- public int saveProcessDefine(User operator, ProcessDefinition processDefinition, Boolean syncDefine, Boolean isFromProcessDefine) {
- ProcessDefinitionLog processDefinitionLog = new ProcessDefinitionLog(processDefinition);
- Integer version = processDefineLogMapper.queryMaxVersionForDefinition(processDefinition.getCode());
- int insertVersion = version == null || version == 0 ? Constants.VERSION_FIRST : version + 1;
- processDefinitionLog.setVersion(insertVersion);
- processDefinitionLog.setReleaseState(!isFromProcessDefine || processDefinitionLog.getReleaseState() == ReleaseState.ONLINE ? ReleaseState.ONLINE : ReleaseState.OFFLINE);
- processDefinitionLog.setOperator(operator.getId());
- processDefinitionLog.setOperateTime(processDefinition.getUpdateTime());
- int insertLog = processDefineLogMapper.insert(processDefinitionLog);
- int result = 1;
- if (Boolean.TRUE.equals(syncDefine)) {
- if (0 == processDefinition.getId()) {
- result = processDefineMapper.insert(processDefinitionLog);
- } else {
- processDefinitionLog.setId(processDefinition.getId());
- result = processDefineMapper.updateById(processDefinitionLog);
- }
- }
- return (insertLog & result) > 0 ? insertVersion : 0;
- }
-
- /**
- * save task relations
- */
- public int saveTaskRelation(User operator, long projectCode, long processDefinitionCode, int processDefinitionVersion,
- List<ProcessTaskRelationLog> taskRelationList, List<TaskDefinitionLog> taskDefinitionLogs,
- Boolean syncDefine) {
- if (taskRelationList.isEmpty()) {
- return Constants.EXIT_CODE_SUCCESS;
- }
- Map<Long, TaskDefinitionLog> taskDefinitionLogMap = null;
- if (CollectionUtils.isNotEmpty(taskDefinitionLogs)) {
- taskDefinitionLogMap = taskDefinitionLogs.stream()
- .collect(Collectors.toMap(TaskDefinition::getCode, taskDefinitionLog -> taskDefinitionLog));
- }
- Date now = new Date();
- for (ProcessTaskRelationLog processTaskRelationLog : taskRelationList) {
- processTaskRelationLog.setProjectCode(projectCode);
- processTaskRelationLog.setProcessDefinitionCode(processDefinitionCode);
- processTaskRelationLog.setProcessDefinitionVersion(processDefinitionVersion);
- if (taskDefinitionLogMap != null) {
- TaskDefinitionLog preTaskDefinitionLog = taskDefinitionLogMap.get(processTaskRelationLog.getPreTaskCode());
- if (preTaskDefinitionLog != null) {
- processTaskRelationLog.setPreTaskVersion(preTaskDefinitionLog.getVersion());
- }
- TaskDefinitionLog postTaskDefinitionLog = taskDefinitionLogMap.get(processTaskRelationLog.getPostTaskCode());
- if (postTaskDefinitionLog != null) {
- processTaskRelationLog.setPostTaskVersion(postTaskDefinitionLog.getVersion());
- }
- }
- processTaskRelationLog.setCreateTime(now);
- processTaskRelationLog.setUpdateTime(now);
- processTaskRelationLog.setOperator(operator.getId());
- processTaskRelationLog.setOperateTime(now);
- }
- int insert = taskRelationList.size();
- if (Boolean.TRUE.equals(syncDefine)) {
- List<ProcessTaskRelation> processTaskRelationList = processTaskRelationMapper.queryByProcessCode(projectCode, processDefinitionCode);
- if (!processTaskRelationList.isEmpty()) {
- Set<Integer> processTaskRelationSet = processTaskRelationList.stream().map(ProcessTaskRelation::hashCode).collect(toSet());
- Set<Integer> taskRelationSet = taskRelationList.stream().map(ProcessTaskRelationLog::hashCode).collect(toSet());
- boolean result = CollectionUtils.isEqualCollection(processTaskRelationSet, taskRelationSet);
- if (result) {
- return Constants.EXIT_CODE_SUCCESS;
- }
- processTaskRelationMapper.deleteByCode(projectCode, processDefinitionCode);
- }
- insert = processTaskRelationMapper.batchInsert(taskRelationList);
- }
- int resultLog = processTaskRelationLogMapper.batchInsert(taskRelationList);
- return (insert & resultLog) > 0 ? Constants.EXIT_CODE_SUCCESS : Constants.EXIT_CODE_FAILURE;
- }
-
- public boolean isTaskOnline(long taskCode) {
- List<ProcessTaskRelation> processTaskRelationList = processTaskRelationMapper.queryByTaskCode(taskCode);
- if (!processTaskRelationList.isEmpty()) {
- Set<Long> processDefinitionCodes = processTaskRelationList
- .stream()
- .map(ProcessTaskRelation::getProcessDefinitionCode)
- .collect(Collectors.toSet());
- List<ProcessDefinition> processDefinitionList = processDefineMapper.queryByCodes(processDefinitionCodes);
- // check process definition is already online
- for (ProcessDefinition processDefinition : processDefinitionList) {
- if (processDefinition.getReleaseState() == ReleaseState.ONLINE) {
- return true;
- }
- }
- }
- return false;
- }
-
- /**
- * Generate the DAG Graph based on the process definition id
- * Use temporarily before refactoring taskNode
- *
- * @param processDefinition process definition
- * @return dag graph
- */
- public DAG<String, TaskNode, TaskNodeRelation> genDagGraph(ProcessDefinition processDefinition) {
- List<ProcessTaskRelation> taskRelations = this.findRelationByCode(processDefinition.getCode(), processDefinition.getVersion());
- List<TaskNode> taskNodeList = transformTask(taskRelations, Lists.newArrayList());
- ProcessDag processDag = DagHelper.getProcessDag(taskNodeList, new ArrayList<>(taskRelations));
- // Generate concrete Dag to be executed
- return DagHelper.buildDagGraph(processDag);
- }
-
- /**
- * generate DagData
- */
- public DagData genDagData(ProcessDefinition processDefinition) {
- List<ProcessTaskRelation> taskRelations = this.findRelationByCode(processDefinition.getCode(), processDefinition.getVersion());
- List<TaskDefinitionLog> taskDefinitionLogList = genTaskDefineList(taskRelations);
- List<TaskDefinition> taskDefinitions = taskDefinitionLogList.stream().map(t -> (TaskDefinition) t).collect(Collectors.toList());
- return new DagData(processDefinition, taskRelations, taskDefinitions);
- }
-
- public List<TaskDefinitionLog> genTaskDefineList(List<ProcessTaskRelation> processTaskRelations) {
- Set<TaskDefinition> taskDefinitionSet = new HashSet<>();
- for (ProcessTaskRelation processTaskRelation : processTaskRelations) {
- if (processTaskRelation.getPreTaskCode() > 0) {
- taskDefinitionSet.add(new TaskDefinition(processTaskRelation.getPreTaskCode(), processTaskRelation.getPreTaskVersion()));
- }
- if (processTaskRelation.getPostTaskCode() > 0) {
- taskDefinitionSet.add(new TaskDefinition(processTaskRelation.getPostTaskCode(), processTaskRelation.getPostTaskVersion()));
- }
- }
- if (taskDefinitionSet.isEmpty()) {
- return Lists.newArrayList();
- }
- return taskDefinitionLogMapper.queryByTaskDefinitions(taskDefinitionSet);
- }
-
- public List<TaskDefinitionLog> getTaskDefineLogListByRelation(List<ProcessTaskRelation> processTaskRelations) {
- List<TaskDefinitionLog> taskDefinitionLogs = new ArrayList<>();
- Map<Long, Integer> taskCodeVersionMap = new HashMap<>();
- for (ProcessTaskRelation processTaskRelation : processTaskRelations) {
- if (processTaskRelation.getPreTaskCode() > 0) {
- taskCodeVersionMap.put(processTaskRelation.getPreTaskCode(), processTaskRelation.getPreTaskVersion());
- }
- if (processTaskRelation.getPostTaskCode() > 0) {
- taskCodeVersionMap.put(processTaskRelation.getPostTaskCode(), processTaskRelation.getPostTaskVersion());
- }
- }
- taskCodeVersionMap.forEach((code, version) -> {
- taskDefinitionLogs.add((TaskDefinitionLog) this.findTaskDefinition(code, version));
- });
- return taskDefinitionLogs;
- }
-
- /**
- * find task definition by code and version
- */
- public TaskDefinition findTaskDefinition(long taskCode, int taskDefinitionVersion) {
- return taskDefinitionLogMapper.queryByDefinitionCodeAndVersion(taskCode, taskDefinitionVersion);
- }
-
- /**
- * find process task relation list by process
- */
- public List<ProcessTaskRelation> findRelationByCode(long processDefinitionCode, int processDefinitionVersion) {
- List<ProcessTaskRelationLog> processTaskRelationLogList = processTaskRelationLogMapper.queryByProcessCodeAndVersion(processDefinitionCode, processDefinitionVersion);
- return processTaskRelationLogList.stream().map(r -> (ProcessTaskRelation) r).collect(Collectors.toList());
- }
-
- /**
- * add authorized resources
- *
- * @param ownResources own resources
- * @param userId userId
- */
- private void addAuthorizedResources(List<Resource> ownResources, int userId) {
- List<Integer> relationResourceIds = resourceUserMapper.queryResourcesIdListByUserIdAndPerm(userId, 7);
- List<Resource> relationResources = CollectionUtils.isNotEmpty(relationResourceIds) ? resourceMapper.queryResourceListById(relationResourceIds) : new ArrayList<>();
- ownResources.addAll(relationResources);
- }
-
- /**
- * Use temporarily before refactoring taskNode
- */
- public List<TaskNode> transformTask(List<ProcessTaskRelation> taskRelationList, List<TaskDefinitionLog> taskDefinitionLogs) {
- Map<Long, List<Long>> taskCodeMap = new HashMap<>();
- for (ProcessTaskRelation processTaskRelation : taskRelationList) {
- taskCodeMap.compute(processTaskRelation.getPostTaskCode(), (k, v) -> {
- if (v == null) {
- v = new ArrayList<>();
- }
- if (processTaskRelation.getPreTaskCode() != 0L) {
- v.add(processTaskRelation.getPreTaskCode());
- }
- return v;
- });
- }
- if (CollectionUtils.isEmpty(taskDefinitionLogs)) {
- taskDefinitionLogs = genTaskDefineList(taskRelationList);
- }
- Map<Long, TaskDefinitionLog> taskDefinitionLogMap = taskDefinitionLogs.stream()
- .collect(Collectors.toMap(TaskDefinitionLog::getCode, taskDefinitionLog -> taskDefinitionLog));
- List<TaskNode> taskNodeList = new ArrayList<>();
- for (Entry<Long, List<Long>> code : taskCodeMap.entrySet()) {
- TaskDefinitionLog taskDefinitionLog = taskDefinitionLogMap.get(code.getKey());
- if (taskDefinitionLog != null) {
- TaskNode taskNode = new TaskNode();
- taskNode.setCode(taskDefinitionLog.getCode());
- taskNode.setVersion(taskDefinitionLog.getVersion());
- taskNode.setName(taskDefinitionLog.getName());
- taskNode.setDesc(taskDefinitionLog.getDescription());
- taskNode.setType(taskDefinitionLog.getTaskType().toUpperCase());
- taskNode.setRunFlag(taskDefinitionLog.getFlag() == Flag.YES ? Constants.FLOWNODE_RUN_FLAG_NORMAL : Constants.FLOWNODE_RUN_FLAG_FORBIDDEN);
- taskNode.setMaxRetryTimes(taskDefinitionLog.getFailRetryTimes());
- taskNode.setRetryInterval(taskDefinitionLog.getFailRetryInterval());
- Map<String, Object> taskParamsMap = taskNode.taskParamsToJsonObj(taskDefinitionLog.getTaskParams());
- taskNode.setConditionResult(JSONUtils.toJsonString(taskParamsMap.get(Constants.CONDITION_RESULT)));
- taskNode.setSwitchResult(JSONUtils.toJsonString(taskParamsMap.get(Constants.SWITCH_RESULT)));
- taskNode.setDependence(JSONUtils.toJsonString(taskParamsMap.get(Constants.DEPENDENCE)));
- taskParamsMap.remove(Constants.CONDITION_RESULT);
- taskParamsMap.remove(Constants.DEPENDENCE);
- taskNode.setParams(JSONUtils.toJsonString(taskParamsMap));
- taskNode.setTaskInstancePriority(taskDefinitionLog.getTaskPriority());
- taskNode.setWorkerGroup(taskDefinitionLog.getWorkerGroup());
- taskNode.setEnvironmentCode(taskDefinitionLog.getEnvironmentCode());
- taskNode.setTimeout(JSONUtils.toJsonString(new TaskTimeoutParameter(taskDefinitionLog.getTimeoutFlag() == TimeoutFlag.OPEN,
- taskDefinitionLog.getTimeoutNotifyStrategy(),
- taskDefinitionLog.getTimeout())));
- taskNode.setDelayTime(taskDefinitionLog.getDelayTime());
- taskNode.setPreTasks(JSONUtils.toJsonString(code.getValue().stream().map(taskDefinitionLogMap::get).map(TaskDefinition::getCode).collect(Collectors.toList())));
- taskNode.setTaskGroupId(taskDefinitionLog.getTaskGroupId());
- taskNode.setTaskGroupPriority(taskDefinitionLog.getTaskGroupPriority());
- taskNodeList.add(taskNode);
- }
- }
- return taskNodeList;
- }
-
- public Map<ProcessInstance, TaskInstance> notifyProcessList(int processId) {
- HashMap<ProcessInstance, TaskInstance> processTaskMap = new HashMap<>();
- //find sub tasks
- ProcessInstanceMap processInstanceMap = processInstanceMapMapper.queryBySubProcessId(processId);
- if (processInstanceMap == null) {
- return processTaskMap;
- }
- ProcessInstance fatherProcess = this.findProcessInstanceById(processInstanceMap.getParentProcessInstanceId());
- TaskInstance fatherTask = this.findTaskInstanceById(processInstanceMap.getParentTaskInstanceId());
-
- if (fatherProcess != null) {
- processTaskMap.put(fatherProcess, fatherTask);
- }
- return processTaskMap;
- }
-
- public DqExecuteResult getDqExecuteResultByTaskInstanceId(int taskInstanceId) {
- return dqExecuteResultMapper.getExecuteResultById(taskInstanceId);
- }
-
- public int updateDqExecuteResultUserId(int taskInstanceId) {
- DqExecuteResult dqExecuteResult =
- dqExecuteResultMapper.selectOne(new QueryWrapper<DqExecuteResult>().eq(TASK_INSTANCE_ID, taskInstanceId));
- if (dqExecuteResult == null) {
- return -1;
- }
-
- ProcessInstance processInstance = processInstanceMapper.selectById(dqExecuteResult.getProcessInstanceId());
- if (processInstance == null) {
- return -1;
- }
-
- ProcessDefinition processDefinition = processDefineMapper.queryByCode(processInstance.getProcessDefinitionCode());
- if (processDefinition == null) {
- return -1;
- }
-
- dqExecuteResult.setProcessDefinitionId(processDefinition.getId());
- dqExecuteResult.setUserId(processDefinition.getUserId());
- dqExecuteResult.setState(DqTaskState.DEFAULT.getCode());
- return dqExecuteResultMapper.updateById(dqExecuteResult);
- }
-
- public int updateDqExecuteResultState(DqExecuteResult dqExecuteResult) {
- return dqExecuteResultMapper.updateById(dqExecuteResult);
- }
-
- public int deleteDqExecuteResultByTaskInstanceId(int taskInstanceId) {
- return dqExecuteResultMapper.delete(
- new QueryWrapper<DqExecuteResult>()
- .eq(TASK_INSTANCE_ID, taskInstanceId));
- }
-
- public int deleteTaskStatisticsValueByTaskInstanceId(int taskInstanceId) {
- return dqTaskStatisticsValueMapper.delete(
- new QueryWrapper<DqTaskStatisticsValue>()
- .eq(TASK_INSTANCE_ID, taskInstanceId));
- }
-
- public DqRule getDqRule(int ruleId) {
- return dqRuleMapper.selectById(ruleId);
- }
-
- public List<DqRuleInputEntry> getRuleInputEntry(int ruleId) {
- return DqRuleUtils.transformInputEntry(dqRuleInputEntryMapper.getRuleInputEntryList(ruleId));
- }
-
- public List<DqRuleExecuteSql> getDqExecuteSql(int ruleId) {
- return dqRuleExecuteSqlMapper.getExecuteSqlList(ruleId);
- }
-
- public DqComparisonType getComparisonTypeById(int id) {
- return dqComparisonTypeMapper.selectById(id);
- }
-
- /**
- * the first time (when submit the task ) get the resource of the task group
- *
- * @param taskId task id
- */
- public boolean acquireTaskGroup(int taskId,
- String taskName, int groupId,
- int processId, int priority) {
- TaskGroup taskGroup = taskGroupMapper.selectById(groupId);
- if (taskGroup == null) {
- return true;
- }
- // if task group is not applicable
- if (taskGroup.getStatus() == Flag.NO.getCode()) {
- return true;
- }
- TaskGroupQueue taskGroupQueue = this.taskGroupQueueMapper.queryByTaskId(taskId);
- if (taskGroupQueue == null) {
- taskGroupQueue = insertIntoTaskGroupQueue(taskId, taskName, groupId, processId, priority, TaskGroupQueueStatus.WAIT_QUEUE);
- } else {
- if (taskGroupQueue.getStatus() == TaskGroupQueueStatus.ACQUIRE_SUCCESS) {
- return true;
- }
- taskGroupQueue.setInQueue(Flag.NO.getCode());
- taskGroupQueue.setStatus(TaskGroupQueueStatus.WAIT_QUEUE);
- this.taskGroupQueueMapper.updateById(taskGroupQueue);
- }
- //check priority
- List<TaskGroupQueue> highPriorityTasks = taskGroupQueueMapper.queryHighPriorityTasks(groupId, priority, TaskGroupQueueStatus.WAIT_QUEUE.getCode());
- if (CollectionUtils.isNotEmpty(highPriorityTasks)) {
- this.taskGroupQueueMapper.updateInQueue(Flag.NO.getCode(), taskGroupQueue.getId());
- return false;
- }
- //try to get taskGroup
- int count = taskGroupMapper.selectAvailableCountById(groupId);
- if (count == 1 && robTaskGroupResouce(taskGroupQueue)) {
- return true;
- }
- this.taskGroupQueueMapper.updateInQueue(Flag.NO.getCode(), taskGroupQueue.getId());
- return false;
- }
-
- /**
- * try to get the task group resource(when other task release the resource)
- */
- public boolean robTaskGroupResouce(TaskGroupQueue taskGroupQueue) {
- TaskGroup taskGroup = taskGroupMapper.selectById(taskGroupQueue.getGroupId());
- int affectedCount = taskGroupMapper.updateTaskGroupResource(taskGroup.getId(), taskGroupQueue.getId(),
- TaskGroupQueueStatus.WAIT_QUEUE.getCode());
- if (affectedCount > 0) {
- taskGroupQueue.setStatus(TaskGroupQueueStatus.ACQUIRE_SUCCESS);
- this.taskGroupQueueMapper.updateById(taskGroupQueue);
- this.taskGroupQueueMapper.updateInQueue(Flag.NO.getCode(), taskGroupQueue.getId());
- return true;
- }
- return false;
- }
-
- public boolean acquireTaskGroupAgain(TaskGroupQueue taskGroupQueue) {
- return robTaskGroupResouce(taskGroupQueue);
- }
-
- public void releaseAllTaskGroup(int processInstanceId) {
- List<TaskInstance> taskInstances = this.taskInstanceMapper.loadAllInfosNoRelease(processInstanceId, TaskGroupQueueStatus.ACQUIRE_SUCCESS.getCode());
- for (TaskInstance info : taskInstances) {
- releaseTaskGroup(info);
- }
- }
-
- /**
- * release the TGQ resource when the corresponding task is finished.
- *
- * @return the result code and msg
- */
- public TaskInstance releaseTaskGroup(TaskInstance taskInstance) {
-
- TaskGroup taskGroup = taskGroupMapper.selectById(taskInstance.getTaskGroupId());
- if (taskGroup == null) {
- return null;
- }
- TaskGroupQueue thisTaskGroupQueue = this.taskGroupQueueMapper.queryByTaskId(taskInstance.getId());
- if (thisTaskGroupQueue.getStatus() == TaskGroupQueueStatus.RELEASE) {
- return null;
- }
- try {
- while (taskGroupMapper.releaseTaskGroupResource(taskGroup.getId(), taskGroup.getUseSize()
- , thisTaskGroupQueue.getId(), TaskGroupQueueStatus.ACQUIRE_SUCCESS.getCode()) != 1) {
- thisTaskGroupQueue = this.taskGroupQueueMapper.queryByTaskId(taskInstance.getId());
- if (thisTaskGroupQueue.getStatus() == TaskGroupQueueStatus.RELEASE) {
- return null;
- }
- taskGroup = taskGroupMapper.selectById(taskInstance.getTaskGroupId());
- }
- } catch (Exception e) {
- logger.error("release the task group error", e);
- }
- logger.info("updateTask:{}", taskInstance.getName());
- changeTaskGroupQueueStatus(taskInstance.getId(), TaskGroupQueueStatus.RELEASE);
- TaskGroupQueue taskGroupQueue = this.taskGroupQueueMapper.queryTheHighestPriorityTasks(taskGroup.getId(),
- TaskGroupQueueStatus.WAIT_QUEUE.getCode(), Flag.NO.getCode(), Flag.NO.getCode());
- if (taskGroupQueue == null) {
- return null;
- }
- while (this.taskGroupQueueMapper.updateInQueueCAS(Flag.NO.getCode(), Flag.YES.getCode(), taskGroupQueue.getId()) != 1) {
- taskGroupQueue = this.taskGroupQueueMapper.queryTheHighestPriorityTasks(taskGroup.getId(),
- TaskGroupQueueStatus.WAIT_QUEUE.getCode(), Flag.NO.getCode(), Flag.NO.getCode());
- if (taskGroupQueue == null) {
- return null;
- }
- }
- return this.taskInstanceMapper.selectById(taskGroupQueue.getTaskId());
- }
-
- /**
- * release the TGQ resource when the corresponding task is finished.
- *
- * @param taskId task id
- * @return the result code and msg
- */
-
- public void changeTaskGroupQueueStatus(int taskId, TaskGroupQueueStatus status) {
- TaskGroupQueue taskGroupQueue = taskGroupQueueMapper.queryByTaskId(taskId);
- taskGroupQueue.setStatus(status);
- taskGroupQueue.setUpdateTime(new Date(System.currentTimeMillis()));
- taskGroupQueueMapper.updateById(taskGroupQueue);
- }
-
- /**
- * insert into task group queue
- *
- * @param taskId task id
- * @param taskName task name
- * @param groupId group id
- * @param processId process id
- * @param priority priority
- * @return result and msg code
- */
- public TaskGroupQueue insertIntoTaskGroupQueue(Integer taskId,
- String taskName, Integer groupId,
- Integer processId, Integer priority, TaskGroupQueueStatus status) {
- TaskGroupQueue taskGroupQueue = new TaskGroupQueue(taskId, taskName, groupId, processId, priority, status);
- taskGroupQueue.setCreateTime(new Date());
- taskGroupQueue.setUpdateTime(new Date());
- taskGroupQueueMapper.insert(taskGroupQueue);
- return taskGroupQueue;
- }
-
- public int updateTaskGroupQueueStatus(Integer taskId, int status) {
- return taskGroupQueueMapper.updateStatusByTaskId(taskId, status);
- }
-
- public int updateTaskGroupQueue(TaskGroupQueue taskGroupQueue) {
- return taskGroupQueueMapper.updateById(taskGroupQueue);
- }
-
- public TaskGroupQueue loadTaskGroupQueue(int taskId) {
- return this.taskGroupQueueMapper.queryByTaskId(taskId);
- }
-
- public void sendStartTask2Master(ProcessInstance processInstance, int taskId,
- org.apache.dolphinscheduler.remote.command.CommandType taskType) {
- String host = processInstance.getHost();
- String address = host.split(":")[0];
- int port = Integer.parseInt(host.split(":")[1]);
- TaskEventChangeCommand taskEventChangeCommand = new TaskEventChangeCommand(
- processInstance.getId(), taskId
- );
- stateEventCallbackService.sendResult(address, port, taskEventChangeCommand.convert2Command(taskType));
- }
-
- public ProcessInstance loadNextProcess4Serial(long code, int state) {
- return this.processInstanceMapper.loadNextProcess4Serial(code, state);
- }
-
- protected void deleteCommandWithCheck(int commandId) {
- int delete = this.commandMapper.deleteById(commandId);
- if (delete != 1) {
- throw new ServiceException("delete command fail, id:" + commandId);
- }
- }
+ void processNeedFailoverProcessInstances(ProcessInstance processInstance);
+
+ List<TaskInstance> queryNeedFailoverTaskInstances(String host);
+
+ DataSource findDataSourceById(int id);
+
+ int updateProcessInstanceState(Integer processInstanceId, ExecutionStatus executionStatus);
+
+ ProcessInstance findProcessInstanceByTaskId(int taskId);
+
+ List<UdfFunc> queryUdfFunListByIds(Integer[] ids);
+
+ String queryTenantCodeByResName(String resName, ResourceType resourceType);
+
+ List<Schedule> selectAllByProcessDefineCode(long[] codes);
+
+ ProcessInstance findLastSchedulerProcessInterval(Long definitionCode, DateInterval dateInterval);
+
+ ProcessInstance findLastManualProcessInterval(Long definitionCode, DateInterval dateInterval);
+
+ ProcessInstance findLastRunningProcess(Long definitionCode, Date startTime, Date endTime);
+
+ String queryUserQueueByProcessInstance(ProcessInstance processInstance);
+
+ ProjectUser queryProjectWithUserByProcessInstanceId(int processInstanceId);
+
+ String getTaskWorkerGroup(TaskInstance taskInstance);
+
+ List<Project> getProjectListHavePerm(int userId);
+
+ <T> List<T> listUnauthorized(int userId, T[] needChecks, AuthorizationType authorizationType);
+
+ User getUserById(int userId);
+
+ Resource getResourceById(int resourceId);
+
+ List<Resource> listResourceByIds(Integer[] resIds);
+
+ String formatTaskAppId(TaskInstance taskInstance);
+
+ int switchVersion(ProcessDefinition processDefinition, ProcessDefinitionLog processDefinitionLog);
+
+ int switchProcessTaskRelationVersion(ProcessDefinition processDefinition);
+
+ int switchTaskDefinitionVersion(long taskCode, int taskVersion);
+
+ String getResourceIds(TaskDefinition taskDefinition);
+
+ int saveTaskDefine(User operator, long projectCode, List<TaskDefinitionLog> taskDefinitionLogs, Boolean syncDefine);
+
+ int saveProcessDefine(User operator, ProcessDefinition processDefinition, Boolean syncDefine, Boolean isFromProcessDefine);
+
+ int saveTaskRelation(User operator, long projectCode, long processDefinitionCode, int processDefinitionVersion,
+ List<ProcessTaskRelationLog> taskRelationList, List<TaskDefinitionLog> taskDefinitionLogs,
+ Boolean syncDefine);
+
+ boolean isTaskOnline(long taskCode);
+
+ DAG<String, TaskNode, TaskNodeRelation> genDagGraph(ProcessDefinition processDefinition);
+
+ DagData genDagData(ProcessDefinition processDefinition);
+
+ List<TaskDefinitionLog> genTaskDefineList(List<ProcessTaskRelation> processTaskRelations);
+
+ List<TaskDefinitionLog> getTaskDefineLogListByRelation(List<ProcessTaskRelation> processTaskRelations);
+
+ TaskDefinition findTaskDefinition(long taskCode, int taskDefinitionVersion);
+
+ List<ProcessTaskRelation> findRelationByCode(long processDefinitionCode, int processDefinitionVersion);
+
+ List<TaskNode> transformTask(List<ProcessTaskRelation> taskRelationList, List<TaskDefinitionLog> taskDefinitionLogs);
+
+ Map<ProcessInstance, TaskInstance> notifyProcessList(int processId);
+
+ DqExecuteResult getDqExecuteResultByTaskInstanceId(int taskInstanceId);
+
+ int updateDqExecuteResultUserId(int taskInstanceId);
+
+ int updateDqExecuteResultState(DqExecuteResult dqExecuteResult);
+
+ int deleteDqExecuteResultByTaskInstanceId(int taskInstanceId);
+
+ int deleteTaskStatisticsValueByTaskInstanceId(int taskInstanceId);
+
+ DqRule getDqRule(int ruleId);
+
+ List<DqRuleInputEntry> getRuleInputEntry(int ruleId);
+
+ List<DqRuleExecuteSql> getDqExecuteSql(int ruleId);
+
+ DqComparisonType getComparisonTypeById(int id);
+
+ boolean acquireTaskGroup(int taskId,
+ String taskName, int groupId,
+ int processId, int priority);
+
+ boolean robTaskGroupResouce(TaskGroupQueue taskGroupQueue);
+
+ boolean acquireTaskGroupAgain(TaskGroupQueue taskGroupQueue);
+
+ void releaseAllTaskGroup(int processInstanceId);
+
+ TaskInstance releaseTaskGroup(TaskInstance taskInstance);
+
+ void changeTaskGroupQueueStatus(int taskId, TaskGroupQueueStatus status);
+
+ TaskGroupQueue insertIntoTaskGroupQueue(Integer taskId,
+ String taskName, Integer groupId,
+ Integer processId, Integer priority, TaskGroupQueueStatus status);
+
+ int updateTaskGroupQueueStatus(Integer taskId, int status);
+
+ int updateTaskGroupQueue(TaskGroupQueue taskGroupQueue);
+
+ TaskGroupQueue loadTaskGroupQueue(int taskId);
+
+ void sendStartTask2Master(ProcessInstance processInstance, int taskId,
+ org.apache.dolphinscheduler.remote.command.CommandType taskType);
+
+ ProcessInstance loadNextProcess4Serial(long code, int state);
}
diff --git a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessServiceImpl.java
similarity index 97%
copy from dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
copy to dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessServiceImpl.java
index 2255c0811b..e6d03a8aaa 100644
--- a/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessService.java
+++ b/dolphinscheduler-service/src/main/java/org/apache/dolphinscheduler/service/process/ProcessServiceImpl.java
@@ -17,19 +17,13 @@
package org.apache.dolphinscheduler.service.process;
-import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_END_DATE;
-import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_START_DATE;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_EMPTY_SUB_PROCESS;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_FATHER_PARAMS;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_RECOVER_PROCESS_ID_STRING;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_SUB_PROCESS;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_SUB_PROCESS_DEFINE_CODE;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_SUB_PROCESS_PARENT_INSTANCE_ID;
-import static org.apache.dolphinscheduler.common.Constants.LOCAL_PARAMS;
-import static org.apache.dolphinscheduler.plugin.task.api.utils.DataQualityConstants.TASK_INSTANCE_ID;
-
-import static java.util.stream.Collectors.toSet;
-
+import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
+import com.fasterxml.jackson.core.type.TypeReference;
+import com.fasterxml.jackson.databind.node.ObjectNode;
+import com.google.common.collect.Lists;
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.lang.StringUtils;
+import org.apache.commons.lang.math.NumberUtils;
import org.apache.dolphinscheduler.common.Constants;
import org.apache.dolphinscheduler.common.enums.AuthorizationType;
import org.apache.dolphinscheduler.common.enums.CommandType;
@@ -130,10 +124,11 @@ import org.apache.dolphinscheduler.service.log.LogClientService;
import org.apache.dolphinscheduler.service.quartz.cron.CronUtils;
import org.apache.dolphinscheduler.service.task.TaskPluginManager;
import org.apache.dolphinscheduler.spi.enums.ResourceType;
-
-import org.apache.commons.collections.CollectionUtils;
-import org.apache.commons.lang.StringUtils;
-import org.apache.commons.lang.math.NumberUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.stereotype.Component;
+import org.springframework.transaction.annotation.Transactional;
import java.util.ArrayList;
import java.util.Arrays;
@@ -148,22 +143,23 @@ import java.util.Objects;
import java.util.Set;
import java.util.stream.Collectors;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import org.springframework.beans.factory.annotation.Autowired;
-import org.springframework.stereotype.Component;
-import org.springframework.transaction.annotation.Transactional;
-
-import com.baomidou.mybatisplus.core.conditions.query.QueryWrapper;
-import com.fasterxml.jackson.core.type.TypeReference;
-import com.fasterxml.jackson.databind.node.ObjectNode;
-import com.google.common.collect.Lists;
+import static java.util.stream.Collectors.toSet;
+import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_END_DATE;
+import static org.apache.dolphinscheduler.common.Constants.CMDPARAM_COMPLEMENT_DATA_START_DATE;
+import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_EMPTY_SUB_PROCESS;
+import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_FATHER_PARAMS;
+import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_RECOVER_PROCESS_ID_STRING;
+import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_SUB_PROCESS;
+import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_SUB_PROCESS_DEFINE_CODE;
+import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_SUB_PROCESS_PARENT_INSTANCE_ID;
+import static org.apache.dolphinscheduler.common.Constants.LOCAL_PARAMS;
+import static org.apache.dolphinscheduler.plugin.task.api.utils.DataQualityConstants.TASK_INSTANCE_ID;
/**
* process relative dao that some mappers in this.
*/
@Component
-public class ProcessService {
+public class ProcessServiceImpl implements ProcessService {
private final Logger logger = LoggerFactory.getLogger(getClass());
@@ -271,11 +267,12 @@ public class ProcessService {
/**
* handle Command (construct ProcessInstance from Command) , wrapped in transaction
*
- * @param logger logger
- * @param host host
+ * @param logger logger
+ * @param host host
* @param command found command
* @return process instance
*/
+ @Override
@Transactional
public ProcessInstance handleCommand(Logger logger, String host, Command command) {
ProcessInstance processInstance = constructProcessInstance(command, host);
@@ -364,6 +361,7 @@ public class ProcessService {
* @param command command
* @param message message
*/
+ @Override
public void moveToErrorCommand(Command command, String message) {
ErrorCommand errorCommand = new ErrorCommand(command, message);
this.errorCommandMapper.insert(errorCommand);
@@ -373,7 +371,7 @@ public class ProcessService {
/**
* set process waiting thread
*
- * @param command command
+ * @param command command
* @param processInstance processInstance
* @return process instance
*/
@@ -394,6 +392,7 @@ public class ProcessService {
* @param command command
* @return create result
*/
+ @Override
public int createCommand(Command command) {
int result = 0;
if (command != null) {
@@ -405,6 +404,7 @@ public class ProcessService {
/**
* get command page
*/
+ @Override
public List<Command> findCommandPage(int pageSize, int pageNumber) {
return commandMapper.queryCommandPage(pageSize, pageNumber * pageSize);
}
@@ -412,6 +412,7 @@ public class ProcessService {
/**
* get command page
*/
+ @Override
public List<Command> findCommandPageBySlot(int pageSize, int pageNumber, int masterCount, int thisMasterSlot) {
if (masterCount <= 0) {
return Lists.newArrayList();
@@ -425,6 +426,7 @@ public class ProcessService {
* @param command command
* @return create command result
*/
+ @Override
public boolean verifyIsNeedCreateCommand(Command command) {
boolean isNeedCreate = true;
EnumMap<CommandType, Integer> cmdTypeMap = new EnumMap<>(CommandType.class);
@@ -458,6 +460,7 @@ public class ProcessService {
* @param processId processId
* @return process instance
*/
+ @Override
public ProcessInstance findProcessInstanceDetailById(int processId) {
return processInstanceMapper.queryDetailById(processId);
}
@@ -465,6 +468,7 @@ public class ProcessService {
/**
* get task node list by definitionId
*/
+ @Override
public List<TaskDefinition> getTaskNodeListByDefinition(long defineCode) {
ProcessDefinition processDefinition = processDefineMapper.queryByCode(defineCode);
if (processDefinition == null) {
@@ -491,6 +495,7 @@ public class ProcessService {
* @param processId processId
* @return process instance
*/
+ @Override
public ProcessInstance findProcessInstanceById(int processId) {
return processInstanceMapper.selectById(processId);
}
@@ -501,6 +506,7 @@ public class ProcessService {
* @param processDefinitionId processDefinitionId
* @return process definition
*/
+ @Override
public ProcessDefinition findProcessDefineById(int processDefinitionId) {
return processDefineMapper.selectById(processDefinitionId);
}
@@ -511,6 +517,7 @@ public class ProcessService {
* @param processDefinitionCode processDefinitionCode
* @return process definition
*/
+ @Override
public ProcessDefinition findProcessDefinition(Long processDefinitionCode, int version) {
ProcessDefinition processDefinition = processDefineMapper.queryByCode(processDefinitionCode);
if (processDefinition == null || processDefinition.getVersion() != version) {
@@ -528,6 +535,7 @@ public class ProcessService {
* @param processDefinitionCode processDefinitionCode
* @return process definition
*/
+ @Override
public ProcessDefinition findProcessDefinitionByCode(Long processDefinitionCode) {
return processDefineMapper.queryByCode(processDefinitionCode);
}
@@ -538,6 +546,7 @@ public class ProcessService {
* @param processInstanceId processInstanceId
* @return delete process instance result
*/
+ @Override
public int deleteWorkProcessInstanceById(int processInstanceId) {
return processInstanceMapper.deleteById(processInstanceId);
}
@@ -548,6 +557,7 @@ public class ProcessService {
* @param processInstanceId processInstanceId
* @return delete all sub process instance result
*/
+ @Override
public int deleteAllSubWorkProcessByParentId(int processInstanceId) {
List<Integer> subProcessIdList = processInstanceMapMapper.querySubIdListByParentId(processInstanceId);
@@ -566,6 +576,7 @@ public class ProcessService {
*
* @param processInstanceId processInstanceId
*/
+ @Override
public void removeTaskLogFile(Integer processInstanceId) {
List<TaskInstance> taskInstanceList = findValidTaskListByProcessId(processInstanceId);
if (CollectionUtils.isEmpty(taskInstanceList)) {
@@ -587,6 +598,7 @@ public class ProcessService {
/**
* recursive delete all task instance by process instance id
*/
+ @Override
public void deleteWorkTaskInstanceByProcessInstanceId(int processInstanceId) {
List<TaskInstance> taskInstanceList = findValidTaskListByProcessId(processInstanceId);
if (CollectionUtils.isEmpty(taskInstanceList)) {
@@ -606,8 +618,9 @@ public class ProcessService {
* recursive query sub process definition id by parent id.
*
* @param parentCode parentCode
- * @param ids ids
+ * @param ids ids
*/
+ @Override
public void recurseFindSubProcess(long parentCode, List<Long> ids) {
List<TaskDefinition> taskNodeList = this.getTaskNodeListByDefinition(parentCode);
@@ -631,9 +644,10 @@ public class ProcessService {
* create recovery waiting thread command and delete origin command at the same time.
* if the recovery command is exists, only update the field update_time
*
- * @param originCommand originCommand
+ * @param originCommand originCommand
* @param processInstance processInstance
*/
+ @Override
public void createRecoveryWaitingThreadCommand(Command originCommand, ProcessInstance processInstance) {
// sub process doesnot need to create wait command
@@ -687,7 +701,7 @@ public class ProcessService {
/**
* get schedule time from command
*
- * @param command command
+ * @param command command
* @param cmdParam cmdParam map
* @return date
*/
@@ -716,8 +730,8 @@ public class ProcessService {
* generate a new work process instance from command.
*
* @param processDefinition processDefinition
- * @param command command
- * @param cmdParam cmdParam map
+ * @param command command
+ * @param cmdParam cmdParam map
* @return process instance
*/
private ProcessInstance generateNewProcessInstance(ProcessDefinition processDefinition,
@@ -802,9 +816,10 @@ public class ProcessService {
* use definition creator's tenant.
*
* @param tenantId tenantId
- * @param userId userId
+ * @param userId userId
* @return tenant
*/
+ @Override
public Tenant getTenantForProcess(int tenantId, int userId) {
Tenant tenant = null;
if (tenantId >= 0) {
@@ -829,6 +844,7 @@ public class ProcessService {
* @param environmentCode environmentCode
* @return Environment
*/
+ @Override
public Environment findEnvironmentByCode(Long environmentCode) {
Environment environment = null;
if (environmentCode >= 0) {
@@ -840,7 +856,7 @@ public class ProcessService {
/**
* check command parameters is valid
*
- * @param command command
+ * @param command command
* @param cmdParam cmdParam map
* @return whether command param is valid
*/
@@ -860,7 +876,7 @@ public class ProcessService {
* construct process instance according to one command.
*
* @param command command
- * @param host host
+ * @param host host
* @return process instance
*/
protected ProcessInstance constructProcessInstance(Command command, String host) {
@@ -1039,7 +1055,7 @@ public class ProcessService {
* return complement data if the process start with complement data
*
* @param processInstance processInstance
- * @param command command
+ * @param command command
* @return command type
*/
private CommandType getCommandTypeIfComplement(ProcessInstance processInstance, Command command) {
@@ -1054,8 +1070,8 @@ public class ProcessService {
* initialize complement data parameters
*
* @param processDefinition processDefinition
- * @param processInstance processInstance
- * @param cmdParam cmdParam
+ * @param processInstance processInstance
+ * @param cmdParam cmdParam
*/
private void initComplementDataParam(ProcessDefinition processDefinition,
ProcessInstance processInstance,
@@ -1086,6 +1102,7 @@ public class ProcessService {
*
* @param subProcessInstance subProcessInstance
*/
+ @Override
public void setSubProcessParam(ProcessInstance subProcessInstance) {
String cmdParam = subProcessInstance.getCommandParam();
if (StringUtils.isEmpty(cmdParam)) {
@@ -1128,7 +1145,7 @@ public class ProcessService {
* only the keys doesn't in sub process global would be joined.
*
* @param parentGlobalParams parentGlobalParams
- * @param subGlobalParams subGlobalParams
+ * @param subGlobalParams subGlobalParams
* @return global params join
*/
private String joinGlobalParams(String parentGlobalParams, String subGlobalParams) {
@@ -1170,6 +1187,7 @@ public class ProcessService {
/**
* retry submit task to db
*/
+ @Override
public TaskInstance submitTaskWithRetry(ProcessInstance processInstance, TaskInstance taskInstance, int commitRetryTimes, int commitInterval) {
int retryTimes = 1;
TaskInstance task = null;
@@ -1195,9 +1213,10 @@ public class ProcessService {
* submit sub process to command
*
* @param processInstance processInstance
- * @param taskInstance taskInstance
+ * @param taskInstance taskInstance
* @return task instance
*/
+ @Override
@Transactional(rollbackFor = Exception.class)
public TaskInstance submitTask(ProcessInstance processInstance, TaskInstance taskInstance) {
logger.info("start submit task : {}, instance id:{}, state: {}",
@@ -1226,7 +1245,7 @@ public class ProcessService {
* set map {parent instance id, task instance id, 0(child instance id)}
*
* @param parentInstance parentInstance
- * @param parentTask parentTask
+ * @param parentTask parentTask
* @return process instance map
*/
private ProcessInstanceMap setProcessInstanceMap(ProcessInstance parentInstance, TaskInstance parentTask) {
@@ -1255,7 +1274,7 @@ public class ProcessService {
* find previous task work process map.
*
* @param parentProcessInstance parentProcessInstance
- * @param parentTask parentTask
+ * @param parentTask parentTask
* @return process instance map
*/
private ProcessInstanceMap findPreviousTaskProcessMap(ProcessInstance parentProcessInstance,
@@ -1281,8 +1300,9 @@ public class ProcessService {
* create sub work process command
*
* @param parentProcessInstance parentProcessInstance
- * @param task task
+ * @param task task
*/
+ @Override
public void createSubWorkProcess(ProcessInstance parentProcessInstance, TaskInstance task) {
if (!task.isSubProcess()) {
return;
@@ -1327,6 +1347,7 @@ public class ProcessService {
return processMapStr;
}
+ @Override
public Map<String, String> getGlobalParamMap(String globalParams) {
List<Property> propList;
Map<String, String> globalParamMap = new HashMap<>();
@@ -1341,6 +1362,7 @@ public class ProcessService {
/**
* create sub work process command
*/
+ @Override
public Command createSubProcessCommand(ProcessInstance parentProcessInstance,
ProcessInstance childInstance,
ProcessInstanceMap instanceMap,
@@ -1415,7 +1437,7 @@ public class ProcessService {
* update sub process definition
*
* @param parentProcessInstance parentProcessInstance
- * @param childDefinitionCode childDefinitionId
+ * @param childDefinitionCode childDefinitionId
*/
private void updateSubProcessDefinitionByParent(ProcessInstance parentProcessInstance, long childDefinitionCode) {
ProcessDefinition fatherDefinition = this.findProcessDefinition(parentProcessInstance.getProcessDefinitionCode(),
@@ -1430,10 +1452,11 @@ public class ProcessService {
/**
* submit task to mysql
*
- * @param taskInstance taskInstance
+ * @param taskInstance taskInstance
* @param processInstance processInstance
* @return task instance
*/
+ @Override
public TaskInstance submitTaskInstanceToDB(TaskInstance taskInstance, ProcessInstance processInstance) {
ExecutionStatus processInstanceState = processInstance.getState();
if (processInstanceState.typeIsFinished()
@@ -1466,10 +1489,11 @@ public class ProcessService {
* return stop if work process state is ready stop
* if all of above are not satisfied, return submit success
*
- * @param taskInstance taskInstance
+ * @param taskInstance taskInstance
* @param processInstance processInstance
* @return process instance state
*/
+ @Override
public ExecutionStatus getSubmitTaskState(TaskInstance taskInstance, ProcessInstance processInstance) {
ExecutionStatus state = taskInstance.getState();
// running, delayed or killed
@@ -1523,6 +1547,7 @@ public class ProcessService {
*
* @param processInstance processInstance
*/
+ @Override
public void saveProcessInstance(ProcessInstance processInstance) {
if (processInstance == null) {
logger.error("save error, process instance is null!");
@@ -1541,6 +1566,7 @@ public class ProcessService {
* @param command command
* @return save command result
*/
+ @Override
public int saveCommand(Command command) {
if (command.getId() != 0) {
return commandMapper.updateById(command);
@@ -1555,6 +1581,7 @@ public class ProcessService {
* @param taskInstance taskInstance
* @return save task instance result
*/
+ @Override
public boolean saveTaskInstance(TaskInstance taskInstance) {
if (taskInstance.getId() != 0) {
return updateTaskInstance(taskInstance);
@@ -1569,6 +1596,7 @@ public class ProcessService {
* @param taskInstance taskInstance
* @return create task instance result
*/
+ @Override
public boolean createTaskInstance(TaskInstance taskInstance) {
int count = taskInstanceMapper.insert(taskInstance);
return count > 0;
@@ -1580,6 +1608,7 @@ public class ProcessService {
* @param taskInstance taskInstance
* @return update task instance result
*/
+ @Override
public boolean updateTaskInstance(TaskInstance taskInstance) {
int count = taskInstanceMapper.updateById(taskInstance);
return count > 0;
@@ -1591,6 +1620,7 @@ public class ProcessService {
* @param taskId task id
* @return task instance
*/
+ @Override
public TaskInstance findTaskInstanceById(Integer taskId) {
return taskInstanceMapper.selectById(taskId);
}
@@ -1601,6 +1631,7 @@ public class ProcessService {
* @param idList task id list
* @return task instance list
*/
+ @Override
public List<TaskInstance> findTaskInstanceByIdList(List<Integer> idList) {
if (CollectionUtils.isEmpty(idList)) {
return new ArrayList<>();
@@ -1611,6 +1642,7 @@ public class ProcessService {
/**
* package task instance
*/
+ @Override
public void packageTaskInstance(TaskInstance taskInstance, ProcessInstance processInstance) {
taskInstance.setProcessInstance(processInstance);
taskInstance.setProcessDefine(processInstance.getProcessDefinition());
@@ -1626,6 +1658,7 @@ public class ProcessService {
*
* @param taskDefinition the given {@link TaskDefinition}
*/
+ @Override
public void updateTaskDefinitionResources(TaskDefinition taskDefinition) {
Map<String, Object> taskParameters = JSONUtils.parseObject(
taskDefinition.getTaskParams(),
@@ -1693,9 +1726,10 @@ public class ProcessService {
* get id list by task state
*
* @param instanceId instanceId
- * @param state state
+ * @param state state
* @return task instance states
*/
+ @Override
public List<Integer> findTaskIdByInstanceState(int instanceId, ExecutionStatus state) {
return taskInstanceMapper.queryTaskByProcessIdAndState(instanceId, state.ordinal());
}
@@ -1706,6 +1740,7 @@ public class ProcessService {
* @param processInstanceId processInstanceId
* @return task instance list
*/
+ @Override
public List<TaskInstance> findValidTaskListByProcessId(Integer processInstanceId) {
return taskInstanceMapper.findValidTaskListByProcessId(processInstanceId, Flag.YES);
}
@@ -1716,6 +1751,7 @@ public class ProcessService {
* @param processInstanceId processInstanceId
* @return task instance list
*/
+ @Override
public List<TaskInstance> findPreviousTaskListByWorkProcessId(Integer processInstanceId) {
return taskInstanceMapper.findValidTaskListByProcessId(processInstanceId, Flag.NO);
}
@@ -1726,6 +1762,7 @@ public class ProcessService {
* @param processInstanceMap processInstanceMap
* @return update process instance result
*/
+ @Override
public int updateWorkProcessInstanceMap(ProcessInstanceMap processInstanceMap) {
return processInstanceMapMapper.updateById(processInstanceMap);
}
@@ -1736,6 +1773,7 @@ public class ProcessService {
* @param processInstanceMap processInstanceMap
* @return create process instance result
*/
+ @Override
public int createWorkProcessInstanceMap(ProcessInstanceMap processInstanceMap) {
int count = 0;
if (processInstanceMap != null) {
@@ -1748,9 +1786,10 @@ public class ProcessService {
* find work process map by parent process id and parent task id.
*
* @param parentWorkProcessId parentWorkProcessId
- * @param parentTaskId parentTaskId
+ * @param parentTaskId parentTaskId
* @return process instance map
*/
+ @Override
public ProcessInstanceMap findWorkProcessMapByParent(Integer parentWorkProcessId, Integer parentTaskId) {
return processInstanceMapMapper.queryByParentId(parentWorkProcessId, parentTaskId);
}
@@ -1761,6 +1800,7 @@ public class ProcessService {
* @param parentWorkProcessId parentWorkProcessId
* @return delete process map result
*/
+ @Override
public int deleteWorkProcessMapByParentId(int parentWorkProcessId) {
return processInstanceMapMapper.deleteByParentProcessId(parentWorkProcessId);
@@ -1770,9 +1810,10 @@ public class ProcessService {
* find sub process instance
*
* @param parentProcessId parentProcessId
- * @param parentTaskId parentTaskId
+ * @param parentTaskId parentTaskId
* @return process instance
*/
+ @Override
public ProcessInstance findSubProcessInstance(Integer parentProcessId, Integer parentTaskId) {
ProcessInstance processInstance = null;
ProcessInstanceMap processInstanceMap = processInstanceMapMapper.queryByParentId(parentProcessId, parentTaskId);
@@ -1789,6 +1830,7 @@ public class ProcessService {
* @param subProcessId subProcessId
* @return process instance
*/
+ @Override
public ProcessInstance findParentProcessInstance(Integer subProcessId) {
ProcessInstance processInstance = null;
ProcessInstanceMap processInstanceMap = processInstanceMapMapper.queryBySubProcessId(subProcessId);
@@ -1805,6 +1847,7 @@ public class ProcessService {
* @param processInstance processInstance
* @return update process instance result
*/
+ @Override
public int updateProcessInstance(ProcessInstance processInstance) {
return processInstanceMapper.updateById(processInstance);
}
@@ -1812,6 +1855,7 @@ public class ProcessService {
/**
* for show in page of taskInstance
*/
+ @Override
public void changeOutParam(TaskInstance taskInstance) {
if (StringUtils.isEmpty(taskInstance.getVarPool())) {
return;
@@ -1850,6 +1894,7 @@ public class ProcessService {
* @param intList intList
* @return string list
*/
+ @Override
public List<String> convertIntListToString(List<Integer> intList) {
if (intList == null) {
return new ArrayList<>();
@@ -1867,6 +1912,7 @@ public class ProcessService {
* @param id id
* @return schedule
*/
+ @Override
public Schedule querySchedule(int id) {
return scheduleMapper.selectById(id);
}
@@ -1877,6 +1923,7 @@ public class ProcessService {
* @param processDefinitionCode processDefinitionCode
* @see Schedule
*/
+ @Override
public List<Schedule> queryReleaseSchedulerListByProcessDefinitionCode(long processDefinitionCode) {
return scheduleMapper.queryReleaseSchedulerListByProcessDefinitionCode(processDefinitionCode);
}
@@ -1887,6 +1934,7 @@ public class ProcessService {
* @param processDefinitionCodeList processDefinitionCodeList
* @see Schedule
*/
+ @Override
public Map<Long, String> queryWorkerGroupByProcessDefinitionCodes(List<Long> processDefinitionCodeList) {
List<Schedule> processDefinitionScheduleList = scheduleMapper.querySchedulesByProcessDefinitionCodes(processDefinitionCodeList);
return processDefinitionScheduleList.stream().collect(Collectors.toMap(Schedule::getProcessDefinitionCode,
@@ -1899,6 +1947,7 @@ public class ProcessService {
* @param processDefinitionCode processDefinitionCode
* @see DependentProcessDefinition
*/
+ @Override
public List<DependentProcessDefinition> queryDependentProcessDefinitionByProcessDefinitionCode(long processDefinitionCode) {
return workFlowLineageMapper.queryDependentProcessDefinitionByProcessDefinitionCode(processDefinitionCode);
}
@@ -1909,10 +1958,12 @@ public class ProcessService {
* @param host host
* @return process instance list
*/
+ @Override
public List<ProcessInstance> queryNeedFailoverProcessInstances(String host) {
return processInstanceMapper.queryByHostAndStatus(host, stateArray);
}
+ @Override
public List<String> queryNeedFailoverProcessInstanceHost() {
return processInstanceMapper.queryNeedFailoverProcessInstanceHost(stateArray);
}
@@ -1922,6 +1973,7 @@ public class ProcessService {
*
* @param processInstance processInstance
*/
+ @Override
@Transactional(rollbackFor = RuntimeException.class)
public void processNeedFailoverProcessInstances(ProcessInstance processInstance) {
//1 update processInstance host is null
@@ -1947,6 +1999,7 @@ public class ProcessService {
* @param host host
* @return task instance list
*/
+ @Override
public List<TaskInstance> queryNeedFailoverTaskInstances(String host) {
return taskInstanceMapper.queryByHostAndStatus(host,
stateArray);
@@ -1958,6 +2011,7 @@ public class ProcessService {
* @param id id
* @return datasource
*/
+ @Override
public DataSource findDataSourceById(int id) {
return dataSourceMapper.selectById(id);
}
@@ -1966,9 +2020,10 @@ public class ProcessService {
* update process instance state by id
*
* @param processInstanceId processInstanceId
- * @param executionStatus executionStatus
+ * @param executionStatus executionStatus
* @return update process result
*/
+ @Override
public int updateProcessInstanceState(Integer processInstanceId, ExecutionStatus executionStatus) {
ProcessInstance instance = processInstanceMapper.selectById(processInstanceId);
instance.setState(executionStatus);
@@ -1981,6 +2036,7 @@ public class ProcessService {
* @param taskId taskId
* @return process instance
*/
+ @Override
public ProcessInstance findProcessInstanceByTaskId(int taskId) {
TaskInstance taskInstance = taskInstanceMapper.selectById(taskId);
if (taskInstance != null) {
@@ -1995,6 +2051,7 @@ public class ProcessService {
* @param ids ids
* @return udf function list
*/
+ @Override
public List<UdfFunc> queryUdfFunListByIds(Integer[] ids) {
return udfFuncMapper.queryUdfByIdStr(ids, null);
}
@@ -2002,10 +2059,11 @@ public class ProcessService {
/**
* find tenant code by resource name
*
- * @param resName resource name
+ * @param resName resource name
* @param resourceType resource type
* @return tenant code
*/
+ @Override
public String queryTenantCodeByResName(String resName, ResourceType resourceType) {
// in order to query tenant code successful although the version is older
String fullName = resName.startsWith("/") ? resName : String.format("/%s", resName);
@@ -2032,6 +2090,7 @@ public class ProcessService {
* @param codes codes
* @return schedule list
*/
+ @Override
public List<Schedule> selectAllByProcessDefineCode(long[] codes) {
return scheduleMapper.selectAllByProcessDefineArray(codes);
}
@@ -2040,9 +2099,10 @@ public class ProcessService {
* find last scheduler process instance in the date interval
*
* @param definitionCode definitionCode
- * @param dateInterval dateInterval
+ * @param dateInterval dateInterval
* @return process instance
*/
+ @Override
public ProcessInstance findLastSchedulerProcessInterval(Long definitionCode, DateInterval dateInterval) {
return processInstanceMapper.queryLastSchedulerProcess(definitionCode,
dateInterval.getStartTime(),
@@ -2053,9 +2113,10 @@ public class ProcessService {
* find last manual process instance interval
*
* @param definitionCode process definition code
- * @param dateInterval dateInterval
+ * @param dateInterval dateInterval
* @return process instance
*/
+ @Override
public ProcessInstance findLastManualProcessInterval(Long definitionCode, DateInterval dateInterval) {
return processInstanceMapper.queryLastManualProcess(definitionCode,
dateInterval.getStartTime(),
@@ -2066,10 +2127,11 @@ public class ProcessService {
* find last running process instance
*
* @param definitionCode process definition code
- * @param startTime start time
- * @param endTime end time
+ * @param startTime start time
+ * @param endTime end time
* @return process instance
*/
+ @Override
public ProcessInstance findLastRunningProcess(Long definitionCode, Date startTime, Date endTime) {
return processInstanceMapper.queryLastRunningProcess(definitionCode,
startTime,
@@ -2083,6 +2145,7 @@ public class ProcessService {
* @param processInstance processInstance
* @return queue
*/
+ @Override
public String queryUserQueueByProcessInstance(ProcessInstance processInstance) {
String queue = "";
@@ -2102,6 +2165,7 @@ public class ProcessService {
* @param processInstanceId processInstanceId
* @return projectName and userName
*/
+ @Override
public ProjectUser queryProjectWithUserByProcessInstanceId(int processInstanceId) {
return projectMapper.queryProjectWithUserByProcessInstanceId(processInstanceId);
}
@@ -2112,6 +2176,7 @@ public class ProcessService {
* @param taskInstance taskInstance
* @return workerGroupId
*/
+ @Override
public String getTaskWorkerGroup(TaskInstance taskInstance) {
String workerGroup = taskInstance.getWorkerGroup();
@@ -2134,6 +2199,7 @@ public class ProcessService {
* @param userId userId
* @return project list
*/
+ @Override
public List<Project> getProjectListHavePerm(int userId) {
List<Project> createProjects = projectMapper.queryProjectCreatedByUser(userId);
List<Project> authedProjects = projectMapper.queryAuthedProjectListByUserId(userId);
@@ -2151,10 +2217,11 @@ public class ProcessService {
/**
* list unauthorized udf function
*
- * @param userId user id
+ * @param userId user id
* @param needChecks data source id array
* @return unauthorized udf function list
*/
+ @Override
public <T> List<T> listUnauthorized(int userId, T[] needChecks, AuthorizationType authorizationType) {
List<T> resultList = new ArrayList<>();
@@ -2199,6 +2266,7 @@ public class ProcessService {
* @param userId user id
* @return User
*/
+ @Override
public User getUserById(int userId) {
return userMapper.selectById(userId);
}
@@ -2209,6 +2277,7 @@ public class ProcessService {
* @param resourceId resource id
* @return Resource
*/
+ @Override
public Resource getResourceById(int resourceId) {
return resourceMapper.selectById(resourceId);
}
@@ -2219,6 +2288,7 @@ public class ProcessService {
* @param resIds resIds
* @return resource list
*/
+ @Override
public List<Resource> listResourceByIds(Integer[] resIds) {
return resourceMapper.listResourceByIds(resIds);
}
@@ -2226,6 +2296,7 @@ public class ProcessService {
/**
* format task app id in task instance
*/
+ @Override
public String formatTaskAppId(TaskInstance taskInstance) {
ProcessInstance processInstance = findProcessInstanceById(taskInstance.getProcessInstanceId());
if (processInstance == null) {
@@ -2241,6 +2312,7 @@ public class ProcessService {
/**
* switch process definition version to process definition log version
*/
+ @Override
public int switchVersion(ProcessDefinition processDefinition, ProcessDefinitionLog processDefinitionLog) {
if (null == processDefinition || null == processDefinitionLog) {
return Constants.DEFINITION_FAILURE;
@@ -2259,6 +2331,7 @@ public class ProcessService {
return result;
}
+ @Override
public int switchProcessTaskRelationVersion(ProcessDefinition processDefinition) {
List<ProcessTaskRelation> processTaskRelationList = processTaskRelationMapper.queryByProcessCode(processDefinition.getProjectCode(), processDefinition.getCode());
if (!processTaskRelationList.isEmpty()) {
@@ -2280,6 +2353,7 @@ public class ProcessService {
}
}
+ @Override
public int switchTaskDefinitionVersion(long taskCode, int taskVersion) {
TaskDefinition taskDefinition = taskDefinitionMapper.queryByCode(taskCode);
if (taskDefinition == null) {
@@ -2303,6 +2377,7 @@ public class ProcessService {
* @param taskDefinition taskDefinition
* @return resource ids
*/
+ @Override
public String getResourceIds(TaskDefinition taskDefinition) {
Set<Integer> resourceIds = null;
AbstractParameters params = taskPluginManager.getParameters(ParametersNode.builder().taskType(taskDefinition.getTaskType()).taskParams(taskDefinition.getTaskParams()).build());
@@ -2320,6 +2395,7 @@ public class ProcessService {
return StringUtils.join(resourceIds, ",");
}
+ @Override
public int saveTaskDefine(User operator, long projectCode, List<TaskDefinitionLog> taskDefinitionLogs, Boolean syncDefine) {
Date now = new Date();
List<TaskDefinitionLog> newTaskDefinitionLogs = new ArrayList<>();
@@ -2391,6 +2467,7 @@ public class ProcessService {
/**
* save processDefinition (including create or update processDefinition)
*/
+ @Override
public int saveProcessDefine(User operator, ProcessDefinition processDefinition, Boolean syncDefine, Boolean isFromProcessDefine) {
ProcessDefinitionLog processDefinitionLog = new ProcessDefinitionLog(processDefinition);
Integer version = processDefineLogMapper.queryMaxVersionForDefinition(processDefinition.getCode());
@@ -2415,6 +2492,7 @@ public class ProcessService {
/**
* save task relations
*/
+ @Override
public int saveTaskRelation(User operator, long projectCode, long processDefinitionCode, int processDefinitionVersion,
List<ProcessTaskRelationLog> taskRelationList, List<TaskDefinitionLog> taskDefinitionLogs,
Boolean syncDefine) {
@@ -2464,6 +2542,7 @@ public class ProcessService {
return (insert & resultLog) > 0 ? Constants.EXIT_CODE_SUCCESS : Constants.EXIT_CODE_FAILURE;
}
+ @Override
public boolean isTaskOnline(long taskCode) {
List<ProcessTaskRelation> processTaskRelationList = processTaskRelationMapper.queryByTaskCode(taskCode);
if (!processTaskRelationList.isEmpty()) {
@@ -2489,6 +2568,7 @@ public class ProcessService {
* @param processDefinition process definition
* @return dag graph
*/
+ @Override
public DAG<String, TaskNode, TaskNodeRelation> genDagGraph(ProcessDefinition processDefinition) {
List<ProcessTaskRelation> taskRelations = this.findRelationByCode(processDefinition.getCode(), processDefinition.getVersion());
List<TaskNode> taskNodeList = transformTask(taskRelations, Lists.newArrayList());
@@ -2500,6 +2580,7 @@ public class ProcessService {
/**
* generate DagData
*/
+ @Override
public DagData genDagData(ProcessDefinition processDefinition) {
List<ProcessTaskRelation> taskRelations = this.findRelationByCode(processDefinition.getCode(), processDefinition.getVersion());
List<TaskDefinitionLog> taskDefinitionLogList = genTaskDefineList(taskRelations);
@@ -2507,6 +2588,7 @@ public class ProcessService {
return new DagData(processDefinition, taskRelations, taskDefinitions);
}
+ @Override
public List<TaskDefinitionLog> genTaskDefineList(List<ProcessTaskRelation> processTaskRelations) {
Set<TaskDefinition> taskDefinitionSet = new HashSet<>();
for (ProcessTaskRelation processTaskRelation : processTaskRelations) {
@@ -2523,6 +2605,7 @@ public class ProcessService {
return taskDefinitionLogMapper.queryByTaskDefinitions(taskDefinitionSet);
}
+ @Override
public List<TaskDefinitionLog> getTaskDefineLogListByRelation(List<ProcessTaskRelation> processTaskRelations) {
List<TaskDefinitionLog> taskDefinitionLogs = new ArrayList<>();
Map<Long, Integer> taskCodeVersionMap = new HashMap<>();
@@ -2543,6 +2626,7 @@ public class ProcessService {
/**
* find task definition by code and version
*/
+ @Override
public TaskDefinition findTaskDefinition(long taskCode, int taskDefinitionVersion) {
return taskDefinitionLogMapper.queryByDefinitionCodeAndVersion(taskCode, taskDefinitionVersion);
}
@@ -2550,6 +2634,7 @@ public class ProcessService {
/**
* find process task relation list by process
*/
+ @Override
public List<ProcessTaskRelation> findRelationByCode(long processDefinitionCode, int processDefinitionVersion) {
List<ProcessTaskRelationLog> processTaskRelationLogList = processTaskRelationLogMapper.queryByProcessCodeAndVersion(processDefinitionCode, processDefinitionVersion);
return processTaskRelationLogList.stream().map(r -> (ProcessTaskRelation) r).collect(Collectors.toList());
@@ -2559,7 +2644,7 @@ public class ProcessService {
* add authorized resources
*
* @param ownResources own resources
- * @param userId userId
+ * @param userId userId
*/
private void addAuthorizedResources(List<Resource> ownResources, int userId) {
List<Integer> relationResourceIds = resourceUserMapper.queryResourcesIdListByUserIdAndPerm(userId, 7);
@@ -2570,6 +2655,7 @@ public class ProcessService {
/**
* Use temporarily before refactoring taskNode
*/
+ @Override
public List<TaskNode> transformTask(List<ProcessTaskRelation> taskRelationList, List<TaskDefinitionLog> taskDefinitionLogs) {
Map<Long, List<Long>> taskCodeMap = new HashMap<>();
for (ProcessTaskRelation processTaskRelation : taskRelationList) {
@@ -2624,6 +2710,7 @@ public class ProcessService {
return taskNodeList;
}
+ @Override
public Map<ProcessInstance, TaskInstance> notifyProcessList(int processId) {
HashMap<ProcessInstance, TaskInstance> processTaskMap = new HashMap<>();
//find sub tasks
@@ -2640,10 +2727,12 @@ public class ProcessService {
return processTaskMap;
}
+ @Override
public DqExecuteResult getDqExecuteResultByTaskInstanceId(int taskInstanceId) {
return dqExecuteResultMapper.getExecuteResultById(taskInstanceId);
}
+ @Override
public int updateDqExecuteResultUserId(int taskInstanceId) {
DqExecuteResult dqExecuteResult =
dqExecuteResultMapper.selectOne(new QueryWrapper<DqExecuteResult>().eq(TASK_INSTANCE_ID, taskInstanceId));
@@ -2667,34 +2756,41 @@ public class ProcessService {
return dqExecuteResultMapper.updateById(dqExecuteResult);
}
+ @Override
public int updateDqExecuteResultState(DqExecuteResult dqExecuteResult) {
return dqExecuteResultMapper.updateById(dqExecuteResult);
}
+ @Override
public int deleteDqExecuteResultByTaskInstanceId(int taskInstanceId) {
return dqExecuteResultMapper.delete(
new QueryWrapper<DqExecuteResult>()
.eq(TASK_INSTANCE_ID, taskInstanceId));
}
+ @Override
public int deleteTaskStatisticsValueByTaskInstanceId(int taskInstanceId) {
return dqTaskStatisticsValueMapper.delete(
new QueryWrapper<DqTaskStatisticsValue>()
.eq(TASK_INSTANCE_ID, taskInstanceId));
}
+ @Override
public DqRule getDqRule(int ruleId) {
return dqRuleMapper.selectById(ruleId);
}
+ @Override
public List<DqRuleInputEntry> getRuleInputEntry(int ruleId) {
return DqRuleUtils.transformInputEntry(dqRuleInputEntryMapper.getRuleInputEntryList(ruleId));
}
+ @Override
public List<DqRuleExecuteSql> getDqExecuteSql(int ruleId) {
return dqRuleExecuteSqlMapper.getExecuteSqlList(ruleId);
}
+ @Override
public DqComparisonType getComparisonTypeById(int id) {
return dqComparisonTypeMapper.selectById(id);
}
@@ -2704,6 +2800,7 @@ public class ProcessService {
*
* @param taskId task id
*/
+ @Override
public boolean acquireTaskGroup(int taskId,
String taskName, int groupId,
int processId, int priority) {
@@ -2744,6 +2841,7 @@ public class ProcessService {
/**
* try to get the task group resource(when other task release the resource)
*/
+ @Override
public boolean robTaskGroupResouce(TaskGroupQueue taskGroupQueue) {
TaskGroup taskGroup = taskGroupMapper.selectById(taskGroupQueue.getGroupId());
int affectedCount = taskGroupMapper.updateTaskGroupResource(taskGroup.getId(), taskGroupQueue.getId(),
@@ -2757,10 +2855,12 @@ public class ProcessService {
return false;
}
+ @Override
public boolean acquireTaskGroupAgain(TaskGroupQueue taskGroupQueue) {
return robTaskGroupResouce(taskGroupQueue);
}
+ @Override
public void releaseAllTaskGroup(int processInstanceId) {
List<TaskInstance> taskInstances = this.taskInstanceMapper.loadAllInfosNoRelease(processInstanceId, TaskGroupQueueStatus.ACQUIRE_SUCCESS.getCode());
for (TaskInstance info : taskInstances) {
@@ -2773,6 +2873,7 @@ public class ProcessService {
*
* @return the result code and msg
*/
+ @Override
public TaskInstance releaseTaskGroup(TaskInstance taskInstance) {
TaskGroup taskGroup = taskGroupMapper.selectById(taskInstance.getTaskGroupId());
@@ -2819,6 +2920,7 @@ public class ProcessService {
* @return the result code and msg
*/
+ @Override
public void changeTaskGroupQueueStatus(int taskId, TaskGroupQueueStatus status) {
TaskGroupQueue taskGroupQueue = taskGroupQueueMapper.queryByTaskId(taskId);
taskGroupQueue.setStatus(status);
@@ -2829,13 +2931,14 @@ public class ProcessService {
/**
* insert into task group queue
*
- * @param taskId task id
- * @param taskName task name
- * @param groupId group id
+ * @param taskId task id
+ * @param taskName task name
+ * @param groupId group id
* @param processId process id
- * @param priority priority
+ * @param priority priority
* @return result and msg code
*/
+ @Override
public TaskGroupQueue insertIntoTaskGroupQueue(Integer taskId,
String taskName, Integer groupId,
Integer processId, Integer priority, TaskGroupQueueStatus status) {
@@ -2846,18 +2949,22 @@ public class ProcessService {
return taskGroupQueue;
}
+ @Override
public int updateTaskGroupQueueStatus(Integer taskId, int status) {
return taskGroupQueueMapper.updateStatusByTaskId(taskId, status);
}
+ @Override
public int updateTaskGroupQueue(TaskGroupQueue taskGroupQueue) {
return taskGroupQueueMapper.updateById(taskGroupQueue);
}
+ @Override
public TaskGroupQueue loadTaskGroupQueue(int taskId) {
return this.taskGroupQueueMapper.queryByTaskId(taskId);
}
+ @Override
public void sendStartTask2Master(ProcessInstance processInstance, int taskId,
org.apache.dolphinscheduler.remote.command.CommandType taskType) {
String host = processInstance.getHost();
@@ -2869,6 +2976,7 @@ public class ProcessService {
stateEventCallbackService.sendResult(address, port, taskEventChangeCommand.convert2Command(taskType));
}
+ @Override
public ProcessInstance loadNextProcess4Serial(long code, int state) {
return this.processInstanceMapper.loadNextProcess4Serial(code, state);
}
@@ -2879,4 +2987,4 @@ public class ProcessService {
throw new ServiceException("delete command fail, id:" + commandId);
}
}
-}
+}
\ No newline at end of file
diff --git a/dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/process/ProcessServiceTest.java b/dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/process/ProcessServiceTest.java
index c57e6ba5c5..e8abd0692e 100644
--- a/dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/process/ProcessServiceTest.java
+++ b/dolphinscheduler-service/src/test/java/org/apache/dolphinscheduler/service/process/ProcessServiceTest.java
@@ -17,12 +17,7 @@
package org.apache.dolphinscheduler.service.process;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_RECOVER_PROCESS_ID_STRING;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_START_PARAMS;
-import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_SUB_PROCESS_DEFINE_CODE;
-
-import static org.mockito.ArgumentMatchers.any;
-
+import com.fasterxml.jackson.databind.JsonNode;
import org.apache.dolphinscheduler.common.Constants;
import org.apache.dolphinscheduler.common.enums.CommandType;
import org.apache.dolphinscheduler.common.enums.Flag;
@@ -78,14 +73,6 @@ import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
import org.apache.dolphinscheduler.service.exceptions.ServiceException;
import org.apache.dolphinscheduler.service.quartz.cron.CronUtilsTest;
import org.apache.dolphinscheduler.spi.params.base.FormType;
-
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.Date;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
import org.junit.Assert;
import org.junit.Rule;
import org.junit.Test;
@@ -99,7 +86,17 @@ import org.powermock.reflect.Whitebox;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
-import com.fasterxml.jackson.databind.JsonNode;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.Date;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_RECOVER_PROCESS_ID_STRING;
+import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_START_PARAMS;
+import static org.apache.dolphinscheduler.common.Constants.CMD_PARAM_SUB_PROCESS_DEFINE_CODE;
+import static org.mockito.ArgumentMatchers.any;
/**
* process service test
@@ -113,7 +110,7 @@ public class ProcessServiceTest {
public final ExpectedException exception = ExpectedException.none();
@InjectMocks
- private ProcessService processService;
+ private ProcessServiceImpl processService;
@Mock
private CommandMapper commandMapper;
@Mock
@@ -850,8 +847,8 @@ public class ProcessServiceTest {
int pageNumber = 0;
int masterCount = 0;
int thisMasterSlot = 2;
- List<Command> commandList = processService.findCommandPageBySlot(pageSize,pageNumber,masterCount,thisMasterSlot);
- Assert.assertEquals(0,commandList.size());
+ List<Command> commandList = processService.findCommandPageBySlot(pageSize, pageNumber, masterCount, thisMasterSlot);
+ Assert.assertEquals(0, commandList.size());
}
private TaskGroupQueue getTaskGroupQueue() {
diff --git a/dolphinscheduler-standalone-server/src/main/assembly/dolphinscheduler-standalone-server.xml b/dolphinscheduler-standalone-server/src/main/assembly/dolphinscheduler-standalone-server.xml
index a201d509db..e66fce1287 100644
--- a/dolphinscheduler-standalone-server/src/main/assembly/dolphinscheduler-standalone-server.xml
+++ b/dolphinscheduler-standalone-server/src/main/assembly/dolphinscheduler-standalone-server.xml
@@ -45,10 +45,6 @@
<directory>${basedir}/../dolphinscheduler-log-server/target/logger-server/libs</directory>
<outputDirectory>libs/logger-server</outputDirectory>
</fileSet>
- <fileSet>
- <directory>${basedir}/../dolphinscheduler-python/target/python-gateway-server/libs</directory>
- <outputDirectory>libs/python-gateway</outputDirectory>
- </fileSet>
<fileSet>
<directory>${basedir}/src/main/resources</directory>
diff --git a/dolphinscheduler-standalone-server/src/main/dist-bin/start.sh b/dolphinscheduler-standalone-server/src/main/dist-bin/start.sh
index 4ef3a11d36..019c61a7bf 100755
--- a/dolphinscheduler-standalone-server/src/main/dist-bin/start.sh
+++ b/dolphinscheduler-standalone-server/src/main/dist-bin/start.sh
@@ -28,7 +28,7 @@ if [[ "$DOCKER" == "true" ]]; then
fi
CP=$DOLPHINSCHEDULER_HOME/libs/standalone-server/*
-for d in alert-server api-server master-server python-gateway-server worker-server; do
+for d in alert-server api-server master-server worker-server; do
for f in $DOLPHINSCHEDULER_HOME/../$d/libs/*.jar; do
CP=$CP:$f
done
diff --git a/dolphinscheduler-standalone-server/src/main/resources/application.yaml b/dolphinscheduler-standalone-server/src/main/resources/application.yaml
index 7950d80177..a2894bfd2a 100644
--- a/dolphinscheduler-standalone-server/src/main/resources/application.yaml
+++ b/dolphinscheduler-standalone-server/src/main/resources/application.yaml
@@ -140,6 +140,8 @@ alert:
port: 50052
python-gateway:
+ # Weather enable python gateway server or not. The default value is true.
+ enabled: true
# The address of Python gateway server start. Set its value to `0.0.0.0` if your Python API run in different
# between Python gateway server. It could be be specific to other address like `127.0.0.1` or `localhost`
gateway-server-address: 0.0.0.0
diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/AbstractParameters.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/AbstractParameters.java
index 8146482793..743fe1b945 100644
--- a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/AbstractParameters.java
+++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/AbstractParameters.java
@@ -75,15 +75,14 @@ public abstract class AbstractParameters implements IParameters {
* @return parameters map
*/
public Map<String, Property> getLocalParametersMap() {
+ Map<String, Property> localParametersMaps = new LinkedHashMap<>();
if (localParams != null) {
- Map<String, Property> localParametersMaps = new LinkedHashMap<>();
for (Property property : localParams) {
localParametersMaps.put(property.getProp(),property);
}
- return localParametersMaps;
}
- return null;
+ return localParametersMaps;
}
/**
@@ -92,14 +91,13 @@ public abstract class AbstractParameters implements IParameters {
* @return parameters map
*/
public Map<String, Property> getVarPoolMap() {
+ Map<String, Property> varPoolMap = new LinkedHashMap<>();
if (varPool != null) {
- Map<String, Property> varPoolMap = new LinkedHashMap<>();
for (Property property : varPool) {
varPoolMap.put(property.getProp(), property);
}
- return varPoolMap;
}
- return null;
+ return varPoolMap;
}
public List<Property> getVarPool() {
diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/resource/AbstractResourceParameters.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/resource/AbstractResourceParameters.java
index 39193c8f2e..b719f95b4e 100644
--- a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/resource/AbstractResourceParameters.java
+++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/resource/AbstractResourceParameters.java
@@ -21,11 +21,12 @@ import com.fasterxml.jackson.annotation.JsonSubTypes;
import com.fasterxml.jackson.annotation.JsonSubTypes.Type;
import com.fasterxml.jackson.annotation.JsonTypeInfo;
-@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type")
+@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, visible = true, property = "resourceType")
@JsonSubTypes({
@Type(value = DataSourceParameters.class, name = "DATASOURCE"),
@Type(value = UdfFuncParameters.class, name = "UDF")
})
public abstract class AbstractResourceParameters {
+
}
diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/resource/DataSourceParameters.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/resource/DataSourceParameters.java
index 71d0dc0dad..ea9cb7ae5a 100644
--- a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/resource/DataSourceParameters.java
+++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/resource/DataSourceParameters.java
@@ -17,12 +17,24 @@
package org.apache.dolphinscheduler.plugin.task.api.parameters.resource;
+import com.fasterxml.jackson.annotation.JsonProperty;
import org.apache.dolphinscheduler.spi.enums.DbType;
public class DataSourceParameters extends AbstractResourceParameters {
private DbType type;
+ public String getResourceType() {
+ return resourceType;
+ }
+
+ public void setResourceType(String resourceType) {
+ this.resourceType = resourceType;
+ }
+
+ @JsonProperty(value = "DATASOURCE")
+ private String resourceType;
+
private String connectionParams;
public DbType getType() {
diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/resource/UdfFuncParameters.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/resource/UdfFuncParameters.java
index d1b1f36e47..f0ce772935 100644
--- a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/resource/UdfFuncParameters.java
+++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/resource/UdfFuncParameters.java
@@ -17,6 +17,7 @@
package org.apache.dolphinscheduler.plugin.task.api.parameters.resource;
+import com.fasterxml.jackson.annotation.JsonProperty;
import org.apache.dolphinscheduler.plugin.task.api.enums.UdfType;
import org.apache.dolphinscheduler.spi.utils.JSONUtils;
@@ -30,6 +31,18 @@ public class UdfFuncParameters extends AbstractResourceParameters {
* id
*/
private int id;
+
+ public String getResourceType() {
+ return resourceType;
+ }
+
+ public void setResourceType(String resourceType) {
+ this.resourceType = resourceType;
+ }
+
+ @JsonProperty(value = "UDF")
+ private String resourceType;
+
/**
* user id
*/
diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parser/ParamUtils.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parser/ParamUtils.java
index 876ec2123f..415bab633a 100644
--- a/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parser/ParamUtils.java
+++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parser/ParamUtils.java
@@ -63,9 +63,10 @@ public class ParamUtils {
// combining local and global parameters
Map<String, Property> localParams = parameters.getLocalParametersMap();
+ //stream pass params
Map<String, Property> varParams = parameters.getVarPoolMap();
- if (globalParams == null && localParams == null) {
+ if (globalParams.size() == 0 && localParams.size() == 0 && varParams.size() == 0) {
return null;
}
// if it is a complement,
@@ -85,15 +86,13 @@ public class ParamUtils {
}
params.put(PARAMETER_TASK_INSTANCE_ID, Integer.toString(taskExecutionContext.getTaskInstanceId()));
- if (globalParams != null && localParams != null) {
- globalParams.putAll(localParams);
- } else if (globalParams == null && localParams != null) {
- globalParams = localParams;
+ if (varParams.size() != 0) {
+ globalParams.putAll(varParams);
}
- if (varParams != null) {
- varParams.putAll(globalParams);
- globalParams = varParams;
+ if (localParams.size() != 0) {
+ globalParams.putAll(localParams);
}
+
Iterator<Map.Entry<String, Property>> iter = globalParams.entrySet().iterator();
while (iter.hasNext()) {
Map.Entry<String, Property> en = iter.next();
@@ -143,16 +142,15 @@ public class ParamUtils {
* @return parameters map
*/
public static Map<String, Property> getUserDefParamsMap(Map<String, String> definedParams) {
+ Map<String, Property> userDefParamsMaps = new HashMap<>();
if (definedParams != null) {
- Map<String, Property> userDefParamsMaps = new HashMap<>();
Iterator<Map.Entry<String, String>> iter = definedParams.entrySet().iterator();
while (iter.hasNext()) {
Map.Entry<String, String> en = iter.next();
Property property = new Property(en.getKey(), Direct.IN, DataType.VARCHAR, en.getValue());
userDefParamsMaps.put(property.getProp(),property);
}
- return userDefParamsMaps;
}
- return null;
+ return userDefParamsMaps;
}
}
\ No newline at end of file
diff --git a/dolphinscheduler-task-plugin/dolphinscheduler-task-sql/src/main/java/org/apache/dolphinscheduler/plugin/task/sql/SqlTask.java b/dolphinscheduler-task-plugin/dolphinscheduler-task-sql/src/main/java/org/apache/dolphinscheduler/plugin/task/sql/SqlTask.java
index e85ccc6432..ad31202004 100644
--- a/dolphinscheduler-task-plugin/dolphinscheduler-task-sql/src/main/java/org/apache/dolphinscheduler/plugin/task/sql/SqlTask.java
+++ b/dolphinscheduler-task-plugin/dolphinscheduler-task-sql/src/main/java/org/apache/dolphinscheduler/plugin/task/sql/SqlTask.java
@@ -243,7 +243,11 @@ public class SqlTask extends AbstractTaskExecutor {
int rowCount = 0;
int limit = sqlParameters.getLimit() == 0 ? QUERY_LIMIT : sqlParameters.getLimit();
- while (rowCount < limit && resultSet.next()) {
+ while (resultSet.next()) {
+ if (rowCount == limit) {
+ logger.info("sql result limit : {} exceeding results are filtered", limit);
+ break;
+ }
ObjectNode mapOfColValues = JSONUtils.createObjectNode();
for (int i = 1; i <= num; i++) {
mapOfColValues.set(md.getColumnLabel(i), JSONUtils.toJsonNode(resultSet.getObject(i)));
@@ -252,17 +256,12 @@ public class SqlTask extends AbstractTaskExecutor {
rowCount++;
}
int displayRows = sqlParameters.getDisplayRows() > 0 ? sqlParameters.getDisplayRows() : TaskConstants.DEFAULT_DISPLAY_ROWS;
- displayRows = Math.min(displayRows, resultJSONArray.size());
+ displayRows = Math.min(displayRows, rowCount);
logger.info("display sql result {} rows as follows:", displayRows);
for (int i = 0; i < displayRows; i++) {
String row = JSONUtils.toJsonString(resultJSONArray.get(i));
logger.info("row {} : {}", i + 1, row);
}
- if (resultSet.next()) {
- logger.info("sql result limit : {} exceeding results are filtered", limit);
- String log = String.format("sql result limit : %d exceeding results are filtered", limit);
- resultJSONArray.add(JSONUtils.toJsonNode(log));
- }
}
String result = JSONUtils.toJsonString(resultJSONArray);
if (sqlParameters.getSendEmail() == null || sqlParameters.getSendEmail()) {
diff --git a/dolphinscheduler-ui-next/pom.xml b/dolphinscheduler-ui-next/pom.xml
index 8dff740557..83646efb20 100644
--- a/dolphinscheduler-ui-next/pom.xml
+++ b/dolphinscheduler-ui-next/pom.xml
@@ -30,7 +30,7 @@
<properties>
<node.version>v16.13.1</node.version>
- <npm.version>8.1.2</npm.version>
+ <pnpm.version>v6.32.6</pnpm.version>
<sonar.sources>src</sonar.sources>
</properties>
@@ -45,19 +45,19 @@
<version>${frontend-maven-plugin.version}</version>
<executions>
<execution>
- <id>install node and npm</id>
+ <id>install node and pnpm</id>
<goals>
- <goal>install-node-and-npm</goal>
+ <goal>install-node-and-pnpm</goal>
</goals>
<configuration>
<nodeVersion>${node.version}</nodeVersion>
- <npmVersion>${npm.version}</npmVersion>
+ <pnpmVersion>${pnpm.version}</pnpmVersion>
</configuration>
</execution>
<execution>
- <id>npm install</id>
+ <id>pnpm install</id>
<goals>
- <goal>npm</goal>
+ <goal>pnpm</goal>
</goals>
<phase>generate-resources</phase>
<configuration>
@@ -65,9 +65,9 @@
</configuration>
</execution>
<execution>
- <id>npm run build:prod</id>
+ <id>pnpm run build:prod</id>
<goals>
- <goal>npm</goal>
+ <goal>pnpm</goal>
</goals>
<configuration>
<arguments>run build:prod</arguments>
diff --git a/dolphinscheduler-ui-next/src/components/chart/index.ts b/dolphinscheduler-ui-next/src/components/chart/index.ts
index 3522e76341..9edd075537 100644
--- a/dolphinscheduler-ui-next/src/components/chart/index.ts
+++ b/dolphinscheduler-ui-next/src/components/chart/index.ts
@@ -38,7 +38,7 @@ function initChart<Opt extends ECBasicOption>(
const init = () => {
chart = globalProperties?.echarts.init(
domRef.value,
- themeStore.darkTheme && 'dark-bold'
+ themeStore.darkTheme ? 'dark-bold' : 'macarons'
)
chart && chart.setOption(option)
}
diff --git a/dolphinscheduler-ui-next/src/locales/modules/en_US.ts b/dolphinscheduler-ui-next/src/locales/modules/en_US.ts
index d1a5fb2552..dec8f5b1fb 100644
--- a/dolphinscheduler-ui-next/src/locales/modules/en_US.ts
+++ b/dolphinscheduler-ui-next/src/locales/modules/en_US.ts
@@ -1113,7 +1113,7 @@ const security = {
Secret: 'Secret',
users: 'Users',
userSendMsg: 'UserSendMsg',
- agentId: 'AgentId',
+ 'agentId/chatId': 'AgentId or ChatId',
showType: 'Show Type',
receivers: 'Receivers',
receiverCcs: 'ReceiverCcs',
diff --git a/dolphinscheduler-ui-next/src/locales/modules/zh_CN.ts b/dolphinscheduler-ui-next/src/locales/modules/zh_CN.ts
index b4f59ea649..77a356414b 100644
--- a/dolphinscheduler-ui-next/src/locales/modules/zh_CN.ts
+++ b/dolphinscheduler-ui-next/src/locales/modules/zh_CN.ts
@@ -1098,7 +1098,7 @@ const security = {
Secret: '密钥',
users: '群员',
userSendMsg: '群员信息',
- agentId: '应用ID',
+ 'agentId/chatId': '应用ID或群聊ID',
showType: '内容展示类型',
receivers: '收件人',
receiverCcs: '抄送人',
diff --git a/pom.xml b/pom.xml
index 2b704acc0e..50b2019acd 100644
--- a/pom.xml
+++ b/pom.xml
@@ -93,7 +93,7 @@
<spotbugs.version>3.1.12</spotbugs.version>
<checkstyle.version>3.1.2</checkstyle.version>
<curator.test>2.12.0</curator.test>
- <frontend-maven-plugin.version>1.6</frontend-maven-plugin.version>
+ <frontend-maven-plugin.version>1.12.1</frontend-maven-plugin.version>
<maven-compiler-plugin.version>3.3</maven-compiler-plugin.version>
<maven-assembly-plugin.version>3.3.0</maven-assembly-plugin.version>
<maven-release-plugin.version>2.5.3</maven-release-plugin.version>
diff --git a/script/dolphinscheduler-daemon.sh b/script/dolphinscheduler-daemon.sh
index 61e732590c..1ded5074fd 100755
--- a/script/dolphinscheduler-daemon.sh
+++ b/script/dolphinscheduler-daemon.sh
@@ -16,7 +16,7 @@
# limitations under the License.
#
-usage="Usage: dolphinscheduler-daemon.sh (start|stop|status) <api-server|master-server|worker-server|alert-server|python-gateway-server|standalone-server> "
+usage="Usage: dolphinscheduler-daemon.sh (start|stop|status) <api-server|master-server|worker-server|alert-server|standalone-server> "
# if no args specified, show usage
if [ $# -le 1 ]; then
@@ -61,8 +61,6 @@ elif [ "$command" = "alert-server" ]; then
log=$DOLPHINSCHEDULER_HOME/alert-server/logs/$command-$HOSTNAME.out
elif [ "$command" = "standalone-server" ]; then
log=$DOLPHINSCHEDULER_HOME/standalone-server/logs/$command-$HOSTNAME.out
-elif [ "$command" = "python-gateway-server" ]; then
- log=$DOLPHINSCHEDULER_HOME/python-gateway-server/logs/$command-$HOSTNAME.out
else
echo "Error: No command named '$command' was found."
exit 1
diff --git a/script/env/install_env.sh b/script/env/install_env.sh
index 2df11c14ae..8b68c0ea00 100644
--- a/script/env/install_env.sh
+++ b/script/env/install_env.sh
@@ -48,11 +48,6 @@ alertServer=${alertServer:-"ds3"}
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers=${apiServers:-"ds1"}
-# A comma separated list of machine hostname or IP would be installed Python gateway server, it
-# must be a subset of configuration `ips`.
-# Example for hostname: pythonGatewayServers="ds1", Example for IP: pythonGatewayServers="192.168.8.1"
-pythonGatewayServers=${pythonGatewayServers:-"ds1"}
-
# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd)
installPath=${installPath:-"/tmp/dolphinscheduler"}
diff --git a/script/scp-hosts.sh b/script/scp-hosts.sh
index 1b216ab41e..2397ede936 100755
--- a/script/scp-hosts.sh
+++ b/script/scp-hosts.sh
@@ -49,7 +49,7 @@ do
echo "scp dirs to $host/$installPath starting"
ssh -p $sshPort $host "cd $installPath/; rm -rf bin/ conf/ lib/ script/ sql/ ui/"
- for dsDir in bin master-server worker-server alert-server api-server ui python-gateway-server
+ for dsDir in bin master-server worker-server alert-server api-server ui
do
# if worker in workersGroupMap
if [[ "${workersGroupMap[${host}]}" ]]; then
diff --git a/script/start-all.sh b/script/start-all.sh
index e7934da8c1..002f99d873 100755
--- a/script/start-all.sh
+++ b/script/start-all.sh
@@ -56,13 +56,6 @@ do
ssh -p $sshPort $apiServer "cd $installPath/; sh bin/dolphinscheduler-daemon.sh start api-server;"
done
-pythonGatewayHost=(${pythonGatewayServers//,/ })
-for pythonGatewayServer in "${pythonGatewayHost[@]}"
-do
- echo "$pythonGatewayServer python gateway server is starting"
- ssh -p $sshPort $pythonGatewayServer "cd $installPath/; sh bin/dolphinscheduler-daemon.sh start python-gateway-server;"
-done
-
# query server status
echo "query server status"
cd $installPath/; sh bin/status-all.sh
diff --git a/script/status-all.sh b/script/status-all.sh
index c3b4591a62..df19eb9c37 100755
--- a/script/status-all.sh
+++ b/script/status-all.sh
@@ -74,11 +74,3 @@ do
apiState=`ssh -p $sshPort $apiServer "cd $installPath/; sh bin/dolphinscheduler-daemon.sh status api-server;"`
echo "$apiServer $apiState"
done
-
-# python gateway server check state
-pythonGatewayHost=(${pythonGatewayServers//,/ })
-for pythonGatewayServer in "${pythonGatewayHost[@]}"
-do
- pythonGatewayState=`ssh -p $sshPort $pythonGatewayServer "cd $installPath/; sh bin/dolphinscheduler-daemon.sh status python-gateway-server;"`
- echo "$pythonGatewayServer $pythonGatewayState"
-done
diff --git a/script/stop-all.sh b/script/stop-all.sh
index 856340c390..6592374659 100755
--- a/script/stop-all.sh
+++ b/script/stop-all.sh
@@ -54,10 +54,3 @@ do
echo "$apiServer api server is stopping"
ssh -p $sshPort $apiServer "cd $installPath/; sh bin/dolphinscheduler-daemon.sh stop api-server;"
done
-
-pythonGatewayHost=(${pythonGatewayServers//,/ })
-for pythonGatewayServer in "${pythonGatewayHost[@]}"
-do
- echo "$pythonGatewayServer python gateway server is stopping"
- ssh -p $sshPort $pythonGatewayServer "cd $installPath/; sh bin/dolphinscheduler-daemon.sh stop python-gateway-server;"
-done