You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/04/11 02:34:45 UTC
[GitHub] [dolphinscheduler] luoyedeyi opened a new pull request, #9425: [Feature][plugin] Add k8s task in task plugin
luoyedeyi opened a new pull request, #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425
<!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
## Purpose of the pull request
<!--(For example: This pull request adds checkstyle plugin).-->
## Brief change log
<!--*(for example:)*
- *Add maven-checkstyle-plugin to root pom.xml*
-->
## Verify this pull request
<!--*(Please pick either of the following options)*-->
This pull request is code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
<!--*(example:)*
- *Added dolphinscheduler-dao tests for end-to-end.*
- *Added CronUtilsTest to verify the change.*
- *Manually verified the change by testing locally.* -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r870926775
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/ParameterUtils.java:
##########
@@ -0,0 +1,61 @@
+package org.apache.dolphinscheduler.plugin.task.api.utils;
+
+import org.apache.commons.lang.math.NumberUtils;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+
+public class ParameterUtils {
+ private static String NO_VALUE_KEY = "__NO_VALUE_KEY";
+ private static String DOUBLE_HYPHEN = "--";
+ private static String HYPHEN = "-";
+
+ public static Map fromArgs(String[] args) throws IllegalArgumentException {
Review Comment:
No need this part anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
devosend commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r873551450
##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-namespace.ts:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import type { IJsonItem } from '../types'
+import { useI18n } from 'vue-i18n'
+import { onMounted, ref, VNodeChild } from 'vue'
+import { queryNamespaceListPaging } from '@/service/modules/k8s-namespace'
+import { SelectOption } from 'naive-ui'
+
+export function useNamespace(): IJsonItem {
+ const { t } = useI18n()
+
+ const options = ref([])
+ const loading = ref(false)
+
+ const getNamespaceList = async () => {
+ if (loading.value) return
+ loading.value = true
+ const { totalList = [] } = await queryNamespaceListPaging({
+ pageNo: 1,
+ pageSize: 2147483647
Review Comment:
You should add a backend interface to get all the namespace, instead of using pageSize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1128458352
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [34.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1129759475
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [34.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1108005695
# [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/9425?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#9425](https://codecov.io/gh/apache/dolphinscheduler/pull/9425?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (688390d) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/29a0ea32c69df836bc3e5a5247810c55fd9bd775?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (29a0ea3) will **decrease** coverage by `0.11%`.
> The diff coverage is `27.62%`.
```diff
@@ Coverage Diff @@
## dev #9425 +/- ##
============================================
- Coverage 40.02% 39.90% -0.12%
- Complexity 4475 4505 +30
============================================
Files 835 845 +10
Lines 33550 33901 +351
Branches 3710 3738 +28
============================================
+ Hits 13427 13527 +100
- Misses 18875 19118 +243
- Partials 1248 1256 +8
```
| [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/9425?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...er/server/builder/TaskExecutionContextBuilder.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL2J1aWxkZXIvVGFza0V4ZWN1dGlvbkNvbnRleHRCdWlsZGVyLmphdmE=) | `68.88% <0.00%> (-3.21%)` | :arrow_down: |
| [...r/server/master/runner/task/BaseTaskProcessor.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvdGFzay9CYXNlVGFza1Byb2Nlc3Nvci5qYXZh) | `13.30% <0.00%> (-0.62%)` | :arrow_down: |
| [...duler/plugin/task/api/K8sTaskExecutionContext.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS9LOHNUYXNrRXhlY3V0aW9uQ29udGV4dC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...olphinscheduler/plugin/task/api/TaskConstants.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS9UYXNrQ29uc3RhbnRzLmphdmE=) | `0.00% <ø> (ø)` | |
| [...scheduler/plugin/task/api/k8s/AbstractK8sTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS9rOHMvQWJzdHJhY3RLOHNUYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ache/dolphinscheduler/plugin/task/k8s/K8sTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stazhzL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2s4cy9LOHNUYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...lphinscheduler/plugin/task/k8s/K8sTaskChannel.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stazhzL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2s4cy9LOHNUYXNrQ2hhbm5lbC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...heduler/plugin/task/k8s/K8sTaskChannelFactory.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stazhzL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2s4cy9LOHNUYXNrQ2hhbm5lbEZhY3RvcnkuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...lphinscheduler/plugin/task/api/utils/K8sUtils.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS91dGlscy9LOHNVdGlscy5qYXZh) | `3.50% <3.50%> (ø)` | |
| [...nscheduler/service/process/ProcessServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9425/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1zZXJ2aWNlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3NlcnZpY2UvcHJvY2Vzcy9Qcm9jZXNzU2VydmljZUltcGwuamF2YQ==) | `30.60% <20.00%> (+0.06%)` | :arrow_up: |
| ... and [14 more](https://codecov.io/gh/apache/dolphinscheduler/pull/9425/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/9425?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/9425?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [29a0ea3...688390d](https://codecov.io/gh/apache/dolphinscheduler/pull/9425?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1108378871
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [21 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [22.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [2.7% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1125634969
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [34.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r847209420
##########
dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/formModel/tasks/k8s.vue:
##########
@@ -0,0 +1,238 @@
+/*
Review Comment:
The UI part need to change in the ui-next.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1127458297
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [34.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] tongwl commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
tongwl commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r873450086
##########
dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts:
##########
@@ -533,6 +540,38 @@ export function formatModel(data: ITaskData) {
params.others = data.taskParams.sparkParameters.others
}
+ if (data.taskParams?.namespace) {
Review Comment:
Sure, thank you for the nice reminder, removed.
##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-k8s.ts:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { useCustomParams } from '.'
+import type { IJsonItem } from '../types'
+import { useI18n } from 'vue-i18n'
+import { onMounted, ref, VNodeChild } from 'vue'
+import { queryNamespaceListPaging } from '@/service/modules/k8s-namespace'
+import { SelectOption } from 'naive-ui'
+
+export function useK8s(model: { [field: string]: any }): IJsonItem[] {
+ const { t } = useI18n()
+
+ const options = ref([])
+ const loading = ref(false)
+
+ const getNamespaceList = async () => {
+ if (loading.value) return
+ loading.value = true
+ const { totalList = [] } = await queryNamespaceListPaging({
Review Comment:
sure, code changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] tongwl commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
tongwl commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r874305928
##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-namespace.ts:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import type { IJsonItem } from '../types'
+import { useI18n } from 'vue-i18n'
+import { onMounted, ref, VNodeChild } from 'vue'
+import { queryNamespaceListPaging } from '@/service/modules/k8s-namespace'
+import { SelectOption } from 'naive-ui'
+
+export function useNamespace(): IJsonItem {
+ const { t } = useI18n()
+
+ const options = ref([])
+ const loading = ref(false)
+
+ const getNamespaceList = async () => {
+ if (loading.value) return
+ loading.value = true
+ const { totalList = [] } = await queryNamespaceListPaging({
+ pageNo: 1,
+ pageSize: 2147483647
Review Comment:
Sure, has used get all namespaces api instead. @devosend
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1128507786
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [35.0% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r872528284
##########
docs/docs/en/guide/task/k8s.md:
##########
@@ -0,0 +1,43 @@
+# K8S Node
Review Comment:
when you add new task, you should also add it to sidebar in https://github.com/apache/dolphinscheduler/blob/dev/docs/configs/docsdev.js#L177
##########
docs/docs/en/guide/task/k8s.md:
##########
@@ -0,0 +1,43 @@
+# K8S Node
+
+## Overview
+
+K8S task type used to execute a batch task. In this task, the worker submits the task by using a k8s client.
+
+## Create Task
+
+- Click `Project -> Management-Project -> Name-Workflow Definition`, and click the `Create Workflow` button to enter the DAG editing page.
+- Drag from the toolbar <img src="/img/tasks/icons/k8s.png" width="15"/> to the canvas.
+
+## Task Parameter
Review Comment:
```suggestion
## Task Parameter
```
##########
docs/docs/en/guide/task/k8s.md:
##########
@@ -0,0 +1,43 @@
+# K8S Node
+
+## Overview
+
+K8S task type used to execute a batch task. In this task, the worker submits the task by using a k8s client.
+
+## Create Task
+
+- Click `Project -> Management-Project -> Name-Workflow Definition`, and click the `Create Workflow` button to enter the DAG editing page.
+- Drag from the toolbar <img src="/img/tasks/icons/k8s.png" width="15"/> to the canvas.
+
+## Task Parameter
+- **Node name**: The node name in a workflow definition is unique.
+- **Run flag**: Identifies whether this node schedules normally, if it does not need to execute, select the `prohibition execution`.
+- **Descriptive information**: Describe the function of the node.
+- **Task priority**: When the number of worker threads is insufficient, execute in the order of priority from high to low, and tasks with the same priority will execute in a first-in first-out order.
+- **Worker grouping**: Assign tasks to the machines of the worker group to execute. If `Default` is selected, randomly select a worker machine for execution.
+- **Environment Name**: Configure the environment name in which to run the task.
+- **Times of failed retry attempts**: The number of times the task failed to resubmit.
+- **Failed retry interval**: The time interval (unit minute) for resubmitting the task after a failed task.
+- **Delayed execution time**: The time (unit minute) that a task delays in execution.
+- **Timeout alarm**: Check the timeout alarm and timeout failure. When the task runs exceed the "timeout", an alarm email will send and the task execution will fail.
+- **Namespace**:：the namespace for running k8s task
+- **Min CPU**：min CPU requirement for running k8s task
+- **Min Memory**：min memory requirement for running k8s task
+- **Image**：the registry url for image
+- **Custom parameter**: It is a local user-defined parameter for K8S task, these params will pass to container as environment variables.
+- **Predecessor task**: Selecting a predecessor task for the current task, will set the selected predecessor task as upstream of the current task.
+## Task Example
+
+#### Configure the K8S Environment in DolphinScheduler
Review Comment:
```suggestion
### Configure the K8S Environment in DolphinScheduler
```
##########
docs/docs/zh/guide/task/k8s.md:
##########
@@ -0,0 +1,46 @@
+# K8S
+
+## 综述
+
+K8S任务类型，用于在K8S上执行一个短时和批处理的任务。worker最终会通过使用K8S client提交任务。
+
+
Review Comment:
```suggestion
```
##########
docs/docs/zh/guide/task/k8s.md:
##########
@@ -0,0 +1,46 @@
+# K8S
+
+## 综述
+
+K8S任务类型，用于在K8S上执行一个短时和批处理的任务。worker最终会通过使用K8S client提交任务。
+
+
+## 创建任务
+
+- 点击项目管理-项目名称-工作流定义，点击"创建工作流"按钮，进入DAG编辑页面。
+- 工具栏中拖动 <img src="/img/tasks/icons/k8s.png" width="25"/> 到画板中，选择需要连接的数据源，即可完成创建。
+
+## 任务参数
+
+- 节点名称：设置任务的名称。一个工作流定义中的节点名称是唯一的。
+- 运行标志：标识这个节点是否能正常调度,如果不需要执行，可以打开禁止执行开关。
+- 描述：描述该节点的功能。
+- 任务优先级：worker 线程数不足时，根据优先级从高到低依次执行，优先级一样时根据先进先出原则执行。
+- Worker 分组：任务分配给 worker 组的机器执行，选择 Default 会随机选择一台 worker 机执行。
+- 环境名称：配置运行任务的环境。
+- 失败重试次数：任务失败重新提交的次数。
+- 失败重试间隔：任务失败重新提交任务的时间间隔，以分为单位。
+- 延迟执行时间：任务延迟执行的时间，以分为单位。
+- 超时警告：勾选超时警告、超时失败，当任务超过“超时时长”后，会发送告警邮件并且任务执行失败。
+- 命名空间：选择K8S集群上存在的命名空间
+- 最小CPU：任务在K8S上运行所需的最小CPU
+- 最小内存：任务在K8S上运行所需的最小内存
+- 镜像：镜像地址
+- 自定义参数：K8S任务局部的用户自定义参数,自定义参数最终会通过环境变量形式存在于容器中,提供给K8S任务使用
+- 前置任务:在当前K8S任务之前需要执行的任务
+
+## 任务样例
+
+### 在 DolphinScheduler 中配置 K8S 集群环境
+
+若生产环境中要是使用到 K8s 任务类型，则需要预先配置好所需的K8S集群环境
+
+### 配置 K8S 任务节点
+
+根据上述参数说明，配置所需的内容即可。
+
+
Review Comment:
please only use English img in our document, you could use English img to Chinese doc too. BTW, change the img file name to `kubernetes-task.png`
##########
docs/docs/en/guide/task/k8s.md:
##########
@@ -0,0 +1,43 @@
+# K8S Node
Review Comment:
BTW, please use the keyword `kubernetes` instead of `K8S` in the document, more formal
##########
docs/docs/en/guide/task/k8s.md:
##########
@@ -0,0 +1,43 @@
+# K8S Node
+
+## Overview
+
+K8S task type used to execute a batch task. In this task, the worker submits the task by using a k8s client.
+
+## Create Task
+
+- Click `Project -> Management-Project -> Name-Workflow Definition`, and click the `Create Workflow` button to enter the DAG editing page.
+- Drag from the toolbar <img src="/img/tasks/icons/k8s.png" width="15"/> to the canvas.
+
+## Task Parameter
+- **Node name**: The node name in a workflow definition is unique.
+- **Run flag**: Identifies whether this node schedules normally, if it does not need to execute, select the `prohibition execution`.
+- **Descriptive information**: Describe the function of the node.
+- **Task priority**: When the number of worker threads is insufficient, execute in the order of priority from high to low, and tasks with the same priority will execute in a first-in first-out order.
+- **Worker grouping**: Assign tasks to the machines of the worker group to execute. If `Default` is selected, randomly select a worker machine for execution.
+- **Environment Name**: Configure the environment name in which to run the task.
+- **Times of failed retry attempts**: The number of times the task failed to resubmit.
+- **Failed retry interval**: The time interval (unit minute) for resubmitting the task after a failed task.
+- **Delayed execution time**: The time (unit minute) that a task delays in execution.
+- **Timeout alarm**: Check the timeout alarm and timeout failure. When the task runs exceed the "timeout", an alarm email will send and the task execution will fail.
+- **Namespace**:：the namespace for running k8s task
+- **Min CPU**：min CPU requirement for running k8s task
+- **Min Memory**：min memory requirement for running k8s task
+- **Image**：the registry url for image
+- **Custom parameter**: It is a local user-defined parameter for K8S task, these params will pass to container as environment variables.
+- **Predecessor task**: Selecting a predecessor task for the current task, will set the selected predecessor task as upstream of the current task.
+## Task Example
+
+#### Configure the K8S Environment in DolphinScheduler
+
+If you are using the K8S task type in a production environment, the K8S cluster environment is required.
+
+#### Configure K8S Nodes
Review Comment:
```suggestion
### Configure K8S Nodes
```
##########
docs/docs/en/guide/task/k8s.md:
##########
@@ -0,0 +1,43 @@
+# K8S Node
Review Comment:
Also change the file name too, thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] tongwl commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
tongwl commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r873449610
##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-k8s.ts:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { useCustomParams } from '.'
+import type { IJsonItem } from '../types'
+import { useI18n } from 'vue-i18n'
+import { onMounted, ref, VNodeChild } from 'vue'
+import { queryNamespaceListPaging } from '@/service/modules/k8s-namespace'
+import { SelectOption } from 'naive-ui'
+
+export function useK8s(model: { [field: string]: any }): IJsonItem[] {
+ const { t } = useI18n()
+
+ const options = ref([])
+ const loading = ref(false)
+
+ const getNamespaceList = async () => {
+ if (loading.value) return
+ loading.value = true
+ const { totalList = [] } = await queryNamespaceListPaging({
+ pageNo: 1,
+ pageSize: 2147483647
+ })
+ options.value = totalList.map(
+ (item: { id: string; namespace: string; k8s: string }) => ({
+ label: `${item.namespace}(${item.k8s})`,
+ value: JSON.stringify({
+ name: item.namespace,
+ cluster: item.k8s
+ })
+ })
+ )
+ loading.value = false
+ }
+
+ onMounted(() => {
+ getNamespaceList()
+ })
+
+ const renderLabel = (option: SelectOption): VNodeChild => {
+ if (option.type === 'group') return option.label as string
+ return [option.label as string]
+ }
+
+ return [
+ {
+ type: 'select',
+ field: 'namespace',
+ name: t('project.node.namespace_cluster'),
+ props: {
+ loading,
+ 'render-label': renderLabel
+ },
+ options: [
+ {
+ type: 'group',
+ label: t('project.node.namespace_cluster'),
+ key: t('project.node.namespace_cluster'),
+ children: options as any
+ }
+ ]
+ },
+ {
+ type: 'input-number',
+ field: 'minCpuCores',
+ span: 12,
+ name: t('project.node.min_cpu'),
+ slots: {
+ suffix: () => t('project.node.cores')
+ }
+ },
+ {
+ type: 'input-number',
+ field: 'minMemorySpace',
+ span: 12,
+ name: t('project.node.min_memory'),
+ slots: {
+ suffix: () => t('project.node.mb')
+ }
+ },
+ {
+ type: 'input',
+ field: 'image',
+ name: t('project.node.image'),
+ props: {
+ placeholder: t('project.node.image_tips')
+ },
+ validate: {
+ trigger: ['input', 'blur'],
+ message: t('project.node.min_memory_tips')
Review Comment:
Yes, thank you for the nice reminder, bug fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1108009091
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [22 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [22.1% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [2.7% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r870927391
##########
dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/formModel/tasks/k8s.vue:
##########
@@ -0,0 +1,238 @@
+/*
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1100861982
this pr close #9310
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1094505988
I had approve and run the CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] tongwl commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
tongwl commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r874305928
##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-namespace.ts:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import type { IJsonItem } from '../types'
+import { useI18n } from 'vue-i18n'
+import { onMounted, ref, VNodeChild } from 'vue'
+import { queryNamespaceListPaging } from '@/service/modules/k8s-namespace'
+import { SelectOption } from 'naive-ui'
+
+export function useNamespace(): IJsonItem {
+ const { t } = useI18n()
+
+ const options = ref([])
+ const loading = ref(false)
+
+ const getNamespaceList = async () => {
+ if (loading.value) return
+ loading.value = true
+ const { totalList = [] } = await queryNamespaceListPaging({
+ pageNo: 1,
+ pageSize: 2147483647
Review Comment:
Sure, has used get all namespaces api instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1122104887
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [34.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r870926287
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java:
##########
@@ -398,4 +403,19 @@ private TaskConstants() {
public static final String AWS_SECRET_ACCESS_KEY= "aws.secret.access.key";
public static final String AWS_REGION = "aws.region";
+ /**
+ * use for k8s task
+ */
+ public static final String API_VERSION = "batch/v1";
+ public static final String IMAGE_PULL_POLICY = "Always";
+ public static final String RESTART_POLICY = "Never";
Review Comment:
1. The api version is defined by the k8s dependency in task module
2. set the value of policy to Always to get latest image for each running
3. ds can retry task when the task failed, so there is no need to take k8s solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1123436498
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [34.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1125698729
PTAL @kezhenxu94 @zhongjiajie @zhuangchong
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r870926534
##########
dolphinscheduler-ui/src/js/conf/home/pages/dag/_source/formModel/tasks/k8s.vue:
##########
@@ -0,0 +1,238 @@
+/*
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r847185721
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/parameters/k8s/K8sParameters.java:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.api.parameters.k8s;
+
+import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * k8s task parameters
+ * @author hezhao2
Review Comment:
remove @author
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/k8s/AbstractK8sTaskExecutor.java:
##########
@@ -0,0 +1,191 @@
+package org.apache.dolphinscheduler.plugin.task.api.k8s;
Review Comment:
license header lack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng merged pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
caishunfeng merged PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
devosend commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r873551450
##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-namespace.ts:
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import type { IJsonItem } from '../types'
+import { useI18n } from 'vue-i18n'
+import { onMounted, ref, VNodeChild } from 'vue'
+import { queryNamespaceListPaging } from '@/service/modules/k8s-namespace'
+import { SelectOption } from 'naive-ui'
+
+export function useNamespace(): IJsonItem {
+ const { t } = useI18n()
+
+ const options = ref([])
+ const loading = ref(false)
+
+ const getNamespaceList = async () => {
+ if (loading.value) return
+ loading.value = true
+ const { totalList = [] } = await queryNamespaceListPaging({
+ pageNo: 1,
+ pageSize: 2147483647
Review Comment:
You should add a backend api to get all the namespace, instead of using pageSize.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1128357101
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [34.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r870926033
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java:
##########
@@ -398,4 +403,19 @@ private TaskConstants() {
public static final String AWS_SECRET_ACCESS_KEY= "aws.secret.access.key";
public static final String AWS_REGION = "aws.region";
+ /**
+ * use for k8s task
+ */
+ public static final String API_VERSION = "batch/v1";
+ public static final String IMAGE_PULL_POLICY = "Always";
+ public static final String RESTART_POLICY = "Never";
Review Comment:
1. The api version is limited by the k8s dependency in task module
2. set the value of policy to Always to get latest image for each running
3. ds can retry task when the task failed, so I believe there is no need to take k8s solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1125670300
I had rerun the fail CI job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1126225547
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [25 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [41.9% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [0.9% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r870927110
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/ParameterUtils.java:
##########
@@ -0,0 +1,61 @@
+package org.apache.dolphinscheduler.plugin.task.api.utils;
+
+import org.apache.commons.lang.math.NumberUtils;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+
+public class ParameterUtils {
+ private static String NO_VALUE_KEY = "__NO_VALUE_KEY";
+ private static String DOUBLE_HYPHEN = "--";
+ private static String HYPHEN = "-";
+
+ public static Map fromArgs(String[] args) throws IllegalArgumentException {
+ Map<String, String> map = new HashMap(args.length / 2);
+ int i = 0;
+ while (true) {
+ while (i < args.length) {
+ String key = getKeyFromArgs(args, i);
+ if (key.isEmpty()) {
+ throw new IllegalArgumentException("The input contains an empty argument");
+ }
+
+ ++i;
+ if (i >= args.length) {
+ map.put(key, NO_VALUE_KEY);
+ } else if (NumberUtils.isNumber(args[i])) {
+ map.put(key, args[i]);
+ ++i;
+ } else if (!args[i].startsWith(DOUBLE_HYPHEN) && !args[i].startsWith(HYPHEN)) {
+ map.put(key, args[i]);
+ ++i;
+ } else {
+ map.put(key, NO_VALUE_KEY);
+ }
+ }
+
+ return map;
+ }
+ }
+
+ public static String getKeyFromArgs(String[] args, int index) {
Review Comment:
Deleted this code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1124495809
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [5 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [34.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r870926033
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java:
##########
@@ -398,4 +403,19 @@ private TaskConstants() {
public static final String AWS_SECRET_ACCESS_KEY= "aws.secret.access.key";
public static final String AWS_REGION = "aws.region";
+ /**
+ * use for k8s task
+ */
+ public static final String API_VERSION = "batch/v1";
+ public static final String IMAGE_PULL_POLICY = "Always";
+ public static final String RESTART_POLICY = "Never";
Review Comment:
1. The api version is defined by the k8s dependency in task module
2. set the value of policy to Always to get latest image for each running
3. ds can retry task when the task failed, so there is no need to take k8s solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r870930179
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/k8s/AbstractK8sTaskExecutor.java:
##########
@@ -0,0 +1,191 @@
+package org.apache.dolphinscheduler.plugin.task.api.k8s;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_FAILURE;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_KILL;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_SUCCESS;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RUNNING_CODE;
+
+import org.apache.dolphinscheduler.plugin.task.api.K8sTaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.TaskException;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContextCacheManager;
+import org.apache.dolphinscheduler.plugin.task.api.enums.TaskTimeoutStrategy;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.utils.K8sUtils;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.Locale;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+
+import io.fabric8.kubernetes.api.model.batch.v1.Job;
+import io.fabric8.kubernetes.api.model.batch.v1.JobStatus;
+import io.fabric8.kubernetes.client.Watch;
+import io.fabric8.kubernetes.client.Watcher;
+import io.fabric8.kubernetes.client.WatcherException;
+
+
+public abstract class AbstractK8sTaskExecutor {
+ protected Logger logger;
+ protected TaskExecutionContext taskRequest;
+ protected K8sUtils k8sUtils;
+ protected Job job;
+ protected StringBuffer logStringBuffer;
+
+ public AbstractK8sTaskExecutor(Logger logger, TaskExecutionContext taskRequest) {
+ this.logger = logger;
+ this.taskRequest = taskRequest;
+ this.k8sUtils = new K8sUtils();
+ this.logStringBuffer = new StringBuffer();
+ }
+
+ public TaskResponse run(String k8sParameterStr) throws Exception {
+ TaskResponse result = new TaskResponse();
+ int taskInstanceId = taskRequest.getTaskInstanceId();
+ K8sTaskMainParameters k8STaskMainParameters = JSONUtils.parseObject(k8sParameterStr, K8sTaskMainParameters.class);
+ try {
+ if (null == TaskExecutionContextCacheManager.getByTaskInstanceId(taskInstanceId)) {
+ result.setExitStatusCode(EXIT_CODE_KILL);
+ return result;
+ }
+ if (StringUtils.isEmpty(k8sParameterStr)) {
+ TaskExecutionContextCacheManager.removeByTaskInstanceId(taskInstanceId);
+ return result;
+ }
+ K8sTaskExecutionContext k8sTaskExecutionContext = taskRequest.getK8sTaskExecutionContext();
+ String configYaml = k8sTaskExecutionContext.getConfigYaml();
+ k8sUtils.buildClient(configYaml);
+ submitJob2k8s(taskRequest, k8STaskMainParameters);
+ registerBatchJobWatcher(job, Integer.toString(taskInstanceId), result, k8STaskMainParameters);
+ } catch (Exception e) {
+ result.setExitStatusCode(EXIT_CODE_FAILURE);
+ throw e;
+ }
+ return result;
+ }
+
+ public void cancelApplication() {
+ K8sTaskMainParameters k8STaskMainParameters = JSONUtils.parseObject(taskRequest.getTaskParams(), K8sTaskMainParameters.class);
+ if (job != null) {
+ stopJobOnK8s(job.getMetadata().getName(), k8STaskMainParameters);
+ }
+ }
+
+ public void registerBatchJobWatcher(Job job, String taskInstanceId, TaskResponse taskResponse, K8sTaskMainParameters k8STaskMainParameters) {
+ CountDownLatch countDownLatch = new CountDownLatch(1);
+ Watcher watcher = new Watcher<Job>() {
+ @Override
+ public void eventReceived(Action action, Job job) {
+ if (action != Action.ADDED) {
+ int jobStatus = getK8sJobStatus(job);
+ if (jobStatus == EXIT_CODE_SUCCESS || jobStatus == EXIT_CODE_FAILURE) {
+ if (null == TaskExecutionContextCacheManager.getByTaskInstanceId(Integer.valueOf(taskInstanceId))) {
+ logStringBuffer.append(String.format("[K8sJobExecutor-%s] killed", job.getMetadata().getName()));
+ taskResponse.setExitStatusCode(EXIT_CODE_KILL);
+ } else if (jobStatus == EXIT_CODE_SUCCESS) {
+ logStringBuffer.append(String.format("[K8sJobExecutor-%s] succeed in k8s", job.getMetadata().getName()));
+ taskResponse.setExitStatusCode(EXIT_CODE_SUCCESS);
+ } else {
+ String errorMessage = k8sUtils.getPodLog(job.getMetadata().getName(), k8STaskMainParameters.getNamespace());
+ logStringBuffer.append(String.format("[K8sJobExecutor-%s] fail in k8s: %s", job.getMetadata().getName(), errorMessage));
+ taskResponse.setExitStatusCode(EXIT_CODE_FAILURE);
+ }
+ countDownLatch.countDown();
Review Comment:
The countdown only happens when job success or fail by accident.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r873507840
##########
docs/docs/zh/guide/task/k8s.md:
##########
@@ -0,0 +1,46 @@
+# K8S
+
+## 综述
+
+K8S任务类型，用于在K8S上执行一个短时和批处理的任务。worker最终会通过使用K8S client提交任务。
+
+
+## 创建任务
+
+- 点击项目管理-项目名称-工作流定义，点击"创建工作流"按钮，进入DAG编辑页面。
+- 工具栏中拖动 <img src="/img/tasks/icons/k8s.png" width="25"/> 到画板中，选择需要连接的数据源，即可完成创建。
+
+## 任务参数
+
+- 节点名称：设置任务的名称。一个工作流定义中的节点名称是唯一的。
+- 运行标志：标识这个节点是否能正常调度,如果不需要执行，可以打开禁止执行开关。
+- 描述：描述该节点的功能。
+- 任务优先级：worker 线程数不足时，根据优先级从高到低依次执行，优先级一样时根据先进先出原则执行。
+- Worker 分组：任务分配给 worker 组的机器执行，选择 Default 会随机选择一台 worker 机执行。
+- 环境名称：配置运行任务的环境。
+- 失败重试次数：任务失败重新提交的次数。
+- 失败重试间隔：任务失败重新提交任务的时间间隔，以分为单位。
+- 延迟执行时间：任务延迟执行的时间，以分为单位。
+- 超时警告：勾选超时警告、超时失败，当任务超过“超时时长”后，会发送告警邮件并且任务执行失败。
+- 命名空间：选择K8S集群上存在的命名空间
+- 最小CPU：任务在K8S上运行所需的最小CPU
+- 最小内存：任务在K8S上运行所需的最小内存
+- 镜像：镜像地址
+- 自定义参数：K8S任务局部的用户自定义参数,自定义参数最终会通过环境变量形式存在于容器中,提供给K8S任务使用
+- 前置任务:在当前K8S任务之前需要执行的任务
+
+## 任务样例
+
+### 在 DolphinScheduler 中配置 K8S 集群环境
+
+若生产环境中要是使用到 K8s 任务类型，则需要预先配置好所需的K8S集群环境
+
+### 配置 K8S 任务节点
+
+根据上述参数说明，配置所需的内容即可。
+
+
Review Comment:
Changed
##########
docs/docs/en/guide/task/k8s.md:
##########
@@ -0,0 +1,43 @@
+# K8S Node
Review Comment:
Changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r873507840
##########
docs/docs/zh/guide/task/k8s.md:
##########
@@ -0,0 +1,46 @@
+# K8S
+
+## 综述
+
+K8S任务类型，用于在K8S上执行一个短时和批处理的任务。worker最终会通过使用K8S client提交任务。
+
+
+## 创建任务
+
+- 点击项目管理-项目名称-工作流定义，点击"创建工作流"按钮，进入DAG编辑页面。
+- 工具栏中拖动 <img src="/img/tasks/icons/k8s.png" width="25"/> 到画板中，选择需要连接的数据源，即可完成创建。
+
+## 任务参数
+
+- 节点名称：设置任务的名称。一个工作流定义中的节点名称是唯一的。
+- 运行标志：标识这个节点是否能正常调度,如果不需要执行，可以打开禁止执行开关。
+- 描述：描述该节点的功能。
+- 任务优先级：worker 线程数不足时，根据优先级从高到低依次执行，优先级一样时根据先进先出原则执行。
+- Worker 分组：任务分配给 worker 组的机器执行，选择 Default 会随机选择一台 worker 机执行。
+- 环境名称：配置运行任务的环境。
+- 失败重试次数：任务失败重新提交的次数。
+- 失败重试间隔：任务失败重新提交任务的时间间隔，以分为单位。
+- 延迟执行时间：任务延迟执行的时间，以分为单位。
+- 超时警告：勾选超时警告、超时失败，当任务超过“超时时长”后，会发送告警邮件并且任务执行失败。
+- 命名空间：选择K8S集群上存在的命名空间
+- 最小CPU：任务在K8S上运行所需的最小CPU
+- 最小内存：任务在K8S上运行所需的最小内存
+- 镜像：镜像地址
+- 自定义参数：K8S任务局部的用户自定义参数,自定义参数最终会通过环境变量形式存在于容器中,提供给K8S任务使用
+- 前置任务:在当前K8S任务之前需要执行的任务
+
+## 任务样例
+
+### 在 DolphinScheduler 中配置 K8S 集群环境
+
+若生产环境中要是使用到 K8s 任务类型，则需要预先配置好所需的K8S集群环境
+
+### 配置 K8S 任务节点
+
+根据上述参数说明，配置所需的内容即可。
+
+
Review Comment:
Changed @zhongjiajie
##########
docs/docs/en/guide/task/k8s.md:
##########
@@ -0,0 +1,43 @@
+# K8S Node
Review Comment:
Changed @zhongjiajie
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1107998375
I had approve to run CI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1104691776
> this pr close #9310
@caishunfeng the keyword `close` only work in PR describe https://github.com/apache/dolphinscheduler/pull/9425#issue-1199308242
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1109262035
SonarCloud Quality Gate failed. [](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9425)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL) [6 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9425&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list) [22.1% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_coverage&view=list)
[](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list) [2.7% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9425&metric=new_duplicated_lines_density&view=list)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r870926033
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java:
##########
@@ -398,4 +403,19 @@ private TaskConstants() {
public static final String AWS_SECRET_ACCESS_KEY= "aws.secret.access.key";
public static final String AWS_REGION = "aws.region";
+ /**
+ * use for k8s task
+ */
+ public static final String API_VERSION = "batch/v1";
+ public static final String IMAGE_PULL_POLICY = "Always";
+ public static final String RESTART_POLICY = "Never";
Review Comment:
1. The api version is limited by the k8s dependency in task module
2. set the value of policy to Always to get latest image for each running
3. ds can retry task when the task failed, so there is no need to take k8s solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r870929267
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/k8s/AbstractK8sTaskExecutor.java:
##########
@@ -0,0 +1,191 @@
+package org.apache.dolphinscheduler.plugin.task.api.k8s;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_FAILURE;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_KILL;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_SUCCESS;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RUNNING_CODE;
+
+import org.apache.dolphinscheduler.plugin.task.api.K8sTaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.TaskException;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContextCacheManager;
+import org.apache.dolphinscheduler.plugin.task.api.enums.TaskTimeoutStrategy;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.utils.K8sUtils;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.Locale;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+
+import io.fabric8.kubernetes.api.model.batch.v1.Job;
+import io.fabric8.kubernetes.api.model.batch.v1.JobStatus;
+import io.fabric8.kubernetes.client.Watch;
+import io.fabric8.kubernetes.client.Watcher;
+import io.fabric8.kubernetes.client.WatcherException;
+
+
+public abstract class AbstractK8sTaskExecutor {
+ protected Logger logger;
+ protected TaskExecutionContext taskRequest;
+ protected K8sUtils k8sUtils;
+ protected Job job;
+ protected StringBuffer logStringBuffer;
+
+ public AbstractK8sTaskExecutor(Logger logger, TaskExecutionContext taskRequest) {
+ this.logger = logger;
+ this.taskRequest = taskRequest;
+ this.k8sUtils = new K8sUtils();
+ this.logStringBuffer = new StringBuffer();
+ }
+
+ public TaskResponse run(String k8sParameterStr) throws Exception {
+ TaskResponse result = new TaskResponse();
+ int taskInstanceId = taskRequest.getTaskInstanceId();
+ K8sTaskMainParameters k8STaskMainParameters = JSONUtils.parseObject(k8sParameterStr, K8sTaskMainParameters.class);
+ try {
+ if (null == TaskExecutionContextCacheManager.getByTaskInstanceId(taskInstanceId)) {
+ result.setExitStatusCode(EXIT_CODE_KILL);
+ return result;
+ }
+ if (StringUtils.isEmpty(k8sParameterStr)) {
+ TaskExecutionContextCacheManager.removeByTaskInstanceId(taskInstanceId);
+ return result;
+ }
+ K8sTaskExecutionContext k8sTaskExecutionContext = taskRequest.getK8sTaskExecutionContext();
+ String configYaml = k8sTaskExecutionContext.getConfigYaml();
+ k8sUtils.buildClient(configYaml);
+ submitJob2k8s(taskRequest, k8STaskMainParameters);
+ registerBatchJobWatcher(job, Integer.toString(taskInstanceId), result, k8STaskMainParameters);
+ } catch (Exception e) {
+ result.setExitStatusCode(EXIT_CODE_FAILURE);
+ throw e;
+ }
+ return result;
+ }
+
+ public void cancelApplication() {
+ K8sTaskMainParameters k8STaskMainParameters = JSONUtils.parseObject(taskRequest.getTaskParams(), K8sTaskMainParameters.class);
+ if (job != null) {
+ stopJobOnK8s(job.getMetadata().getName(), k8STaskMainParameters);
+ }
+ }
+
+ public void registerBatchJobWatcher(Job job, String taskInstanceId, TaskResponse taskResponse, K8sTaskMainParameters k8STaskMainParameters) {
+ CountDownLatch countDownLatch = new CountDownLatch(1);
Review Comment:
countDownLatch used to wait for job is completed in k8s and get the status of job. Once the job is finished in k8s, the running task worker will get corresponding result from watcher, then return the right task status to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] luoyedeyi commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
luoyedeyi commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r870926775
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/ParameterUtils.java:
##########
@@ -0,0 +1,61 @@
+package org.apache.dolphinscheduler.plugin.task.api.utils;
+
+import org.apache.commons.lang.math.NumberUtils;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+
+public class ParameterUtils {
+ private static String NO_VALUE_KEY = "__NO_VALUE_KEY";
+ private static String DOUBLE_HYPHEN = "--";
+ private static String HYPHEN = "-";
+
+ public static Map fromArgs(String[] args) throws IllegalArgumentException {
Review Comment:
Deleted this code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] devosend commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
devosend commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r873295503
##########
dolphinscheduler-ui/src/views/projects/task/components/node/format-data.ts:
##########
@@ -533,6 +540,38 @@ export function formatModel(data: ITaskData) {
params.others = data.taskParams.sparkParameters.others
}
+ if (data.taskParams?.namespace) {
Review Comment:
Lines 543-574 are redundant. You can delete it
##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-k8s.ts:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { useCustomParams } from '.'
+import type { IJsonItem } from '../types'
+import { useI18n } from 'vue-i18n'
+import { onMounted, ref, VNodeChild } from 'vue'
+import { queryNamespaceListPaging } from '@/service/modules/k8s-namespace'
+import { SelectOption } from 'naive-ui'
+
+export function useK8s(model: { [field: string]: any }): IJsonItem[] {
+ const { t } = useI18n()
+
+ const options = ref([])
+ const loading = ref(false)
+
+ const getNamespaceList = async () => {
+ if (loading.value) return
+ loading.value = true
+ const { totalList = [] } = await queryNamespaceListPaging({
Review Comment:
It's better to use a new interface to get all namespace here.
##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-k8s.ts:
##########
@@ -0,0 +1,108 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { useCustomParams } from '.'
+import type { IJsonItem } from '../types'
+import { useI18n } from 'vue-i18n'
+import { onMounted, ref, VNodeChild } from 'vue'
+import { queryNamespaceListPaging } from '@/service/modules/k8s-namespace'
+import { SelectOption } from 'naive-ui'
+
+export function useK8s(model: { [field: string]: any }): IJsonItem[] {
+ const { t } = useI18n()
+
+ const options = ref([])
+ const loading = ref(false)
+
+ const getNamespaceList = async () => {
+ if (loading.value) return
+ loading.value = true
+ const { totalList = [] } = await queryNamespaceListPaging({
+ pageNo: 1,
+ pageSize: 2147483647
+ })
+ options.value = totalList.map(
+ (item: { id: string; namespace: string; k8s: string }) => ({
+ label: `${item.namespace}(${item.k8s})`,
+ value: JSON.stringify({
+ name: item.namespace,
+ cluster: item.k8s
+ })
+ })
+ )
+ loading.value = false
+ }
+
+ onMounted(() => {
+ getNamespaceList()
+ })
+
+ const renderLabel = (option: SelectOption): VNodeChild => {
+ if (option.type === 'group') return option.label as string
+ return [option.label as string]
+ }
+
+ return [
+ {
+ type: 'select',
+ field: 'namespace',
+ name: t('project.node.namespace_cluster'),
+ props: {
+ loading,
+ 'render-label': renderLabel
+ },
+ options: [
+ {
+ type: 'group',
+ label: t('project.node.namespace_cluster'),
+ key: t('project.node.namespace_cluster'),
+ children: options as any
+ }
+ ]
+ },
+ {
+ type: 'input-number',
+ field: 'minCpuCores',
+ span: 12,
+ name: t('project.node.min_cpu'),
+ slots: {
+ suffix: () => t('project.node.cores')
+ }
+ },
+ {
+ type: 'input-number',
+ field: 'minMemorySpace',
+ span: 12,
+ name: t('project.node.min_memory'),
+ slots: {
+ suffix: () => t('project.node.mb')
+ }
+ },
+ {
+ type: 'input',
+ field: 'image',
+ name: t('project.node.image'),
+ props: {
+ placeholder: t('project.node.image_tips')
+ },
+ validate: {
+ trigger: ['input', 'blur'],
+ message: t('project.node.min_memory_tips')
Review Comment:
`min_memory_tips` is missed in locales language
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#discussion_r847195686
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/k8s/AbstractK8sTaskExecutor.java:
##########
@@ -0,0 +1,191 @@
+package org.apache.dolphinscheduler.plugin.task.api.k8s;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_FAILURE;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_KILL;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_SUCCESS;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RUNNING_CODE;
+
+import org.apache.dolphinscheduler.plugin.task.api.K8sTaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.TaskException;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContextCacheManager;
+import org.apache.dolphinscheduler.plugin.task.api.enums.TaskTimeoutStrategy;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.utils.K8sUtils;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.Locale;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+
+import io.fabric8.kubernetes.api.model.batch.v1.Job;
+import io.fabric8.kubernetes.api.model.batch.v1.JobStatus;
+import io.fabric8.kubernetes.client.Watch;
+import io.fabric8.kubernetes.client.Watcher;
+import io.fabric8.kubernetes.client.WatcherException;
+
+
+public abstract class AbstractK8sTaskExecutor {
+ protected Logger logger;
+ protected TaskExecutionContext taskRequest;
+ protected K8sUtils k8sUtils;
+ protected Job job;
+ protected StringBuffer logStringBuffer;
+
+ public AbstractK8sTaskExecutor(Logger logger, TaskExecutionContext taskRequest) {
+ this.logger = logger;
+ this.taskRequest = taskRequest;
+ this.k8sUtils = new K8sUtils();
+ this.logStringBuffer = new StringBuffer();
+ }
+
+ public TaskResponse run(String k8sParameterStr) throws Exception {
+ TaskResponse result = new TaskResponse();
+ int taskInstanceId = taskRequest.getTaskInstanceId();
+ K8sTaskMainParameters k8STaskMainParameters = JSONUtils.parseObject(k8sParameterStr, K8sTaskMainParameters.class);
+ try {
+ if (null == TaskExecutionContextCacheManager.getByTaskInstanceId(taskInstanceId)) {
+ result.setExitStatusCode(EXIT_CODE_KILL);
+ return result;
+ }
+ if (StringUtils.isEmpty(k8sParameterStr)) {
+ TaskExecutionContextCacheManager.removeByTaskInstanceId(taskInstanceId);
+ return result;
+ }
+ K8sTaskExecutionContext k8sTaskExecutionContext = taskRequest.getK8sTaskExecutionContext();
+ String configYaml = k8sTaskExecutionContext.getConfigYaml();
+ k8sUtils.buildClient(configYaml);
+ submitJob2k8s(taskRequest, k8STaskMainParameters);
+ registerBatchJobWatcher(job, Integer.toString(taskInstanceId), result, k8STaskMainParameters);
+ } catch (Exception e) {
+ result.setExitStatusCode(EXIT_CODE_FAILURE);
+ throw e;
+ }
+ return result;
+ }
+
+ public void cancelApplication() {
+ K8sTaskMainParameters k8STaskMainParameters = JSONUtils.parseObject(taskRequest.getTaskParams(), K8sTaskMainParameters.class);
+ if (job != null) {
+ stopJobOnK8s(job.getMetadata().getName(), k8STaskMainParameters);
+ }
+ }
+
+ public void registerBatchJobWatcher(Job job, String taskInstanceId, TaskResponse taskResponse, K8sTaskMainParameters k8STaskMainParameters) {
+ CountDownLatch countDownLatch = new CountDownLatch(1);
+ Watcher watcher = new Watcher<Job>() {
+ @Override
+ public void eventReceived(Action action, Job job) {
+ if (action != Action.ADDED) {
+ int jobStatus = getK8sJobStatus(job);
+ if (jobStatus == EXIT_CODE_SUCCESS || jobStatus == EXIT_CODE_FAILURE) {
+ if (null == TaskExecutionContextCacheManager.getByTaskInstanceId(Integer.valueOf(taskInstanceId))) {
+ logStringBuffer.append(String.format("[K8sJobExecutor-%s] killed", job.getMetadata().getName()));
+ taskResponse.setExitStatusCode(EXIT_CODE_KILL);
+ } else if (jobStatus == EXIT_CODE_SUCCESS) {
+ logStringBuffer.append(String.format("[K8sJobExecutor-%s] succeed in k8s", job.getMetadata().getName()));
+ taskResponse.setExitStatusCode(EXIT_CODE_SUCCESS);
+ } else {
+ String errorMessage = k8sUtils.getPodLog(job.getMetadata().getName(), k8STaskMainParameters.getNamespace());
+ logStringBuffer.append(String.format("[K8sJobExecutor-%s] fail in k8s: %s", job.getMetadata().getName(), errorMessage));
+ taskResponse.setExitStatusCode(EXIT_CODE_FAILURE);
+ }
+ countDownLatch.countDown();
Review Comment:
Will this count down more than one time?
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/ParameterUtils.java:
##########
@@ -0,0 +1,61 @@
+package org.apache.dolphinscheduler.plugin.task.api.utils;
+
+import org.apache.commons.lang.math.NumberUtils;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+
+public class ParameterUtils {
+ private static String NO_VALUE_KEY = "__NO_VALUE_KEY";
+ private static String DOUBLE_HYPHEN = "--";
+ private static String HYPHEN = "-";
+
+ public static Map fromArgs(String[] args) throws IllegalArgumentException {
Review Comment:
Please add some note or example for it.
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java:
##########
@@ -398,4 +403,19 @@ private TaskConstants() {
public static final String AWS_SECRET_ACCESS_KEY= "aws.secret.access.key";
public static final String AWS_REGION = "aws.region";
+ /**
+ * use for k8s task
+ */
+ public static final String API_VERSION = "batch/v1";
+ public static final String IMAGE_PULL_POLICY = "Always";
Review Comment:
Is it better to be configurable?
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/utils/ParameterUtils.java:
##########
@@ -0,0 +1,61 @@
+package org.apache.dolphinscheduler.plugin.task.api.utils;
+
+import org.apache.commons.lang.math.NumberUtils;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Map;
+
+public class ParameterUtils {
+ private static String NO_VALUE_KEY = "__NO_VALUE_KEY";
+ private static String DOUBLE_HYPHEN = "--";
+ private static String HYPHEN = "-";
+
+ public static Map fromArgs(String[] args) throws IllegalArgumentException {
+ Map<String, String> map = new HashMap(args.length / 2);
+ int i = 0;
+ while (true) {
+ while (i < args.length) {
+ String key = getKeyFromArgs(args, i);
+ if (key.isEmpty()) {
+ throw new IllegalArgumentException("The input contains an empty argument");
+ }
+
+ ++i;
+ if (i >= args.length) {
+ map.put(key, NO_VALUE_KEY);
+ } else if (NumberUtils.isNumber(args[i])) {
+ map.put(key, args[i]);
+ ++i;
+ } else if (!args[i].startsWith(DOUBLE_HYPHEN) && !args[i].startsWith(HYPHEN)) {
+ map.put(key, args[i]);
+ ++i;
+ } else {
+ map.put(key, NO_VALUE_KEY);
+ }
+ }
+
+ return map;
+ }
+ }
+
+ public static String getKeyFromArgs(String[] args, int index) {
Review Comment:
Please add some note or example for it.
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/k8s/AbstractK8sTaskExecutor.java:
##########
@@ -0,0 +1,191 @@
+package org.apache.dolphinscheduler.plugin.task.api.k8s;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_FAILURE;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_KILL;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_SUCCESS;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RUNNING_CODE;
+
+import org.apache.dolphinscheduler.plugin.task.api.K8sTaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.TaskException;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContextCacheManager;
+import org.apache.dolphinscheduler.plugin.task.api.enums.TaskTimeoutStrategy;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.utils.K8sUtils;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.Locale;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+
+import org.slf4j.Logger;
+
+import io.fabric8.kubernetes.api.model.batch.v1.Job;
+import io.fabric8.kubernetes.api.model.batch.v1.JobStatus;
+import io.fabric8.kubernetes.client.Watch;
+import io.fabric8.kubernetes.client.Watcher;
+import io.fabric8.kubernetes.client.WatcherException;
+
+
+public abstract class AbstractK8sTaskExecutor {
+ protected Logger logger;
+ protected TaskExecutionContext taskRequest;
+ protected K8sUtils k8sUtils;
+ protected Job job;
+ protected StringBuffer logStringBuffer;
+
+ public AbstractK8sTaskExecutor(Logger logger, TaskExecutionContext taskRequest) {
+ this.logger = logger;
+ this.taskRequest = taskRequest;
+ this.k8sUtils = new K8sUtils();
+ this.logStringBuffer = new StringBuffer();
+ }
+
+ public TaskResponse run(String k8sParameterStr) throws Exception {
+ TaskResponse result = new TaskResponse();
+ int taskInstanceId = taskRequest.getTaskInstanceId();
+ K8sTaskMainParameters k8STaskMainParameters = JSONUtils.parseObject(k8sParameterStr, K8sTaskMainParameters.class);
+ try {
+ if (null == TaskExecutionContextCacheManager.getByTaskInstanceId(taskInstanceId)) {
+ result.setExitStatusCode(EXIT_CODE_KILL);
+ return result;
+ }
+ if (StringUtils.isEmpty(k8sParameterStr)) {
+ TaskExecutionContextCacheManager.removeByTaskInstanceId(taskInstanceId);
+ return result;
+ }
+ K8sTaskExecutionContext k8sTaskExecutionContext = taskRequest.getK8sTaskExecutionContext();
+ String configYaml = k8sTaskExecutionContext.getConfigYaml();
+ k8sUtils.buildClient(configYaml);
+ submitJob2k8s(taskRequest, k8STaskMainParameters);
+ registerBatchJobWatcher(job, Integer.toString(taskInstanceId), result, k8STaskMainParameters);
+ } catch (Exception e) {
+ result.setExitStatusCode(EXIT_CODE_FAILURE);
+ throw e;
+ }
+ return result;
+ }
+
+ public void cancelApplication() {
+ K8sTaskMainParameters k8STaskMainParameters = JSONUtils.parseObject(taskRequest.getTaskParams(), K8sTaskMainParameters.class);
+ if (job != null) {
+ stopJobOnK8s(job.getMetadata().getName(), k8STaskMainParameters);
+ }
+ }
+
+ public void registerBatchJobWatcher(Job job, String taskInstanceId, TaskResponse taskResponse, K8sTaskMainParameters k8STaskMainParameters) {
+ CountDownLatch countDownLatch = new CountDownLatch(1);
Review Comment:
What does `countDownLatch` do? It seems multiple places countdown.
##########
dolphinscheduler-task-plugin/dolphinscheduler-task-api/src/main/java/org/apache/dolphinscheduler/plugin/task/api/TaskConstants.java:
##########
@@ -398,4 +403,19 @@ private TaskConstants() {
public static final String AWS_SECRET_ACCESS_KEY= "aws.secret.access.key";
public static final String AWS_REGION = "aws.region";
+ /**
+ * use for k8s task
+ */
+ public static final String API_VERSION = "batch/v1";
+ public static final String IMAGE_PULL_POLICY = "Always";
+ public static final String RESTART_POLICY = "Never";
Review Comment:
Is it better to be configurable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] davidzollo commented on pull request #9425: [Feature][plugin] Add k8s task in task plugin
Posted by GitBox <gi...@apache.org>.
davidzollo commented on PR #9425:
URL: https://github.com/apache/dolphinscheduler/pull/9425#issuecomment-1161809188
good job, thanks for your first contribution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org