You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/07/13 09:14:42 UTC

[GitHub] [dolphinscheduler] tracehh opened a new pull request, #10937: [Feature][Task] Support chunjun (FlinkX) task-backend

tracehh opened a new pull request, #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937

   <!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
   
   Fix: #10724 
   
   ## Purpose of the pull request
   
   <!--(For example: This pull request adds checkstyle plugin).-->
   
   ## Brief change log
   
   <!--*(for example:)*
     - *Add maven-checkstyle-plugin to root pom.xml*
   -->
   ## Verify this pull request
   
   <!--*(Please pick either of the following options)*-->
   
   This pull request is code cleanup without any test coverage.
   
   *(or)*
   
   This pull request is already covered by existing tests, such as *(please describe tests)*.
   
   (or)
   
   This change added tests and can be verified as follows:
   
   <!--*(example:)*
     - *Added dolphinscheduler-dao tests for end-to-end.*
     - *Added CronUtilsTest to verify the change.*
     - *Manually verified the change by testing locally.* -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1191567227

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![16.5%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '16.5%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [16.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![5.5%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '5.5%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [5.5% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task-backend

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r920624824


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/pom.xml:
##########
@@ -0,0 +1,57 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>dolphinscheduler-task-plugin</artifactId>
+        <groupId>org.apache.dolphinscheduler</groupId>
+        <version>dev-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <artifactId>dolphinscheduler-task-chunjun</artifactId>
+    <packaging>jar</packaging>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-datasource-all</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-spi</artifactId>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-api</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-datasource-api</artifactId>
+            <version>${project.version}</version>
+        </dependency>

Review Comment:
   dolphinscheduler-datasource-all and dolphinscheduler-datasource-api  are separate module



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

codecov-commenter commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1183926582

   # [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10937?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#10937](https://codecov.io/gh/apache/dolphinscheduler/pull/10937?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2a5d956) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/b245e7c973a5282a1405097605a4afacac1b5eab?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b245e7c) will **decrease** coverage by `0.15%`.
   > The diff coverage is `10.28%`.
   
   > :exclamation: Current head 2a5d956 differs from pull request most recent head 598c104. Consider uploading reports for the commit 598c104 to get more accurate results
   
   ```diff
   @@             Coverage Diff              @@
   ##                dev   #10937      +/-   ##
   ============================================
   - Coverage     40.30%   40.15%   -0.16%     
   - Complexity     4858     4866       +8     
   ============================================
     Files           943      949       +6     
     Lines         37015    37184     +169     
     Branches       4061     4070       +9     
   ============================================
   + Hits          14920    14932      +12     
   - Misses        20594    20746     +152     
   - Partials       1501     1506       +5     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/10937?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...cheduler/plugin/task/chunjun/ChunJunConstants.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10937/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stY2h1bmp1bi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZG9scGhpbnNjaGVkdWxlci9wbHVnaW4vdGFzay9jaHVuanVuL0NodW5KdW5Db25zdGFudHMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...phinscheduler/plugin/task/chunjun/ChunJunTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10937/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stY2h1bmp1bi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZG9scGhpbnNjaGVkdWxlci9wbHVnaW4vdGFzay9jaHVuanVuL0NodW5KdW5UYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...eduler/plugin/task/chunjun/ChunJunTaskChannel.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10937/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stY2h1bmp1bi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZG9scGhpbnNjaGVkdWxlci9wbHVnaW4vdGFzay9jaHVuanVuL0NodW5KdW5UYXNrQ2hhbm5lbC5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...plugin/task/chunjun/ChunJunTaskChannelFactory.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10937/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stY2h1bmp1bi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZG9scGhpbnNjaGVkdWxlci9wbHVnaW4vdGFzay9jaHVuanVuL0NodW5KdW5UYXNrQ2hhbm5lbEZhY3RvcnkuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...ugin/task/chunjun/ChunJunTaskExecutionContext.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10937/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stY2h1bmp1bi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZG9scGhpbnNjaGVkdWxlci9wbHVnaW4vdGFzay9jaHVuanVuL0NodW5KdW5UYXNrRXhlY3V0aW9uQ29udGV4dC5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...heduler/plugin/task/chunjun/ChunJunParameters.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10937/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stY2h1bmp1bi9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvZG9scGhpbnNjaGVkdWxlci9wbHVnaW4vdGFzay9jaHVuanVuL0NodW5KdW5QYXJhbWV0ZXJzLmphdmE=) | `25.71% <25.71%> (ø)` | |
   | [...e/dolphinscheduler/remote/NettyRemotingClient.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10937/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL05ldHR5UmVtb3RpbmdDbGllbnQuamF2YQ==) | `50.00% <0.00%> (-2.78%)` | :arrow_down: |
   | [...dolphinscheduler/remote/future/ResponseFuture.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10937/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL2Z1dHVyZS9SZXNwb25zZUZ1dHVyZS5qYXZh) | `81.96% <0.00%> (-1.64%)` | :arrow_down: |
   | [...rver/master/runner/task/BlockingTaskProcessor.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10937/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9ydW5uZXIvdGFzay9CbG9ja2luZ1Rhc2tQcm9jZXNzb3IuamF2YQ==) | `75.60% <0.00%> (-0.30%)` | :arrow_down: |
   | [...eduler/api/service/impl/DataSourceServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10937/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9EYXRhU291cmNlU2VydmljZUltcGwuamF2YQ==) | `29.49% <0.00%> (ø)` | |
   | ... and [5 more](https://codecov.io/gh/apache/dolphinscheduler/pull/10937/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10937?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10937?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b245e7c...598c104](https://codecov.io/gh/apache/dolphinscheduler/pull/10937?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

zhongjiajie commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r925523199


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunConstants.java:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+/**
+ * ChunJun constants
+ */
+public class ChunJunConstants {
+
+    public static final String FLINK_CONF_DIR = "${FLINK_HOME}/conf";
+
+    public static final String FLINK_LIB_DIR = "${FLINK_HOME}/lib";
+
+    public static final String HADOOP_CONF_DIR = "${HADOOP_HOME}/etc/hadoop";

Review Comment:
   I can not see you added related content in docs @tracehh 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] devosend commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

devosend commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r930851736


##########
dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-chunjun.ts:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { reactive } from 'vue'
+import * as Fields from '../fields/index'
+import type { IJsonItem, INodeData } from '../types'
+import { ITaskData } from '../types'
+
+export function useChunjun({
+  projectCode,
+  from = 0,
+  readonly,
+  data
+}: {
+  projectCode: number
+  from?: number
+  readonly?: boolean
+  data?: ITaskData
+}) {
+  const model = reactive({
+    name: '',
+    taskType: 'CHUNJUN',
+    flag: 'YES',
+    description: '',
+    deployMode: 'local',
+    timeoutFlag: false,
+    localParams: [],
+    environmentCode: null,
+    failRetryInterval: 1,
+    failRetryTimes: 0,
+    workerGroup: 'default',
+    cpuQuota: -1,
+    memoryMax: -1,

Review Comment:
   `memoryMax` is not used



##########
dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-chunjun.ts:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { reactive } from 'vue'
+import * as Fields from '../fields/index'
+import type { IJsonItem, INodeData } from '../types'
+import { ITaskData } from '../types'
+
+export function useChunjun({
+  projectCode,
+  from = 0,
+  readonly,
+  data
+}: {
+  projectCode: number
+  from?: number
+  readonly?: boolean
+  data?: ITaskData
+}) {
+  const model = reactive({
+    name: '',
+    taskType: 'CHUNJUN',
+    flag: 'YES',
+    description: '',
+    deployMode: 'local',
+    timeoutFlag: false,
+    localParams: [],
+    environmentCode: null,
+    failRetryInterval: 1,
+    failRetryTimes: 0,
+    workerGroup: 'default',
+    cpuQuota: -1,
+    memoryMax: -1,
+    delayTime: 0,
+    timeout: 30,
+    customConfig: false,
+    dsType: 'MYSQL',

Review Comment:
   `dsType` and `dtType` are not used in the current task component.



##########
dolphinscheduler-ui/src/views/projects/task/components/node/tasks/use-chunjun.ts:
##########
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { reactive } from 'vue'
+import * as Fields from '../fields/index'
+import type { IJsonItem, INodeData } from '../types'
+import { ITaskData } from '../types'
+
+export function useChunjun({
+  projectCode,
+  from = 0,
+  readonly,
+  data
+}: {
+  projectCode: number
+  from?: number
+  readonly?: boolean
+  data?: ITaskData
+}) {
+  const model = reactive({
+    name: '',
+    taskType: 'CHUNJUN',
+    flag: 'YES',
+    description: '',
+    deployMode: 'local',
+    timeoutFlag: false,
+    localParams: [],
+    environmentCode: null,
+    failRetryInterval: 1,
+    failRetryTimes: 0,
+    workerGroup: 'default',
+    cpuQuota: -1,

Review Comment:
   `cpuQuota` is not used



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1191186892

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![16.6%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '16.6%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [16.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![5.8%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '5.8%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [5.8% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] zhongjiajie commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

zhongjiajie commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r921269344


##########
docs/docs/en/guide/task/chunjun.md:
##########
@@ -0,0 +1,73 @@
+# CHUNJUN
+
+## Overview
+
+CHUNJUN task type for executing CHUNJUN programs. For CHUNJUN nodes, the worker will execute `${CHUNJUN_HOME}/bin/start-chunjun` to analyze the input json file.
+
+## Create Task
+
+- Click `Project Management -> Project Name -> Workflow Definition`, and click the `Create Workflow` button to enter the DAG editing page.
+- Drag the <img src="../../../../img/tasks/icons/chunjun.png" width="15"/> from the toolbar to the drawing board.
+
+## Task Parameters
+
+| **Parameter** | **Description** |
+| ------- | ---------- |
+| Node name | The node name in a workflow definition is unique. |
+| Run flag | Identifies whether this node schedules normally, if it does not need to execute, select the prohibition execution. |
+| Task priority | When the number of worker threads is insufficient, execute in the order of priority from high to low, and tasks with the same priority will execute in a first-in first-out order. |
+| Description | Describe the function of the node. |
+| Worker group | Assign tasks to the machines of the worker group to execute. If `Default` is selected, randomly select a worker machine for execution. |
+| Environment Name | Configure the environment name in which run the script. |
+| Number of failed retries | The number of times the task failed to resubmit. |
+| Failed retry interval | The time interval (unit minute) for resubmitting the task after a failed task. |
+| Task group name | The task group name. |
+| Priority | The task priority. |
+| Delayed execution time |  The time, in cents, that a task is delayed in execution. |
+| Timeout alarm | Check the timeout alarm and timeout failure. When the task exceeds the "timeout period", an alarm email will be sent and the task execution will fail. |
+| Custom template | Custom the content of the CHUNJUN node's json profile. |
+| json | json configuration file for CHUNJUN synchronization. |
+| Custom parameters | SQL task type, and stored procedure is a custom parameter order to set values for the method. The custom parameter type and data type are the same as the stored procedure task type. The difference is that the SQL task type custom parameter will replace the \${variable} in the SQL statement. |
+| Deploy mode | Execute chunjun task mode, eg local standalone. |
+| Option Parameters | Support such as `-confProp "{\"flink.checkpoint.interval\":60000}"` |
+| Predecessor task | Selecting a predecessor task for the current task will set the selected predecessor task as upstream of the current task. |
+
+## Task Example
+
+This example demonstrates importing data from Hive into MySQL.
+
+### Configuring the CHUNJUN environment in DolphinScheduler
+
+If you are using the CHUNJUN task type in a production environment, it is necessary to configure the required environment first. The configuration file is as follows: `/dolphinscheduler/conf/env/dolphinscheduler_env.sh`.
+
+![chunjun_task01](../../../../img/tasks/demo/chunjun_task01.png)
+
+After the environment has been configured, DolphinScheduler needs to be restarted.
+
+### Configuring CHUNJUN Task Node
+
+As the data to be read from Hive, a custom json is required, refer to: the template json in directory chunjun/chunjun-examples/json/hive.
+
+After writing the required json file, you can configure the node content by following the steps in the diagram below.
+
+![chunjun_task02](../../../../img/tasks/demo/chunjun_task02.png)
+
+### View run results
+
+![chunjun_task03](../../../../img/tasks/demo/chunjun_task03.png)
+
+### Note
+
+Before execute ${CHUNJUN_HOME}/bin/start-chunjun, need to change the shell ${CHUNJUN_HOME}/bin/start-chunjun, remove '&' in order to run in front. 
+
+ such as:
+
+```shell
+nohup $JAVA_RUN -cp $JAR_DIR $CLASS_NAME $@ &
+```
+
+update to following:
+
+```shell
+nohup $JAVA_RUN -cp $JAR_DIR $CLASS_NAME $@

Review Comment:
   It is seems you are using the different format in this for English and Chinses docs, could you migrate them to the same format?



##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunParameters.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.plugin.task.api.enums.ResourceType;
+import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * chunjun parameters
+ */
+public class ChunJunParameters extends AbstractParameters {

Review Comment:
   I wonder can we use `@Data` here to getter and setter?



##########
docs/docs/en/guide/task/chunjun.md:
##########
@@ -0,0 +1,73 @@
+# CHUNJUN
+
+## Overview
+
+CHUNJUN task type for executing CHUNJUN programs. For CHUNJUN nodes, the worker will execute `${CHUNJUN_HOME}/bin/start-chunjun` to analyze the input json file.

Review Comment:
   could you add some hyper link for `CHUNJUN`? and I wonder is it have to use uppercase for all characters?



##########
docs/docs/en/guide/task/chunjun.md:
##########
@@ -0,0 +1,73 @@
+# CHUNJUN
+
+## Overview
+
+CHUNJUN task type for executing CHUNJUN programs. For CHUNJUN nodes, the worker will execute `${CHUNJUN_HOME}/bin/start-chunjun` to analyze the input json file.

Review Comment:
   Add you have to add `ChuJun` to https://github.com/apache/dolphinscheduler/blob/81930e54208f7c0f305c0b9f8846149bfc38f428/docs/configs/docsdev.js#L603 and https://github.com/apache/dolphinscheduler/blob/81930e54208f7c0f305c0b9f8846149bfc38f428/docs/configs/docsdev.js#L183 to display it in the dolphinscheduler website sidebar



##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunConstants.java:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+/**
+ * ChunJun constants
+ */
+public class ChunJunConstants {
+
+    public static final String FLINK_CONF_DIR = "${FLINK_HOME}/conf";
+
+    public static final String FLINK_LIB_DIR = "${FLINK_HOME}/lib";
+
+    public static final String HADOOP_CONF_DIR = "${HADOOP_HOME}/etc/hadoop";

Review Comment:
   Does ChuJun have to run base on flink or hadoop? and have to config both flink and hadoop home? if so please add them to our docs



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1186797312

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![10.8%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '10.8%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [10.8% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![5.9%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '5.9%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [5.9% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r922923121


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunTaskChannelFactory.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.plugin.task.api.TaskChannel;
+import org.apache.dolphinscheduler.plugin.task.api.TaskChannelFactory;
+import org.apache.dolphinscheduler.spi.params.base.PluginParams;
+
+import java.util.List;
+
+import com.google.auto.service.AutoService;
+
+/**
+ * chunjun task channelFactory
+ */
+@AutoService(TaskChannelFactory.class)
+public class ChunJunTaskChannelFactory implements TaskChannelFactory {
+
+    @Override
+    public TaskChannel create() {
+        return new ChunJunTaskChannel();
+    }
+
+    /**
+     * plugin name
+     * Must be UNIQUE .
+     *
+     * @return this alert plugin name

Review Comment:
   ok update it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1184137603

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [22 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![24.9%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '24.9%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [24.9% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![7.8%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '7.8%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [7.8% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1183931657

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![10.2%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '10.2%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [10.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![14.2%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/20-16px.png '14.2%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [14.2% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] EricGao888 commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

EricGao888 commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r925579915


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunParameters.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.plugin.task.api.enums.ResourceType;
+import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * chunjun parameters
+ */
+public class ChunJunParameters extends AbstractParameters {

Review Comment:
   see: https://projectlombok.org/features/Data



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

zhongjiajie commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1195284392

   ping @songjianet @Amy0104 @devosend  again


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

zhongjiajie commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1197641880

   it is seem UI issue fix, could you please check again @devosend 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1184135279

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [22 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![24.9%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '24.9%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [24.9% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![7.8%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '7.8%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [7.8% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1183990659

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [19 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![23.7%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '23.7%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [23.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![8.1%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '8.1%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [8.1% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r921731738


##########
docs/docs/en/guide/task/chunjun.md:
##########
@@ -0,0 +1,73 @@
+# CHUNJUN
+
+## Overview
+
+CHUNJUN task type for executing CHUNJUN programs. For CHUNJUN nodes, the worker will execute `${CHUNJUN_HOME}/bin/start-chunjun` to analyze the input json file.

Review Comment:
   Thanks very much，update it



##########
docs/docs/en/guide/task/chunjun.md:
##########
@@ -0,0 +1,73 @@
+# CHUNJUN
+
+## Overview
+
+CHUNJUN task type for executing CHUNJUN programs. For CHUNJUN nodes, the worker will execute `${CHUNJUN_HOME}/bin/start-chunjun` to analyze the input json file.
+
+## Create Task
+
+- Click `Project Management -> Project Name -> Workflow Definition`, and click the `Create Workflow` button to enter the DAG editing page.
+- Drag the <img src="../../../../img/tasks/icons/chunjun.png" width="15"/> from the toolbar to the drawing board.
+
+## Task Parameters
+
+| **Parameter** | **Description** |
+| ------- | ---------- |
+| Node name | The node name in a workflow definition is unique. |
+| Run flag | Identifies whether this node schedules normally, if it does not need to execute, select the prohibition execution. |
+| Task priority | When the number of worker threads is insufficient, execute in the order of priority from high to low, and tasks with the same priority will execute in a first-in first-out order. |
+| Description | Describe the function of the node. |
+| Worker group | Assign tasks to the machines of the worker group to execute. If `Default` is selected, randomly select a worker machine for execution. |
+| Environment Name | Configure the environment name in which run the script. |
+| Number of failed retries | The number of times the task failed to resubmit. |
+| Failed retry interval | The time interval (unit minute) for resubmitting the task after a failed task. |
+| Task group name | The task group name. |
+| Priority | The task priority. |
+| Delayed execution time |  The time, in cents, that a task is delayed in execution. |
+| Timeout alarm | Check the timeout alarm and timeout failure. When the task exceeds the "timeout period", an alarm email will be sent and the task execution will fail. |
+| Custom template | Custom the content of the CHUNJUN node's json profile. |
+| json | json configuration file for CHUNJUN synchronization. |
+| Custom parameters | SQL task type, and stored procedure is a custom parameter order to set values for the method. The custom parameter type and data type are the same as the stored procedure task type. The difference is that the SQL task type custom parameter will replace the \${variable} in the SQL statement. |
+| Deploy mode | Execute chunjun task mode, eg local standalone. |
+| Option Parameters | Support such as `-confProp "{\"flink.checkpoint.interval\":60000}"` |
+| Predecessor task | Selecting a predecessor task for the current task will set the selected predecessor task as upstream of the current task. |
+
+## Task Example
+
+This example demonstrates importing data from Hive into MySQL.
+
+### Configuring the CHUNJUN environment in DolphinScheduler
+
+If you are using the CHUNJUN task type in a production environment, it is necessary to configure the required environment first. The configuration file is as follows: `/dolphinscheduler/conf/env/dolphinscheduler_env.sh`.
+
+![chunjun_task01](../../../../img/tasks/demo/chunjun_task01.png)
+
+After the environment has been configured, DolphinScheduler needs to be restarted.
+
+### Configuring CHUNJUN Task Node
+
+As the data to be read from Hive, a custom json is required, refer to: the template json in directory chunjun/chunjun-examples/json/hive.
+
+After writing the required json file, you can configure the node content by following the steps in the diagram below.
+
+![chunjun_task02](../../../../img/tasks/demo/chunjun_task02.png)
+
+### View run results
+
+![chunjun_task03](../../../../img/tasks/demo/chunjun_task03.png)
+
+### Note
+
+Before execute ${CHUNJUN_HOME}/bin/start-chunjun, need to change the shell ${CHUNJUN_HOME}/bin/start-chunjun, remove '&' in order to run in front. 
+
+ such as:
+
+```shell
+nohup $JAVA_RUN -cp $JAR_DIR $CLASS_NAME $@ &
+```
+
+update to following:
+
+```shell
+nohup $JAVA_RUN -cp $JAR_DIR $CLASS_NAME $@

Review Comment:
   update it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r921735186


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunParameters.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.plugin.task.api.enums.ResourceType;
+import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * chunjun parameters
+ */
+public class ChunJunParameters extends AbstractParameters {

Review Comment:
   ok update it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1196700824

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![16.5%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '16.5%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [16.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![5.9%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '5.9%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [5.9% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r932073082


##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-chunjun.ts:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { ref, onMounted, watch } from 'vue'
+import { useI18n } from 'vue-i18n'
+import type { IJsonItem } from '../types'
+import { useChunjunDeployMode } from './'
+
+export function useChunjun(model: { [field: string]: any }): IJsonItem[] {
+  const { t } = useI18n()
+
+  const jsonEditorSpan = ref(0)
+  const customParameterSpan = ref(0)
+
+  const initConstants = () => {
+    if (model.customConfig) {
+      jsonEditorSpan.value = 24
+      customParameterSpan.value = 24
+    } else {
+      jsonEditorSpan.value = 0
+      customParameterSpan.value = 0
+    }
+  }
+
+  onMounted(() => {
+    initConstants()
+  })
+
+  watch(
+    () => model.customConfig,
+    () => {
+      initConstants()
+    }
+  )
+
+  return [
+    {
+      type: 'switch',
+      field: 'customConfig',
+      value: true,
+      name: t('project.node.chunjun_custom_template'),
+      props: {
+        disabled: true

Review Comment:
   Default support custom template



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task-backend

Posted by GitBox <gi...@apache.org>.

tracehh commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1183851463

   > 
   
   
   
   > 
   
   front-end pr： https://github.com/apache/dolphinscheduler/pull/10946


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] EricGao888 commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

EricGao888 commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r925576950


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunParameters.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.plugin.task.api.enums.ResourceType;
+import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * chunjun parameters
+ */
+public class ChunJunParameters extends AbstractParameters {

Review Comment:
   Hi~ I recommend replacing `@Data` with `@Getter`, `@Setter` and `@ToString` because `@Data` will generate some code we may not need and cause the test coverage to decrease. Thx : )



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1191185611

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![16.5%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '16.5%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [16.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![5.8%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '5.8%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [5.8% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r926370681


##########
docs/docs/en/guide/task/chunjun.md:
##########
@@ -0,0 +1,73 @@
+# ChunJun
+
+## Overview
+
+ChunJun task type for executing ChunJun programs. For ChunJun nodes, the worker will execute `${CHUNJUN_HOME}/bin/start-chunjun` to analyze the input json file.
+
+## Create Task
+
+- Click `Project Management -> Project Name -> Workflow Definition`, and click the `Create Workflow` button to enter the DAG editing page.
+- Drag the <img src="../../../../img/tasks/icons/chunjun.png" width="15"/> from the toolbar to the drawing board.
+
+## Task Parameters
+
+| **Parameter** | **Description** |
+| ------- | ---------- |
+| Node name | The node name in a workflow definition is unique. |
+| Run flag | Identifies whether this node schedules normally, if it does not need to execute, select the prohibition execution. |
+| Task priority | When the number of worker threads is insufficient, execute in the order of priority from high to low, and tasks with the same priority will execute in a first-in first-out order. |
+| Description | Describe the function of the node. |
+| Worker group | Assign tasks to the machines of the worker group to execute. If `Default` is selected, randomly select a worker machine for execution. |
+| Environment Name | Configure the environment name in which run the script. |
+| Number of failed retries | The number of times the task failed to resubmit. |
+| Failed retry interval | The time interval (unit minute) for resubmitting the task after a failed task. |
+| Task group name | The task group name. |
+| Priority | The task priority. |
+| Delayed execution time |  The time, in cents, that a task is delayed in execution. |

Review Comment:
   means "in minutes"



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r926202030


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunParameters.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.plugin.task.api.enums.ResourceType;
+import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * chunjun parameters
+ */
+public class ChunJunParameters extends AbstractParameters {

Review Comment:
   Replaced @Data with @Getter, @Setter and @ToString



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature] Add new task type chunjun

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r932800324


##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-chunjun.ts:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { ref, onMounted, watch } from 'vue'
+import { useI18n } from 'vue-i18n'
+import type { IJsonItem } from '../types'
+import { useChunjunDeployMode } from './'
+
+export function useChunjun(model: { [field: string]: any }): IJsonItem[] {
+  const { t } = useI18n()
+
+  const jsonEditorSpan = ref(0)
+  const customParameterSpan = ref(0)
+
+  const initConstants = () => {
+    if (model.customConfig) {
+      jsonEditorSpan.value = 24
+      customParameterSpan.value = 24
+    } else {
+      jsonEditorSpan.value = 0
+      customParameterSpan.value = 0
+    }
+  }
+
+  onMounted(() => {
+    initConstants()
+  })
+
+  watch(
+    () => model.customConfig,
+    () => {
+      initConstants()
+    }
+  )
+
+  return [
+    {
+      type: 'switch',
+      field: 'customConfig',
+      value: true,
+      name: t('project.node.chunjun_custom_template'),
+      props: {
+        disabled: true

Review Comment:
   OK, get it, thanks very much @devosend 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] EricGao888 commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

EricGao888 commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r926177108


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunParameters.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.plugin.task.api.enums.ResourceType;
+import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * chunjun parameters
+ */
+public class ChunJunParameters extends AbstractParameters {

Review Comment:
   > SonarCloud Quality Gate failed.    [![Quality Gate failed](https://camo.githubusercontent.com/4ea51c1f64ee3746f631653a02ab678ca6a3efb5f5cb474402faed2e3dcf90b5/68747470733a2f2f736f6e6172736f757263652e6769746875622e696f2f736f6e6172636c6f75642d6769746875622d7374617469632d7265736f75726365732f76322f636865636b732f5175616c6974794761746542616467652f6661696c65642d313670782e706e67)](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   > 
   > [![Bug](https://camo.githubusercontent.com/4c6102327f5a954f9c8acaf2e2714183157a9e41717b371b2cd585cf25057310/68747470733a2f2f736f6e6172736f757263652e6769746875622e696f2f736f6e6172636c6f75642d6769746875622d7374617469632d7265736f75726365732f76322f636f6d6d6f6e2f6275672d313670782e706e67)](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://camo.githubusercontent.com/1cba125a897d7fa47033a3b3b2be2bbee680d34d4f004a215564659b853fb201/68747470733a2f2f736f6e6172736f757263652e6769746875622e696f2f736f6e6172636c6f75642d6769746875622d7374617469632d7265736f75726365732f76322f636865636b732f526174696e6742616467652f412d313670782e706e67)](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![Vulnerability](https://camo.githubusercontent.com/3ba1ee49636ffc3427e
 38649a9f8a65ee392f28e8a662fcf96ce24cefbb520e9/68747470733a2f2f736f6e6172736f757263652e6769746875622e696f2f736f6e6172636c6f75642d6769746875622d7374617469632d7265736f75726365732f76322f636f6d6d6f6e2f76756c6e65726162696c6974792d313670782e706e67)](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://camo.githubusercontent.com/1cba125a897d7fa47033a3b3b2be2bbee680d34d4f004a215564659b853fb201/68747470733a2f2f736f6e6172736f757263652e6769746875622e696f2f736f6e6172636c6f75642d6769746875622d7374617469632d7265736f75726365732f76322f636865636b732f526174696e6742616467652f412d313670782e706e67)](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![Security Hotspot](https://camo.githubusercontent.com/fb735cbe76f8d5e1679c7
 6ce83b740ceb1eaf62de4f7bf88623dc9953261aff7/68747470733a2f2f736f6e6172736f757263652e6769746875622e696f2f736f6e6172636c6f75642d6769746875622d7374617469632d7265736f75726365732f76322f636f6d6d6f6e2f73656375726974795f686f7473706f742d313670782e706e67)](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://camo.githubusercontent.com/ca3e5c9e7ad5fd04244d2d793976efbe479a024b145a815384556548a9884b5f/68747470733a2f2f736f6e6172736f757263652e6769746875622e696f2f736f6e6172636c6f75642d6769746875622d7374617469632d7265736f75726365732f76322f636865636b732f526174696e6742616467652f452d313670782e706e67)](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![Code Smell](https://camo.gi
 thubusercontent.com/8fe18b2dfb6f7d4e44582f281b29f617eb5ae07c248d2002ca586e91da219212/68747470733a2f2f736f6e6172736f757263652e6769746875622e696f2f736f6e6172636c6f75642d6769746875622d7374617469632d7265736f75726365732f76322f636f6d6d6f6e2f636f64655f736d656c6c2d313670782e706e67)](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://camo.githubusercontent.com/1cba125a897d7fa47033a3b3b2be2bbee680d34d4f004a215564659b853fb201/68747470733a2f2f736f6e6172736f757263652e6769746875622e696f2f736f6e6172636c6f75642d6769746875622d7374617469632d7265736f75726365732f76322f636865636b732f526174696e6742616467652f412d313670782e706e67)](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   > 
   > [![10.7%](https://camo.githubusercontent.com/3f04cff3eeef8477afe696ae55c570cbb6ed02f16152497c14251828329a3e91/68747470733a2f2f736f6e6172736f757263652e6769746875622e696f2f736f6e6172636c6f75642d6769746875622d7374617469632d7265736f75726365732f76322f636865636b732f436f76657261676543686172742f302d313670782e706e67)](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [10.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [![5.9%](https://camo.githubusercontent.com/6d10b2752bda1e1762d255930f4a4807428112b982706db9e339669f1165f525/68747470733a2f2f736f6e6172736f757263652e6769746875622e696f2f736f6e6172636c6f75642d6769746875622d7374617469632d7265736f75726365732f76322f636865636b732f4475706c69636174696f6e732f31302d313670782e706e67)](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density
 &view=list) [5.9% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   `Sonar` will probably no more fail for low test coverage once you fix this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1186798942

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![10.7%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '10.7%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [10.7% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![5.9%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '5.9%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [5.9% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] zhuxt2015 commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

zhuxt2015 commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r926316224


##########
docs/docs/zh/guide/task/chunjun.md:
##########
@@ -0,0 +1,47 @@
+# ChunJun节点
+
+## 综述
+
+ChunJun 任务类型，用于执行 ChunJun 程序。对于 ChunJun 节点，worker 会通过执行 `${CHUNJUN_HOME}/bin/start-chunjun` 来解析传入的 json 文件。
+
+## 创建任务
+
+- 点击项目管理 -> 项目名称 -> 工作流定义，点击“创建工作流”按钮，进入 DAG 编辑页面；
+- 拖动工具栏的<img src="../../../../img/tasks/icons/chunjun.png" width="15"/> 任务节点到画板中。
+
+## 任务参数
+
+- 节点名称：设置任务节点的名称。一个工作流定义中的节点名称是唯一的。
+- 运行标志：标识这个结点是否能正常调度，如果不需要执行，可以打开禁止执行开关。
+- 描述：描述该节点的功能。
+- 任务优先级：worker 线程数不足时，根据优先级从高到低依次执行，优先级一样时根据先进先出原则执行。
+- Worker 分组：任务分配给 worker 组的机器执行，选择 Default ，会随机选择一台 worker 机执行。
+- 环境名称：配置运行脚本的环境。
+- 任务组名称：任务组的名称。
+- 组内优先级：一个任务组内此任务的优先级。
+- 失败重试次数：任务失败重新提交的次数。
+- 失败重试间隔：任务失败重新提交任务的时间间隔，以分为单位。
+- 延时执行时间：任务延迟执行的时间，以分为单位。
+- 超时警告：勾选超时警告、超时失败，当任务超过“超时时长”后，会发送告警邮件并且任务执行失败。
+- 自定义模板：自定义 ChunJun 节点的 json 配置文件内容，当前支持此种方式。
+- json：ChunJun 同步的 json 配置文件。
+- 自定义参数：sql 任务类型，而存储过程是自定义参数顺序的给方法设置值自定义参数类型和数据类型同存储过程任务类型一样。区别在于SQL任务类型自定义参数会替换 sql 语句中 ${变量}。

Review Comment:
   Please optimize this sentence



##########
docs/docs/en/guide/task/chunjun.md:
##########
@@ -0,0 +1,73 @@
+# ChunJun
+
+## Overview
+
+ChunJun task type for executing ChunJun programs. For ChunJun nodes, the worker will execute `${CHUNJUN_HOME}/bin/start-chunjun` to analyze the input json file.
+
+## Create Task
+
+- Click `Project Management -> Project Name -> Workflow Definition`, and click the `Create Workflow` button to enter the DAG editing page.
+- Drag the <img src="../../../../img/tasks/icons/chunjun.png" width="15"/> from the toolbar to the drawing board.
+
+## Task Parameters
+
+| **Parameter** | **Description** |
+| ------- | ---------- |
+| Node name | The node name in a workflow definition is unique. |
+| Run flag | Identifies whether this node schedules normally, if it does not need to execute, select the prohibition execution. |
+| Task priority | When the number of worker threads is insufficient, execute in the order of priority from high to low, and tasks with the same priority will execute in a first-in first-out order. |
+| Description | Describe the function of the node. |
+| Worker group | Assign tasks to the machines of the worker group to execute. If `Default` is selected, randomly select a worker machine for execution. |
+| Environment Name | Configure the environment name in which run the script. |
+| Number of failed retries | The number of times the task failed to resubmit. |
+| Failed retry interval | The time interval (unit minute) for resubmitting the task after a failed task. |
+| Task group name | The task group name. |
+| Priority | The task priority. |
+| Delayed execution time |  The time, in cents, that a task is delayed in execution. |

Review Comment:
   `in cents` typos? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] devosend commented on a diff in pull request #10937: [Feature] Add new task type chunjun

Posted by GitBox <gi...@apache.org>.

devosend commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r932799027


##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-chunjun.ts:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { ref, onMounted, watch } from 'vue'
+import { useI18n } from 'vue-i18n'
+import type { IJsonItem } from '../types'
+import { useChunjunDeployMode } from './'
+
+export function useChunjun(model: { [field: string]: any }): IJsonItem[] {
+  const { t } = useI18n()
+
+  const jsonEditorSpan = ref(0)
+  const customParameterSpan = ref(0)
+
+  const initConstants = () => {
+    if (model.customConfig) {
+      jsonEditorSpan.value = 24
+      customParameterSpan.value = 24
+    } else {
+      jsonEditorSpan.value = 0
+      customParameterSpan.value = 0
+    }
+  }
+
+  onMounted(() => {
+    initConstants()
+  })
+
+  watch(
+    () => model.customConfig,
+    () => {
+      initConstants()
+    }
+  )
+
+  return [
+    {
+      type: 'switch',
+      field: 'customConfig',
+      value: true,
+      name: t('project.node.chunjun_custom_template'),
+      props: {
+        disabled: true

Review Comment:
   Instead of setting the default value, it will cause the Switch to be disabled and cannot be switched. You should delete it. @tracehh 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1185096764

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![10.3%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '10.3%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [10.3% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![5.8%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '5.8%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [5.8% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task-backend

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r920629367


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunTaskExecutionContext.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.spi.enums.DbType;
+
+import java.io.Serializable;
+
+/**
+ * chunjun  taskExecutionContext
+ */
+public class ChunJunTaskExecutionContext implements Serializable {

Review Comment:
   ok update it



##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunTask.java:
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_FAILURE;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.SystemUtils;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * chunjun task
+ */
+public class ChunJunTask extends AbstractTaskExecutor {
+    /**
+     * chunjun path
+     */
+    private static final String CHUNJUN_PATH = "${CHUNJUN_HOME}/bin/start-chunjun";
+
+    /**
+     * chunjun dist
+     */
+    private static final String CHUNJUN_DIST_DIR = "${CHUNJUN_HOME}/chunjun-dist";
+
+    /**
+     * chunJun parameters
+     */
+    private ChunJunParameters chunJunParameters;
+
+    /**
+     * shell command executor
+     */
+    private ShellCommandExecutor shellCommandExecutor;
+
+    /**
+     * taskExecutionContext
+     */
+    private TaskExecutionContext taskExecutionContext;
+
+    public ChunJunTask(TaskExecutionContext taskExecutionContext) {
+        super(taskExecutionContext);
+        this.taskExecutionContext = taskExecutionContext;
+
+        this.shellCommandExecutor = new ShellCommandExecutor(this::logHandle,
+            taskExecutionContext, logger);
+    }
+
+    /**
+     * init chunjun config
+     */
+    @Override
+    public void init() {
+        logger.info("chunjun task params {}", taskExecutionContext.getTaskParams());
+        chunJunParameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), ChunJunParameters.class);
+
+        if (!chunJunParameters.checkParameters()) {
+            throw new RuntimeException("chunjun task params is not valid");
+        }
+    }
+
+    /**
+     * run chunjun process
+     *
+     * @throws Exception exception
+     */
+    @Override
+    public void handle() throws Exception {
+        try {
+            Map<String, Property> paramsMap = taskExecutionContext.getPrepareParamsMap();
+
+            String jsonFilePath = buildChunJunJsonFile(paramsMap);
+            String shellCommandFilePath = buildShellCommandFile(jsonFilePath, paramsMap);
+            TaskResponse commandExecuteResult = shellCommandExecutor.run(shellCommandFilePath);
+
+            setExitStatusCode(commandExecuteResult.getExitStatusCode());
+            setAppIds(commandExecuteResult.getAppIds());
+            setProcessId(commandExecuteResult.getProcessId());
+        } catch (Exception e) {
+            logger.error("chunjun task failed.", e);
+            setExitStatusCode(EXIT_CODE_FAILURE);
+            throw e;
+        }
+    }
+
+    /**
+     * build chunjun json file
+     *
+     * @param paramsMap
+     * @return
+     * @throws Exception
+     */
+    private String buildChunJunJsonFile(Map<String, Property> paramsMap)
+        throws Exception {
+        // generate json
+        String fileName = String.format("%s/%s_job.json",
+            taskExecutionContext.getExecutePath(),
+            taskExecutionContext.getTaskAppId());
+
+        String json = null;
+
+        Path path = new File(fileName).toPath();
+        if (Files.exists(path)) {
+            return fileName;
+        }
+
+        if (chunJunParameters.getCustomConfig() == Flag.YES.ordinal()) {
+            json = chunJunParameters.getJson().replaceAll("\\r\\n", "\n");
+        }
+
+        // replace placeholder
+        json = ParameterUtils.convertParameterPlaceholders(json, ParamUtils.convert(paramsMap));
+
+        logger.debug("chunjun job json : {}", json);
+
+        // create chunjun json file
+        FileUtils.writeStringToFile(new File(fileName), json, StandardCharsets.UTF_8);
+        return fileName;
+    }
+
+
+    /**
+     * create command
+     *
+     * @return shell command file name
+     * @throws Exception if error throws Exception
+     */
+    private String buildShellCommandFile(String jobConfigFilePath, Map<String, Property> paramsMap)
+        throws Exception {
+        // generate scripts
+        String fileName = String.format("%s/%s_node.%s",
+            taskExecutionContext.getExecutePath(),
+            taskExecutionContext.getTaskAppId(),
+            SystemUtils.IS_OS_WINDOWS ? "bat" : "sh");
+
+        Path path = new File(fileName).toPath();
+
+        if (Files.exists(path)) {
+            return fileName;
+        }
+
+        // chunjun command
+        StringBuilder sbr = new StringBuilder();

Review Comment:
   ok update it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

SbloodyS commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r920711323


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunTask.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_FAILURE;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.SystemUtils;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * chunjun task
+ */
+public class ChunJunTask extends AbstractTaskExecutor {
+    /**
+     * chunjun path
+     */
+    private static final String CHUNJUN_PATH = "${CHUNJUN_HOME}/bin/start-chunjun";

Review Comment:
   We should put this environment variable to `script/env/dolphinscheduler_env.sh`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1183989024

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [19 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![23.6%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '23.6%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [23.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![8.1%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '8.1%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [8.1% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r921733318


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunConstants.java:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+/**
+ * ChunJun constants
+ */
+public class ChunJunConstants {
+
+    public static final String FLINK_CONF_DIR = "${FLINK_HOME}/conf";
+
+    public static final String FLINK_LIB_DIR = "${FLINK_HOME}/lib";
+
+    public static final String HADOOP_CONF_DIR = "${HADOOP_HOME}/etc/hadoop";

Review Comment:
   yarn-per-job based on flink or hadoop，HADOOP_HOME，FLINK_HOME had added



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1190982667

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [17 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![19.4%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '19.4%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [19.4% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![1.9%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '1.9%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [1.9% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1186673338

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![10.2%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '10.2%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [10.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![5.8%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '5.8%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [5.8% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1186883536

   @caishunfeng @songjianet @Amy0104 @Tianqi-Dotes @zhuangchong  please review it, thanks very much.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r928635890


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunConstants.java:
##########
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+/**
+ * ChunJun constants
+ */
+public class ChunJunConstants {
+
+    public static final String FLINK_CONF_DIR = "${FLINK_HOME}/conf";
+
+    public static final String FLINK_LIB_DIR = "${FLINK_HOME}/lib";
+
+    public static final String HADOOP_CONF_DIR = "${HADOOP_HOME}/etc/hadoop";

Review Comment:
   In  script/env/dolphinscheduler_env.sh
   
   export FLINK_HOME=${FLINK_HOME:-/opt/soft/flink}
   export DATAX_HOME=${DATAX_HOME:-/opt/soft/datax}
   export SEATUNNEL_HOME=${SEATUNNEL_HOME:-/opt/soft/seatunnel}
   export CHUNJUN_HOME=${CHUNJUN_HOME:-/opt/soft/chunjun}



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1191568987

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![16.6%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '16.6%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [16.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![5.5%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '5.5%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [5.5% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] devosend commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

devosend commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r932054649


##########
dolphinscheduler-ui/src/views/projects/task/components/node/fields/use-chunjun.ts:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import { ref, onMounted, watch } from 'vue'
+import { useI18n } from 'vue-i18n'
+import type { IJsonItem } from '../types'
+import { useChunjunDeployMode } from './'
+
+export function useChunjun(model: { [field: string]: any }): IJsonItem[] {
+  const { t } = useI18n()
+
+  const jsonEditorSpan = ref(0)
+  const customParameterSpan = ref(0)
+
+  const initConstants = () => {
+    if (model.customConfig) {
+      jsonEditorSpan.value = 24
+      customParameterSpan.value = 24
+    } else {
+      jsonEditorSpan.value = 0
+      customParameterSpan.value = 0
+    }
+  }
+
+  onMounted(() => {
+    initConstants()
+  })
+
+  watch(
+    () => model.customConfig,
+    () => {
+      initConstants()
+    }
+  )
+
+  return [
+    {
+      type: 'switch',
+      field: 'customConfig',
+      value: true,
+      name: t('project.node.chunjun_custom_template'),
+      props: {
+        disabled: true

Review Comment:
   Why do you set disabled to true?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] EricGao888 commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

EricGao888 commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r925580830


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunParameters.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.plugin.task.api.enums.ResourceType;
+import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * chunjun parameters
+ */
+public class ChunJunParameters extends AbstractParameters {

Review Comment:
   https://stackoverflow.com/questions/45569085/sonarqube-bad-coverage-because-of-lombok-data



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1186675041

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![10.3%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '10.3%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [10.3% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![5.8%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '5.8%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [5.8% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1186884282

   @caishunfeng @songjianet @Amy0104 @Tianqi-Dotes @zhuangchong please review it, thanks very much.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

kezhenxu94 commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r922759995


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunTaskChannelFactory.java:
##########
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.plugin.task.api.TaskChannel;
+import org.apache.dolphinscheduler.plugin.task.api.TaskChannelFactory;
+import org.apache.dolphinscheduler.spi.params.base.PluginParams;
+
+import java.util.List;
+
+import com.google.auto.service.AutoService;
+
+/**
+ * chunjun task channelFactory
+ */
+@AutoService(TaskChannelFactory.class)
+public class ChunJunTaskChannelFactory implements TaskChannelFactory {
+
+    @Override
+    public TaskChannel create() {
+        return new ChunJunTaskChannel();
+    }
+
+    /**
+     * plugin name
+     * Must be UNIQUE .
+     *
+     * @return this alert plugin name

Review Comment:
   This is not alert plugin, it's task plugin



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1196699574

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [7 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![16.6%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '16.6%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [16.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![5.9%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '5.9%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [5.9% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1190981625

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [17 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![19.4%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '19.4%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [19.4% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![1.9%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '1.9%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [1.9% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r921735422


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunParameters.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.plugin.task.api.enums.ResourceType;
+import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * chunjun parameters
+ */
+public class ChunJunParameters extends AbstractParameters {

Review Comment:
   ok update it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1185097412

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![10.2%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '10.2%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [10.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![5.8%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/10-16px.png '5.8%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [5.8% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task-backend

Posted by GitBox <gi...@apache.org>.

SbloodyS commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1183853992

   > front-end pr： #10946
   
   It's better to add front-end in this PR instead of separate it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1184195956

   had add docs @SbloodyS 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r920731589


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunTask.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_FAILURE;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.SystemUtils;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * chunjun task
+ */
+public class ChunJunTask extends AbstractTaskExecutor {
+    /**
+     * chunjun path
+     */
+    private static final String CHUNJUN_PATH = "${CHUNJUN_HOME}/bin/start-chunjun";

Review Comment:
   ok  update it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1184106190

   > 
   
   has add docs


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] ruanwenjun commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task-backend

Posted by GitBox <gi...@apache.org>.

ruanwenjun commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r919856920


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunTaskExecutionContext.java:
##########
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.spi.enums.DbType;
+
+import java.io.Serializable;
+
+/**
+ * chunjun  taskExecutionContext
+ */
+public class ChunJunTaskExecutionContext implements Serializable {

Review Comment:
   Please use lombok to generate get/set/toString
   ```suggestion
   @Data
   public class ChunJunTaskExecutionContext implements Serializable {
   ```



##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/pom.xml:
##########
@@ -0,0 +1,57 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>dolphinscheduler-task-plugin</artifactId>
+        <groupId>org.apache.dolphinscheduler</groupId>
+        <version>dev-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <artifactId>dolphinscheduler-task-chunjun</artifactId>
+    <packaging>jar</packaging>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-datasource-all</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-spi</artifactId>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-api</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-datasource-api</artifactId>
+            <version>${project.version}</version>
+        </dependency>

Review Comment:
   Can we remove this? since we already import `dolphinscheduler-datasource-all`



##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunTask.java:
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_FAILURE;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.SystemUtils;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * chunjun task
+ */
+public class ChunJunTask extends AbstractTaskExecutor {
+    /**
+     * chunjun path
+     */
+    private static final String CHUNJUN_PATH = "${CHUNJUN_HOME}/bin/start-chunjun";
+
+    /**
+     * chunjun dist
+     */
+    private static final String CHUNJUN_DIST_DIR = "${CHUNJUN_HOME}/chunjun-dist";
+
+    /**
+     * chunJun parameters
+     */
+    private ChunJunParameters chunJunParameters;
+
+    /**
+     * shell command executor
+     */
+    private ShellCommandExecutor shellCommandExecutor;
+
+    /**
+     * taskExecutionContext
+     */
+    private TaskExecutionContext taskExecutionContext;
+
+    public ChunJunTask(TaskExecutionContext taskExecutionContext) {
+        super(taskExecutionContext);
+        this.taskExecutionContext = taskExecutionContext;
+
+        this.shellCommandExecutor = new ShellCommandExecutor(this::logHandle,
+            taskExecutionContext, logger);
+    }
+
+    /**
+     * init chunjun config
+     */
+    @Override
+    public void init() {
+        logger.info("chunjun task params {}", taskExecutionContext.getTaskParams());
+        chunJunParameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), ChunJunParameters.class);
+
+        if (!chunJunParameters.checkParameters()) {
+            throw new RuntimeException("chunjun task params is not valid");
+        }
+    }
+
+    /**
+     * run chunjun process
+     *
+     * @throws Exception exception
+     */
+    @Override
+    public void handle() throws Exception {
+        try {
+            Map<String, Property> paramsMap = taskExecutionContext.getPrepareParamsMap();
+
+            String jsonFilePath = buildChunJunJsonFile(paramsMap);
+            String shellCommandFilePath = buildShellCommandFile(jsonFilePath, paramsMap);
+            TaskResponse commandExecuteResult = shellCommandExecutor.run(shellCommandFilePath);
+
+            setExitStatusCode(commandExecuteResult.getExitStatusCode());
+            setAppIds(commandExecuteResult.getAppIds());
+            setProcessId(commandExecuteResult.getProcessId());
+        } catch (Exception e) {
+            logger.error("chunjun task failed.", e);
+            setExitStatusCode(EXIT_CODE_FAILURE);
+            throw e;
+        }
+    }
+
+    /**
+     * build chunjun json file
+     *
+     * @param paramsMap
+     * @return
+     * @throws Exception
+     */
+    private String buildChunJunJsonFile(Map<String, Property> paramsMap)
+        throws Exception {
+        // generate json
+        String fileName = String.format("%s/%s_job.json",
+            taskExecutionContext.getExecutePath(),
+            taskExecutionContext.getTaskAppId());
+
+        String json = null;
+
+        Path path = new File(fileName).toPath();
+        if (Files.exists(path)) {
+            return fileName;
+        }
+
+        if (chunJunParameters.getCustomConfig() == Flag.YES.ordinal()) {
+            json = chunJunParameters.getJson().replaceAll("\\r\\n", "\n");
+        }
+
+        // replace placeholder
+        json = ParameterUtils.convertParameterPlaceholders(json, ParamUtils.convert(paramsMap));
+
+        logger.debug("chunjun job json : {}", json);
+
+        // create chunjun json file
+        FileUtils.writeStringToFile(new File(fileName), json, StandardCharsets.UTF_8);
+        return fileName;
+    }
+
+
+    /**
+     * create command
+     *
+     * @return shell command file name
+     * @throws Exception if error throws Exception
+     */
+    private String buildShellCommandFile(String jobConfigFilePath, Map<String, Property> paramsMap)
+        throws Exception {
+        // generate scripts
+        String fileName = String.format("%s/%s_node.%s",
+            taskExecutionContext.getExecutePath(),
+            taskExecutionContext.getTaskAppId(),
+            SystemUtils.IS_OS_WINDOWS ? "bat" : "sh");
+
+        Path path = new File(fileName).toPath();
+
+        if (Files.exists(path)) {
+            return fileName;
+        }
+
+        // chunjun command
+        StringBuilder sbr = new StringBuilder();

Review Comment:
   It's better to use Joiner or list to concat the parameters with " ".



##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/pom.xml:
##########
@@ -0,0 +1,57 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~     http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <parent>
+        <artifactId>dolphinscheduler-task-plugin</artifactId>
+        <groupId>org.apache.dolphinscheduler</groupId>
+        <version>dev-SNAPSHOT</version>
+    </parent>
+    <modelVersion>4.0.0</modelVersion>
+
+    <artifactId>dolphinscheduler-task-chunjun</artifactId>
+    <packaging>jar</packaging>
+
+    <dependencies>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-datasource-all</artifactId>
+        </dependency>
+
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-spi</artifactId>
+            <scope>provided</scope>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-task-api</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.dolphinscheduler</groupId>
+            <artifactId>dolphinscheduler-datasource-api</artifactId>
+            <version>${project.version}</version>
+        </dependency>
+
+        <dependency>
+            <groupId>com.alibaba</groupId>
+            <artifactId>druid</artifactId>
+        </dependency>

Review Comment:
   It seems we don't need to add this?
   ```suggestion
           
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] zhongjiajie merged pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

zhongjiajie merged PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

SbloodyS commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r922621030


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/test/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunConstantsTest.java:
##########
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+public class ChunJunConstantsTest {
+
+    private String flinkConfDir;
+
+    private String flinkLibDir;
+
+    private String hadoopConfDir;
+
+    @Before
+    public void setUp() {
+        flinkConfDir = "${FLINK_HOME}/conf";
+        flinkLibDir = "${FLINK_HOME}/lib";
+        hadoopConfDir = "${HADOOP_HOME}/etc/hadoop";
+    }
+
+    @Test
+    public void testEqualsString() {
+        Assert.assertEquals(flinkConfDir, ChunJunConstants.FLINK_CONF_DIR);
+        Assert.assertEquals(flinkLibDir, ChunJunConstants.FLINK_LIB_DIR);
+        Assert.assertEquals(hadoopConfDir, ChunJunConstants.HADOOP_CONF_DIR);

Review Comment:
   ```suggestion
           Assert.assertEquals(ChunJunConstants.FLINK_CONF_DIR, flinkConfDir);
           Assert.assertEquals(ChunJunConstants.FLINK_LIB_DIR, flinkLibDir);
           Assert.assertEquals(ChunJunConstants.HADOOP_CONF_DIR, hadoopConfDir);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] kezhenxu94 commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

kezhenxu94 commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r922759927


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunTask.java:
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_FAILURE;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.SystemUtils;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * chunjun task
+ */
+public class ChunJunTask extends AbstractTaskExecutor {
+    /**
+     * chunjun path
+     */
+    private static final String CHUNJUN_PATH = "${CHUNJUN_HOME}/bin/start-chunjun";
+
+    /**
+     * chunjun dist
+     */
+    private static final String CHUNJUN_DIST_DIR = "${CHUNJUN_HOME}/chunjun-dist";
+
+    /**
+     * chunJun parameters
+     */
+    private ChunJunParameters chunJunParameters;
+
+    /**
+     * shell command executor
+     */
+    private ShellCommandExecutor shellCommandExecutor;
+
+    /**
+     * taskExecutionContext
+     */
+    private TaskExecutionContext taskExecutionContext;
+
+    public ChunJunTask(TaskExecutionContext taskExecutionContext) {
+        super(taskExecutionContext);
+        this.taskExecutionContext = taskExecutionContext;
+
+        this.shellCommandExecutor = new ShellCommandExecutor(this::logHandle,
+            taskExecutionContext, logger);
+    }
+
+    /**
+     * init chunjun config
+     */
+    @Override
+    public void init() {
+        logger.info("chunjun task params {}", taskExecutionContext.getTaskParams());
+        chunJunParameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), ChunJunParameters.class);
+
+        if (!chunJunParameters.checkParameters()) {
+            throw new RuntimeException("chunjun task params is not valid");
+        }
+    }
+
+    /**
+     * run chunjun process
+     *
+     * @throws Exception exception
+     */
+    @Override
+    public void handle() throws Exception {
+        try {
+            Map<String, Property> paramsMap = taskExecutionContext.getPrepareParamsMap();
+
+            String jsonFilePath = buildChunJunJsonFile(paramsMap);
+            String shellCommandFilePath = buildShellCommandFile(jsonFilePath, paramsMap);
+            TaskResponse commandExecuteResult = shellCommandExecutor.run(shellCommandFilePath);
+
+            setExitStatusCode(commandExecuteResult.getExitStatusCode());
+            setAppIds(commandExecuteResult.getAppIds());
+            setProcessId(commandExecuteResult.getProcessId());
+        } catch (Exception e) {
+            logger.error("chunjun task failed.", e);
+            setExitStatusCode(EXIT_CODE_FAILURE);
+            throw e;
+        }
+    }
+
+    /**
+     * build chunjun json file
+     *
+     * @param paramsMap
+     * @return
+     * @throws Exception
+     */
+    private String buildChunJunJsonFile(Map<String, Property> paramsMap)
+        throws Exception {
+        // generate json
+        String fileName = String.format("%s/%s_job.json",
+            taskExecutionContext.getExecutePath(),
+            taskExecutionContext.getTaskAppId());
+
+        String json = null;
+
+        Path path = new File(fileName).toPath();
+        if (Files.exists(path)) {
+            return fileName;
+        }
+
+        if (chunJunParameters.getCustomConfig() == Flag.YES.ordinal()) {
+            json = chunJunParameters.getJson().replaceAll("\\r\\n", "\n");
+        }
+
+        // replace placeholder
+        json = ParameterUtils.convertParameterPlaceholders(json, ParamUtils.convert(paramsMap));
+
+        logger.debug("chunjun job json : {}", json);
+
+        // create chunjun json file
+        FileUtils.writeStringToFile(new File(fileName), json, StandardCharsets.UTF_8);
+        return fileName;
+    }
+
+
+    /**
+     * create command
+     *
+     * @return shell command file name
+     * @throws Exception if error throws Exception
+     */
+    private String buildShellCommandFile(String jobConfigFilePath, Map<String, Property> paramsMap)
+        throws Exception {
+        // generate scripts
+        String fileName = String.format("%s/%s_node.%s",
+            taskExecutionContext.getExecutePath(),
+            taskExecutionContext.getTaskAppId(),
+            SystemUtils.IS_OS_WINDOWS ? "bat" : "sh");
+
+        Path path = new File(fileName).toPath();
+
+        if (Files.exists(path)) {
+            return fileName;
+        }
+
+        // chunjun command
+        StringBuilder sbr = new StringBuilder();

Review Comment:
   I don't see you updated this, please don't mark as resolved if it's not resolved actually



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] tracehh commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

tracehh commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r923025753


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunTask.java:
##########
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.EXIT_CODE_FAILURE;
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
+
+import org.apache.dolphinscheduler.plugin.task.api.AbstractTaskExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.ShellCommandExecutor;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
+import org.apache.dolphinscheduler.plugin.task.api.model.Property;
+import org.apache.dolphinscheduler.plugin.task.api.model.TaskResponse;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParamUtils;
+import org.apache.dolphinscheduler.plugin.task.api.parser.ParameterUtils;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.JSONUtils;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang.SystemUtils;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
+import java.util.Map;
+import java.util.Set;
+
+/**
+ * chunjun task
+ */
+public class ChunJunTask extends AbstractTaskExecutor {
+    /**
+     * chunjun path
+     */
+    private static final String CHUNJUN_PATH = "${CHUNJUN_HOME}/bin/start-chunjun";
+
+    /**
+     * chunjun dist
+     */
+    private static final String CHUNJUN_DIST_DIR = "${CHUNJUN_HOME}/chunjun-dist";
+
+    /**
+     * chunJun parameters
+     */
+    private ChunJunParameters chunJunParameters;
+
+    /**
+     * shell command executor
+     */
+    private ShellCommandExecutor shellCommandExecutor;
+
+    /**
+     * taskExecutionContext
+     */
+    private TaskExecutionContext taskExecutionContext;
+
+    public ChunJunTask(TaskExecutionContext taskExecutionContext) {
+        super(taskExecutionContext);
+        this.taskExecutionContext = taskExecutionContext;
+
+        this.shellCommandExecutor = new ShellCommandExecutor(this::logHandle,
+            taskExecutionContext, logger);
+    }
+
+    /**
+     * init chunjun config
+     */
+    @Override
+    public void init() {
+        logger.info("chunjun task params {}", taskExecutionContext.getTaskParams());
+        chunJunParameters = JSONUtils.parseObject(taskExecutionContext.getTaskParams(), ChunJunParameters.class);
+
+        if (!chunJunParameters.checkParameters()) {
+            throw new RuntimeException("chunjun task params is not valid");
+        }
+    }
+
+    /**
+     * run chunjun process
+     *
+     * @throws Exception exception
+     */
+    @Override
+    public void handle() throws Exception {
+        try {
+            Map<String, Property> paramsMap = taskExecutionContext.getPrepareParamsMap();
+
+            String jsonFilePath = buildChunJunJsonFile(paramsMap);
+            String shellCommandFilePath = buildShellCommandFile(jsonFilePath, paramsMap);
+            TaskResponse commandExecuteResult = shellCommandExecutor.run(shellCommandFilePath);
+
+            setExitStatusCode(commandExecuteResult.getExitStatusCode());
+            setAppIds(commandExecuteResult.getAppIds());
+            setProcessId(commandExecuteResult.getProcessId());
+        } catch (Exception e) {
+            logger.error("chunjun task failed.", e);
+            setExitStatusCode(EXIT_CODE_FAILURE);
+            throw e;
+        }
+    }
+
+    /**
+     * build chunjun json file
+     *
+     * @param paramsMap
+     * @return
+     * @throws Exception
+     */
+    private String buildChunJunJsonFile(Map<String, Property> paramsMap)
+        throws Exception {
+        // generate json
+        String fileName = String.format("%s/%s_job.json",
+            taskExecutionContext.getExecutePath(),
+            taskExecutionContext.getTaskAppId());
+
+        String json = null;
+
+        Path path = new File(fileName).toPath();
+        if (Files.exists(path)) {
+            return fileName;
+        }
+
+        if (chunJunParameters.getCustomConfig() == Flag.YES.ordinal()) {
+            json = chunJunParameters.getJson().replaceAll("\\r\\n", "\n");
+        }
+
+        // replace placeholder
+        json = ParameterUtils.convertParameterPlaceholders(json, ParamUtils.convert(paramsMap));
+
+        logger.debug("chunjun job json : {}", json);
+
+        // create chunjun json file
+        FileUtils.writeStringToFile(new File(fileName), json, StandardCharsets.UTF_8);
+        return fileName;
+    }
+
+
+    /**
+     * create command
+     *
+     * @return shell command file name
+     * @throws Exception if error throws Exception
+     */
+    private String buildShellCommandFile(String jobConfigFilePath, Map<String, Property> paramsMap)
+        throws Exception {
+        // generate scripts
+        String fileName = String.format("%s/%s_node.%s",
+            taskExecutionContext.getExecutePath(),
+            taskExecutionContext.getTaskAppId(),
+            SystemUtils.IS_OS_WINDOWS ? "bat" : "sh");
+
+        Path path = new File(fileName).toPath();
+
+        if (Files.exists(path)) {
+            return fileName;
+        }
+
+        // chunjun command
+        StringBuilder sbr = new StringBuilder();

Review Comment:
   I understant incorrectly before. update it now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

sonarcloud[bot] commented on PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#issuecomment-1183929008

   SonarCloud Quality Gate failed.&nbsp; &nbsp; [![Quality Gate failed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/failed-16px.png 'Quality Gate failed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10937)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10937&resolved=false&types=CODE_SMELL)
   
   [![10.3%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/0-16px.png '10.3%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list) [10.3% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_coverage&view=list)  
   [![14.2%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/20-16px.png '14.2%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list) [14.2% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10937&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] EricGao888 commented on a diff in pull request #10937: [Feature][Task] Support chunjun (FlinkX) task

Posted by GitBox <gi...@apache.org>.

EricGao888 commented on code in PR #10937:
URL: https://github.com/apache/dolphinscheduler/pull/10937#discussion_r925576950


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-chunjun/src/main/java/org/apache/dolphinscheduler/plugin/task/chunjun/ChunJunParameters.java:
##########
@@ -0,0 +1,264 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.dolphinscheduler.plugin.task.chunjun;
+
+import org.apache.dolphinscheduler.plugin.task.api.enums.ResourceType;
+import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.AbstractParameters;
+import org.apache.dolphinscheduler.plugin.task.api.parameters.resource.ResourceParametersHelper;
+import org.apache.dolphinscheduler.spi.enums.Flag;
+import org.apache.dolphinscheduler.spi.utils.StringUtils;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * chunjun parameters
+ */
+public class ChunJunParameters extends AbstractParameters {

Review Comment:
   Hi~ I recommend replacing `@Data` with `@Getter`, `@Setter` and `@ToString` because `@Data` will generate some codes we may not need and cause the test coverage to decrease. Thx : )



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org