You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/04/26 10:26:07 UTC

[GitHub] [dolphinscheduler] sq-q opened a new pull request, #9790: [Feature-9772][plugin/ui] support SparkSQL Task

sq-q opened a new pull request, #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790

     <!--Thanks very much for contributing to Apache DolphinScheduler. Please review https://dolphinscheduler.apache.org/en-us/community/development/pull-request.html before opening a pull request.-->
   
   
   ## Purpose of the pull request
   
   According to user requirements, we hope to support spark sql for SparkTask.
   
   Users want to have an editor to fill in the sql script, which is convenient for ETL.
   
   The renderings we need are as follows:
   ![lQLPDhtd5ypCZK3NA5bNBCywAi-2JJ5_quwCaoYbSoA-AA_1068_918](https://user-images.githubusercontent.com/83269577/165266197-c89ae2a2-ca62-461c-8591-7f8001ba91b4.png)
   
   ## Brief change log
   
   - ProgramType.class add sql type
   - SparkTask.class and SparkArgsUtils.class add main logical for support Spark sql command
   - use-spark.ts adds some logic to support front-end pages supporting spark sql editing SQL
   
   ## Verify this pull request
   
   This change added tests and can be verified as follows:
   
   org.apache.dolphinscheduler.plugin.task.spark.SparkTaskTest


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS merged pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
SbloodyS merged PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#discussion_r859315280


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkArgsUtils.java:
##########
@@ -17,18 +17,37 @@
 
 package org.apache.dolphinscheduler.plugin.task.spark;
 
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
 import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.utils.ArgsUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.OSUtils;
 import org.apache.dolphinscheduler.spi.utils.StringUtils;
 
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Set;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
 
 /**
  * spark args utils
+ * add main logical for support Spark command
  */
 public class SparkArgsUtils {
 
+    private final static Logger logger = LoggerFactory.getLogger(SparkArgsUtils.class);

Review Comment:
   I think we should print the info log in the task log instead of the worker log to facilitate user troubleshooting.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#issuecomment-1110685680

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9790)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=CODE_SMELL)
   
   [![63.4%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/60-16px.png '63.4%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9790&metric=new_coverage&view=list) [63.4% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9790&metric=new_coverage&view=list)  
   [![3.5%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/5-16px.png '3.5%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9790&metric=new_duplicated_lines_density&view=list) [3.5% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9790&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#issuecomment-1110794893

   Thanks for your contribution. This is a good beginning. Considering that it's your first contribution, I think we can get deep communication, you can contact me through mail or add WeChat(SbloodyS), when mail or added, please tell me who you are. I think I can help to familiar with the Dolphinscheduler if you meet with problems. @sq-q 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sq-q commented on a diff in pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
sq-q commented on code in PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#discussion_r859418326


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkArgsUtils.java:
##########
@@ -17,19 +17,37 @@
 
 package org.apache.dolphinscheduler.plugin.task.spark;
 
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
 import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.utils.ArgsUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.OSUtils;
 import org.apache.dolphinscheduler.spi.utils.StringUtils;
 
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Set;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * spark args utils
+ * add main logical for support Spark command
  */
 public class SparkArgsUtils {
 
-    private static final String SPARK_CLUSTER = "cluster";
+    private static final Logger logger = LoggerFactory.getLogger(String.format(TaskConstants.TASK_LOG_LOGGER_NAME_FORMAT, SparkArgsUtils.class));

Review Comment:
   SparkArgsUtils and AbstractTaskExecutor have no inheritance relationship. If you use the same logger, you need to move SparkArgsUtils to SparkTask, and you can define only one logger. Do you think this method is OK?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#discussion_r859425095


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkArgsUtils.java:
##########
@@ -17,19 +17,37 @@
 
 package org.apache.dolphinscheduler.plugin.task.spark;
 
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
 import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.utils.ArgsUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.OSUtils;
 import org.apache.dolphinscheduler.spi.utils.StringUtils;
 
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Set;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * spark args utils
+ * add main logical for support Spark command
  */
 public class SparkArgsUtils {
 
-    private static final String SPARK_CLUSTER = "cluster";
+    private static final Logger logger = LoggerFactory.getLogger(String.format(TaskConstants.TASK_LOG_LOGGER_NAME_FORMAT, SparkArgsUtils.class));

Review Comment:
   I'm not quite sure about that. @ruanwenjun @zhuangchong Do you have time to take a look at this PR?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] labbomb commented on pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
labbomb commented on PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#issuecomment-1110448950

   @Amy0104 @devosend Take the time to review the front-end code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] songjianet commented on pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
songjianet commented on PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#issuecomment-1110451123

   Does this PR have a corresponding issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sq-q commented on pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
sq-q commented on PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#issuecomment-1110455309

   > Does this PR have a corresponding issue? @sq-q
   
   This is my issue (https://github.com/apache/dolphinscheduler/issues/9772).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#issuecomment-1110628814

   # [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/9790?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#9790](https://codecov.io/gh/apache/dolphinscheduler/pull/9790?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a2e5971) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/637028735dbf8a78efa63cf8358d2bead7bb5c6e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6370287) will **increase** coverage by `0.31%`.
   > The diff coverage is `58.94%`.
   
   > :exclamation: Current head a2e5971 differs from pull request most recent head 02cccd7. Consider uploading reports for the commit 02cccd7 to get more accurate results
   
   ```diff
   @@             Coverage Diff              @@
   ##                dev    #9790      +/-   ##
   ============================================
   + Coverage     40.07%   40.39%   +0.31%     
   - Complexity     4484     4523      +39     
   ============================================
     Files           835      834       -1     
     Lines         33575    33622      +47     
     Branches       3712     3718       +6     
   ============================================
   + Hits          13456    13582     +126     
   + Misses        18865    18762     -103     
   - Partials       1254     1278      +24     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/9790?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...che/dolphinscheduler/common/enums/ProgramType.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9790/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL2VudW1zL1Byb2dyYW1UeXBlLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...he/dolphinscheduler/common/enums/SparkVersion.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9790/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvY29tbW9uL2VudW1zL1NwYXJrVmVyc2lvbi5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...hinscheduler/plugin/task/api/AbstractYarnTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9790/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS9BYnN0cmFjdFlhcm5UYXNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
   | [...hinscheduler/plugin/task/spark/SparkConstants.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9790/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stc3Bhcmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svc3BhcmsvU3BhcmtDb25zdGFudHMuamF2YQ==) | `0.00% <ø> (ø)` | |
   | [.../dolphinscheduler/plugin/task/spark/SparkTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9790/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stc3Bhcmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svc3BhcmsvU3BhcmtUYXNrLmphdmE=) | `58.03% <60.97%> (+58.03%)` | :arrow_up: |
   | [...inscheduler/plugin/task/spark/SparkParameters.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9790/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stc3Bhcmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svc3BhcmsvU3BhcmtQYXJhbWV0ZXJzLmphdmE=) | `98.14% <75.00%> (+80.50%)` | :arrow_up: |
   | [...olphinscheduler/plugin/task/spark/ProgramType.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9790/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stc3Bhcmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svc3BhcmsvUHJvZ3JhbVR5cGUuamF2YQ==) | `100.00% <100.00%> (+100.00%)` | :arrow_up: |
   | [...lphinscheduler/plugin/task/spark/SparkVersion.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9790/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stc3Bhcmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svc3BhcmsvU3BhcmtWZXJzaW9uLmphdmE=) | `83.33% <100.00%> (+83.33%)` | :arrow_up: |
   | [...er/master/dispatch/host/assign/RandomSelector.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9790/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9kaXNwYXRjaC9ob3N0L2Fzc2lnbi9SYW5kb21TZWxlY3Rvci5qYXZh) | `77.77% <0.00%> (-5.56%)` | :arrow_down: |
   | [...dolphinscheduler/plugin/alert/http/HttpSender.java](https://codecov.io/gh/apache/dolphinscheduler/pull/9790/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hbGVydC9kb2xwaGluc2NoZWR1bGVyLWFsZXJ0LXBsdWdpbnMvZG9scGhpbnNjaGVkdWxlci1hbGVydC1odHRwL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi9hbGVydC9odHRwL0h0dHBTZW5kZXIuamF2YQ==) | `53.96% <0.00%> (-4.97%)` | :arrow_down: |
   | ... and [10 more](https://codecov.io/gh/apache/dolphinscheduler/pull/9790/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/9790?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/9790?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [6370287...02cccd7](https://codecov.io/gh/apache/dolphinscheduler/pull/9790?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sq-q commented on pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
sq-q commented on PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#issuecomment-1110796520

   > 感谢您的贡献。这是一个好的开始。考虑到是你的第一次投稿，我想我们可以深入交流，你可以通过邮件联系我或者加微信（SbloodyS），邮件或者加的时候请告诉我你是谁。如果您遇到问题，我想我可以帮助您熟悉 Dolphinscheduler。@sq-q
   
   Grazie


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#discussion_r859408076


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkArgsUtils.java:
##########
@@ -17,19 +17,37 @@
 
 package org.apache.dolphinscheduler.plugin.task.spark;
 
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
 import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.utils.ArgsUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.OSUtils;
 import org.apache.dolphinscheduler.spi.utils.StringUtils;
 
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Set;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * spark args utils
+ * add main logical for support Spark command
  */
 public class SparkArgsUtils {
 
-    private static final String SPARK_CLUSTER = "cluster";
+    private static final Logger logger = LoggerFactory.getLogger(String.format(TaskConstants.TASK_LOG_LOGGER_NAME_FORMAT, SparkArgsUtils.class));

Review Comment:
   Maybe we should reuse SparkTask's logger instead of redefining one. So that we can unify the log output.
   
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sq-q commented on a diff in pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
sq-q commented on code in PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#discussion_r859550558


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkArgsUtils.java:
##########
@@ -17,19 +17,37 @@
 
 package org.apache.dolphinscheduler.plugin.task.spark;
 
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
 import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.utils.ArgsUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.OSUtils;
 import org.apache.dolphinscheduler.spi.utils.StringUtils;
 
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Set;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * spark args utils
+ * add main logical for support Spark command
  */
 public class SparkArgsUtils {
 
-    private static final String SPARK_CLUSTER = "cluster";
+    private static final Logger logger = LoggerFactory.getLogger(String.format(TaskConstants.TASK_LOG_LOGGER_NAME_FORMAT, SparkArgsUtils.class));

Review Comment:
   No, I think this method is also feasible, so I will change it first. If I find it inappropriate in the future, I will change it back.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#issuecomment-1109810373

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9790)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=CODE_SMELL) [8 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=CODE_SMELL)
   
   [![58.6%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/50-16px.png '58.6%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9790&metric=new_coverage&view=list) [58.6% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9790&metric=new_coverage&view=list)  
   [![4.4%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/5-16px.png '4.4%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9790&metric=new_duplicated_lines_density&view=list) [4.4% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9790&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#issuecomment-1110509423

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=9790)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=SECURITY_HOTSPOT) [![E](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/E-16px.png 'E')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=SECURITY_HOTSPOT) [1 Security Hotspot](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=CODE_SMELL) [2 Code Smells](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=9790&resolved=false&types=CODE_SMELL)
   
   [![59.2%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/50-16px.png '59.2%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9790&metric=new_coverage&view=list) [59.2% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9790&metric=new_coverage&view=list)  
   [![4.1%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/5-16px.png '4.1%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9790&metric=new_duplicated_lines_density&view=list) [4.1% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=9790&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sq-q commented on a diff in pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
sq-q commented on code in PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#discussion_r859316017


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkArgsUtils.java:
##########
@@ -17,18 +17,37 @@
 
 package org.apache.dolphinscheduler.plugin.task.spark;
 
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
 import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.utils.ArgsUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.OSUtils;
 import org.apache.dolphinscheduler.spi.utils.StringUtils;
 
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Set;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
 
 /**
  * spark args utils
+ * add main logical for support Spark command
  */
 public class SparkArgsUtils {
 
+    private final static Logger logger = LoggerFactory.getLogger(SparkArgsUtils.class);

Review Comment:
   Some code smells in sonarcloud are inappropriate, I am refactoring here, please wait a moment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sq-q commented on a diff in pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
sq-q commented on code in PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#discussion_r859538031


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkArgsUtils.java:
##########
@@ -17,19 +17,37 @@
 
 package org.apache.dolphinscheduler.plugin.task.spark;
 
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
 import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.utils.ArgsUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.OSUtils;
 import org.apache.dolphinscheduler.spi.utils.StringUtils;
 
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Set;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * spark args utils
+ * add main logical for support Spark command
  */
 public class SparkArgsUtils {
 
-    private static final String SPARK_CLUSTER = "cluster";
+    private static final Logger logger = LoggerFactory.getLogger(String.format(TaskConstants.TASK_LOG_LOGGER_NAME_FORMAT, SparkArgsUtils.class));

Review Comment:
   I have refactored.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on a diff in pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on code in PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#discussion_r859546923


##########
dolphinscheduler-task-plugin/dolphinscheduler-task-spark/src/main/java/org/apache/dolphinscheduler/plugin/task/spark/SparkArgsUtils.java:
##########
@@ -17,19 +17,37 @@
 
 package org.apache.dolphinscheduler.plugin.task.spark;
 
+import org.apache.dolphinscheduler.plugin.task.api.TaskConstants;
+import org.apache.dolphinscheduler.plugin.task.api.TaskExecutionContext;
 import org.apache.dolphinscheduler.plugin.task.api.model.ResourceInfo;
 import org.apache.dolphinscheduler.plugin.task.api.utils.ArgsUtils;
+import org.apache.dolphinscheduler.plugin.task.api.utils.OSUtils;
 import org.apache.dolphinscheduler.spi.utils.StringUtils;
 
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.nio.file.attribute.FileAttribute;
+import java.nio.file.attribute.PosixFilePermission;
+import java.nio.file.attribute.PosixFilePermissions;
 import java.util.ArrayList;
 import java.util.List;
+import java.util.Set;
+
+import static org.apache.dolphinscheduler.plugin.task.api.TaskConstants.RWXR_XR_X;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * spark args utils
+ * add main logical for support Spark command
  */
 public class SparkArgsUtils {
 
-    private static final String SPARK_CLUSTER = "cluster";
+    private static final Logger logger = LoggerFactory.getLogger(String.format(TaskConstants.TASK_LOG_LOGGER_NAME_FORMAT, SparkArgsUtils.class));

Review Comment:
   @sq-q Don't get me wrong. I'm just not familiar about this part. Please wait patiently for others to reply.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] SbloodyS commented on pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
SbloodyS commented on PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#issuecomment-1110810913

   > This kind of code that involves changes to the front and back ends should be noted that it needs to wait for the approval of the front end before it can be merged. @SbloodyS
   
   Sorry. I'll pay attention next time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] songjianet commented on pull request #9790: [Feature-9772][plugin/ui] support SparkSQL Task

Posted by GitBox <gi...@apache.org>.
songjianet commented on PR #9790:
URL: https://github.com/apache/dolphinscheduler/pull/9790#issuecomment-1110808432

   This kind of code that involves changes to the front and back ends should be noted that it needs to wait for the approval of the front end before it can be merged. @SbloodyS 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org