You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/07/07 02:31:15 UTC

[GitHub] [dolphinscheduler] huangchenguang123 opened a new pull request, #10821: [fix][plugin]: fix spark sql use custom parma.

huangchenguang123 opened a new pull request, #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821

   fix #10820 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#discussion_r918478114


##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java:
##########
@@ -1300,6 +1301,10 @@ public void getPreVarPool(TaskInstance taskInstance, Set<String> preTask) {
             }
             if (allProperty.size() > 0) {
                 taskInstance.setVarPool(JSONUtils.toJsonString(allProperty.values()));
+                Map<String, Object> taskParams = JSONUtils.parseObject(taskInstance.getTaskParams(), new TypeReference<Map<String, Object>>() {
+                });
+                taskParams.put("varPool", JSONUtils.toJsonString(allProperty.values()));

Review Comment:
   TaskParams and VarPool is different fields, so you can't add varpool into taskParams.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#discussion_r918521722


##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java:
##########
@@ -1300,6 +1301,10 @@ public void getPreVarPool(TaskInstance taskInstance, Set<String> preTask) {
             }
             if (allProperty.size() > 0) {
                 taskInstance.setVarPool(JSONUtils.toJsonString(allProperty.values()));
+                Map<String, Object> taskParams = JSONUtils.parseObject(taskInstance.getTaskParams(), new TypeReference<Map<String, Object>>() {
+                });
+                taskParams.put("varPool", JSONUtils.toJsonString(allProperty.values()));

Review Comment:
   I think this is a unified process for all task, so after you fix the spark ui, the custome params can pass to worker.
   BTW, the custome params for the current task is localParams.
   This is a shell task example, hope it can help you. 
   ![image](https://user-images.githubusercontent.com/11962619/178404246-ad166c76-1ebf-44e5-9b69-ac5c3be41a1c.png)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] huangchenguang123 commented on a diff in pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
huangchenguang123 commented on code in PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#discussion_r918523562


##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java:
##########
@@ -1300,6 +1301,10 @@ public void getPreVarPool(TaskInstance taskInstance, Set<String> preTask) {
             }
             if (allProperty.size() > 0) {
                 taskInstance.setVarPool(JSONUtils.toJsonString(allProperty.values()));
+                Map<String, Object> taskParams = JSONUtils.parseObject(taskInstance.getTaskParams(), new TypeReference<Map<String, Object>>() {
+                });
+                taskParams.put("varPool", JSONUtils.toJsonString(allProperty.values()));

Review Comment:
   localParams is currently useful, but this is limited to [in] parameters, but for upstream [out] parameters, spark sql currently doesn't work, which is why I use varPool.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] huangchenguang123 commented on pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
huangchenguang123 commented on PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#issuecomment-1203822662

   > this pr do not cherry pick, but we already add other PR to fix it
   
   Thanks, I'm looking forward to seeing it in release.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#discussion_r918521995


##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java:
##########
@@ -1300,6 +1301,10 @@ public void getPreVarPool(TaskInstance taskInstance, Set<String> preTask) {
             }
             if (allProperty.size() > 0) {
                 taskInstance.setVarPool(JSONUtils.toJsonString(allProperty.values()));
+                Map<String, Object> taskParams = JSONUtils.parseObject(taskInstance.getTaskParams(), new TypeReference<Map<String, Object>>() {
+                });
+                taskParams.put("varPool", JSONUtils.toJsonString(allProperty.values()));

Review Comment:
   BTW, I found the flink task also has the same issue in ui.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#issuecomment-1181678642

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10821)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=CODE_SMELL)
   
   [![62.5%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/60-16px.png '62.5%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10821&metric=new_coverage&view=list) [62.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10821&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10821&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10821&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] huangchenguang123 commented on a diff in pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
huangchenguang123 commented on code in PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#discussion_r918571185


##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java:
##########
@@ -1300,6 +1301,10 @@ public void getPreVarPool(TaskInstance taskInstance, Set<String> preTask) {
             }
             if (allProperty.size() > 0) {
                 taskInstance.setVarPool(JSONUtils.toJsonString(allProperty.values()));
+                Map<String, Object> taskParams = JSONUtils.parseObject(taskInstance.getTaskParams(), new TypeReference<Map<String, Object>>() {
+                });
+                taskParams.put("varPool", JSONUtils.toJsonString(allProperty.values()));

Review Comment:
   I will create a feature, but do I need to resubmit a mr about parameter transfer?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] caishunfeng merged pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
caishunfeng merged PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#discussion_r918892700


##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java:
##########
@@ -1300,6 +1301,10 @@ public void getPreVarPool(TaskInstance taskInstance, Set<String> preTask) {
             }
             if (allProperty.size() > 0) {
                 taskInstance.setVarPool(JSONUtils.toJsonString(allProperty.values()));
+                Map<String, Object> taskParams = JSONUtils.parseObject(taskInstance.getTaskParams(), new TypeReference<Map<String, Object>>() {
+                });
+                taskParams.put("varPool", JSONUtils.toJsonString(allProperty.values()));

Review Comment:
   > I will create a feature, but do I need to resubmit a mr about parameter transfer?
   
   You can submit other pr to the feature issue if you are interested in.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] zhongjiajie commented on pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
zhongjiajie commented on PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#issuecomment-1203790187

   this pr do not cherry pick, but we already add other PR to fix it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] huangchenguang123 commented on pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
huangchenguang123 commented on PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#issuecomment-1181642737

   PTAL @caishunfeng 
   I had roll back some code


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] sonarcloud[bot] commented on pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#issuecomment-1181679774

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache-dolphinscheduler&pullRequest=10821)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache-dolphinscheduler&pullRequest=10821&resolved=false&types=CODE_SMELL)
   
   [![62.5%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/60-16px.png '62.5%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10821&metric=new_coverage&view=list) [62.5% Coverage](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10821&metric=new_coverage&view=list)  
   [![0.0%](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/3-16px.png '0.0%')](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10821&metric=new_duplicated_lines_density&view=list) [0.0% Duplication](https://sonarcloud.io/component_measures?id=apache-dolphinscheduler&pullRequest=10821&metric=new_duplicated_lines_density&view=list)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] huangchenguang123 commented on a diff in pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
huangchenguang123 commented on code in PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#discussion_r918492584


##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java:
##########
@@ -1300,6 +1301,10 @@ public void getPreVarPool(TaskInstance taskInstance, Set<String> preTask) {
             }
             if (allProperty.size() > 0) {
                 taskInstance.setVarPool(JSONUtils.toJsonString(allProperty.values()));
+                Map<String, Object> taskParams = JSONUtils.parseObject(taskInstance.getTaskParams(), new TypeReference<Map<String, Object>>() {
+                });
+                taskParams.put("varPool", JSONUtils.toJsonString(allProperty.values()));

Review Comment:
   org.apache.dolphinscheduler.service.expand.CuringGlobalParams#paramParsingPreparation use taskInstance.varParams to build prepareParamsMap, and prepareParamsMap  was used to replace sql param.But taskInstance.varParams is always null, so I put it.This is why I did it, maybe you have a better idea？



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] codecov-commenter commented on pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#issuecomment-1181671163

   # [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10821?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#10821](https://codecov.io/gh/apache/dolphinscheduler/pull/10821?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (d5cf976) into [dev](https://codecov.io/gh/apache/dolphinscheduler/commit/2d8b4932b30861076d636fdd35510917e4c157d7?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2d8b493) will **decrease** coverage by `0.43%`.
   > The diff coverage is `83.33%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##                dev   #10821      +/-   ##
   ============================================
   - Coverage     40.61%   40.18%   -0.44%     
   - Complexity     4817     4837      +20     
   ============================================
     Files           902      944      +42     
     Lines         36221    36967     +746     
     Branches       3999     4041      +42     
   ============================================
   + Hits          14711    14854     +143     
   - Misses        20045    20613     +568     
   - Partials       1465     1500      +35     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/dolphinscheduler/pull/10821?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...plugin/task/api/parameters/AbstractParameters.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10821/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stYXBpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2FwaS9wYXJhbWV0ZXJzL0Fic3RyYWN0UGFyYW1ldGVycy5qYXZh) | `23.28% <0.00%> (ø)` | |
   | [.../dolphinscheduler/plugin/task/spark/SparkTask.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10821/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stc3Bhcmsvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcGx1Z2luL3Rhc2svc3BhcmsvU3BhcmtUYXNrLmphdmE=) | `59.82% <100.00%> (+1.48%)` | :arrow_up: |
   | [...phinscheduler/api/controller/TenantController.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10821/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL2NvbnRyb2xsZXIvVGVuYW50Q29udHJvbGxlci5qYXZh) | `62.50% <0.00%> (-25.00%)` | :arrow_down: |
   | [...org/apache/dolphinscheduler/dao/entity/Tenant.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10821/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYW8vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvZGFvL2VudGl0eS9UZW5hbnQuamF2YQ==) | `49.01% <0.00%> (-23.71%)` | :arrow_down: |
   | [...olphinscheduler/plugin/task/emr/EmrParameters.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10821/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci10YXNrLXBsdWdpbi9kb2xwaGluc2NoZWR1bGVyLXRhc2stZW1yL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9kb2xwaGluc2NoZWR1bGVyL3BsdWdpbi90YXNrL2Vtci9FbXJQYXJhbWV0ZXJzLmphdmE=) | `50.00% <0.00%> (-21.43%)` | :arrow_down: |
   | [...dolphinscheduler/remote/command/CommandHeader.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10821/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1yZW1vdGUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvcmVtb3RlL2NvbW1hbmQvQ29tbWFuZEhlYWRlci5qYXZh) | `83.33% <0.00%> (-16.67%)` | :arrow_down: |
   | [.../org/apache/dolphinscheduler/dao/entity/Queue.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10821/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1kYW8vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvZGFvL2VudGl0eS9RdWV1ZS5qYXZh) | `36.17% <0.00%> (-15.45%)` | :arrow_down: |
   | [...uler/api/service/impl/K8SNamespaceServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10821/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9LOFNOYW1lc3BhY2VTZXJ2aWNlSW1wbC5qYXZh) | `68.71% <0.00%> (-5.68%)` | :arrow_down: |
   | [...er/master/dispatch/host/assign/RandomSelector.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10821/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1tYXN0ZXIvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvc2VydmVyL21hc3Rlci9kaXNwYXRjaC9ob3N0L2Fzc2lnbi9SYW5kb21TZWxlY3Rvci5qYXZh) | `77.77% <0.00%> (-5.56%)` | :arrow_down: |
   | [...nscheduler/api/service/impl/TenantServiceImpl.java](https://codecov.io/gh/apache/dolphinscheduler/pull/10821/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZG9scGhpbnNjaGVkdWxlci1hcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2RvbHBoaW5zY2hlZHVsZXIvYXBpL3NlcnZpY2UvaW1wbC9UZW5hbnRTZXJ2aWNlSW1wbC5qYXZh) | `63.71% <0.00%> (-4.88%)` | :arrow_down: |
   | ... and [147 more](https://codecov.io/gh/apache/dolphinscheduler/pull/10821/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10821?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/dolphinscheduler/pull/10821?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [2d8b493...d5cf976](https://codecov.io/gh/apache/dolphinscheduler/pull/10821?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] caishunfeng commented on a diff in pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
caishunfeng commented on code in PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#discussion_r918536105


##########
dolphinscheduler-master/src/main/java/org/apache/dolphinscheduler/server/master/runner/WorkflowExecuteRunnable.java:
##########
@@ -1300,6 +1301,10 @@ public void getPreVarPool(TaskInstance taskInstance, Set<String> preTask) {
             }
             if (allProperty.size() > 0) {
                 taskInstance.setVarPool(JSONUtils.toJsonString(allProperty.values()));
+                Map<String, Object> taskParams = JSONUtils.parseObject(taskInstance.getTaskParams(), new TypeReference<Map<String, Object>>() {
+                });
+                taskParams.put("varPool", JSONUtils.toJsonString(allProperty.values()));

Review Comment:
   Now ds just support three task types for parameter transfer, see https://dolphinscheduler.apache.org/en-us/docs/dev/user_doc/guide/parameter/context.html
   So it's a feature, not a bug, you would better to create another feature issue to do it. WDYT?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [dolphinscheduler] huangchenguang123 commented on pull request #10821: [fix][plugin]: fix spark sql use custom parma.

Posted by GitBox <gi...@apache.org>.
huangchenguang123 commented on PR #10821:
URL: https://github.com/apache/dolphinscheduler/pull/10821#issuecomment-1180144954

   PTAL


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org