You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "alextinng (via GitHub)" <gi...@apache.org> on 2023/02/10 09:53:31 UTC
[GitHub] [dolphinscheduler] alextinng opened a new issue, #13543: [Bug] [plugin] shell task plugin replace time variable with a wrong time
alextinng opened a new issue, #13543:
URL: https://github.com/apache/dolphinscheduler/issues/13543
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
task params defined in shell script was replaced with a wrong time.
schedule_time is 2023-02-01 00:00:00, but plugin replace time variable dt_1(values is ) with 20230129.
### What you expected to happen
task params was replaced correctly
### How to reproduce
step 1. create a shell task like this:
task config:
global variables:
step 2: run task in complement mode
### Anything else
task log:
```
[INFO] 2023-02-10 09:17:30.897 +0000 - shell task params {"localParams":[{"prop":"dt_1","direct":"IN","type":"VARCHAR","value":"$[yyyyMMdd-2]"},{"prop":"start_date","direct":"IN","type":"VARCHAR","value":"$[yyMMdd000000-1]"}],"rawScript":"echo \"dt: ${dt}\"\r\necho \"dt_1: ${dt_1}\"\r\necho \"start_date: ${start_date}\"","resourceList":[]}
[INFO] 2023-02-10 09:17:30.901 +0000 - alextinng: {"taskInstanceId":2551,"taskName":"echo","firstSubmitTime":"2023-02-10 09:17:30","startTime":"2023-02-10 09:17:30","taskType":"SHELL","host":"10.0.1.182:1234","executePath":"/tmp/dolphinscheduler/exec/process/8166510133248/8188354628864_16/2103/2551","logPath":"/opt/dolphinscheduler/worker-server/logs/20230210/8188354628864_16-2103-2551.log","taskJson":null,"processId":0,"processDefineCode":8188354628864,"processDefineVersion":16,"appIds":null,"processInstanceId":2103,"scheduleTime":"2023-01-31 16:00:00","globalParams":"[{\"prop\":\"dt\",\"direct\":\"IN\",\"type\":\"VARCHAR\",\"value\":\"20230131\"},{\"prop\":\"dt_1\",\"direct\":\"IN\",\"type\":\"VARCHAR\",\"value\":\"20230130\"}]","executorId":3,"cmdTypeIfComplement":5,"tenantCode":"root","queue":"default","processDefineId":0,"projectId":0,"projectCode":8166510133248,"taskParams":"{\"localParams\":[{\"prop\":\"dt_1\",\"direct\":\"IN\",\"type\":\"VARCHAR\",\"value\":\"$[yyyyMMdd-2]
\"},{\"prop\":\"start_date\",\"direct\":\"IN\",\"type\":\"VARCHAR\",\"value\":\"$[yyMMdd000000-1]\"}],\"rawScript\":\"echo \\\"dt: ${dt}\\\"\\r\\necho \\\"dt_1: ${dt_1}\\\"\\r\\necho \\\"start_date: ${start_date}\\\"\",\"resourceList\":[]}","envFile":"/opt/dolphinscheduler/worker-server/conf/dolphinscheduler_env.sh","environmentConfig":"export JAVA_HOME=/usr/lib/java-1.8.0\nexport HADOOP_HOME=/opt/hadoop\nexport HIVE_HOME=/opt/hive\nexport DATAX_HOME=/opt/datax\nexport SPARK_HOME=/opt/spark2\nexport YARN_CONF_DIR=/etc/hadoop/conf\nexport PATH=$PATH:$HADOOP_HOME/bin:$SPARK_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$DATAX_HOME/bin","definedParams":{"dt":"20230131","dt_1":"20230130"},"taskAppId":"2103_2551","taskTimeoutStrategy":null,"taskTimeout":2147483647,"workerGroup":"default","delayTime":0,"currentExecutionStatus":"RUNNING_EXECUTION","taskLogName":"taskAppId=TASK-20230210-8188354628864_16-2103-2551","resourceParametersHelper":null,"endTime":null,"sqlTaskExecutionContext":null,"resou
rces":{},"varPool":null,"dryRun":0,"paramsMap":{"system.datetime":{"prop":"system.datetime","direct":null,"type":null,"value":"20230131160000"}},"dataQualityTaskExecutionContext":{"ruleId":0,"ruleName":null,"ruleType":0,"ruleInputEntryList":null,"executeSqlList":null,"comparisonNeedStatisticsValueTable":false,"compareWithFixedValue":false,"hdfsPath":null,"sourceConnectorType":null,"sourceType":0,"sourceConnectionParams":null,"targetConnectorType":null,"targetType":0,"targetConnectionParams":null,"writerConnectorType":null,"writerType":0,"writerTable":null,"writerConnectionParams":null,"statisticsValueConnectorType":null,"statisticsValueType":0,"statisticsValueTable":null,"statisticsValueWriterConnectionParams":null}}
[INFO] 2023-02-10 09:17:30.901 +0000 - alextinng: {"taskInstanceId":2551,"taskName":"echo","firstSubmitTime":"2023-02-10 09:17:30","startTime":"2023-02-10 09:17:30","taskType":"SHELL","host":"10.0.1.182:1234","executePath":"/tmp/dolphinscheduler/exec/process/8166510133248/8188354628864_16/2103/2551","logPath":"/opt/dolphinscheduler/worker-server/logs/20230210/8188354628864_16-2103-2551.log","taskJson":null,"processId":0,"processDefineCode":8188354628864,"processDefineVersion":16,"appIds":null,"processInstanceId":2103,"scheduleTime":"2023-01-31 16:00:00","globalParams":"[{\"prop\":\"dt\",\"direct\":\"IN\",\"type\":\"VARCHAR\",\"value\":\"20230131\"},{\"prop\":\"dt_1\",\"direct\":\"IN\",\"type\":\"VARCHAR\",\"value\":\"20230130\"}]","executorId":3,"cmdTypeIfComplement":5,"tenantCode":"root","queue":"default","processDefineId":0,"projectId":0,"projectCode":8166510133248,"taskParams":"{\"localParams\":[{\"prop\":\"dt_1\",\"direct\":\"IN\",\"type\":\"VARCHAR\",\"value\":\"$[yyyyMMdd-2]
\"},{\"prop\":\"start_date\",\"direct\":\"IN\",\"type\":\"VARCHAR\",\"value\":\"$[yyMMdd000000-1]\"}],\"rawScript\":\"echo \\\"dt: ${dt}\\\"\\r\\necho \\\"dt_1: ${dt_1}\\\"\\r\\necho \\\"start_date: ${start_date}\\\"\",\"resourceList\":[]}","envFile":"/opt/dolphinscheduler/worker-server/conf/dolphinscheduler_env.sh","environmentConfig":"export JAVA_HOME=/usr/lib/java-1.8.0\nexport HADOOP_HOME=/opt/hadoop\nexport HIVE_HOME=/opt/hive\nexport DATAX_HOME=/opt/datax\nexport SPARK_HOME=/opt/spark2\nexport YARN_CONF_DIR=/etc/hadoop/conf\nexport PATH=$PATH:$HADOOP_HOME/bin:$SPARK_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$DATAX_HOME/bin","definedParams":{"dt":"20230131","dt_1":"20230130"},"taskAppId":"2103_2551","taskTimeoutStrategy":null,"taskTimeout":2147483647,"workerGroup":"default","delayTime":0,"currentExecutionStatus":"RUNNING_EXECUTION","taskLogName":"taskAppId=TASK-20230210-8188354628864_16-2103-2551","resourceParametersHelper":null,"endTime":null,"sqlTaskExecutionContext":null,"resou
rces":{},"varPool":null,"dryRun":0,"paramsMap":{"system.datetime":{"prop":"system.datetime","direct":null,"type":null,"value":"20230131160000"}},"dataQualityTaskExecutionContext":{"ruleId":0,"ruleName":null,"ruleType":0,"ruleInputEntryList":null,"executeSqlList":null,"comparisonNeedStatisticsValueTable":false,"compareWithFixedValue":false,"hdfsPath":null,"sourceConnectorType":null,"sourceType":0,"sourceConnectionParams":null,"targetConnectorType":null,"targetType":0,"targetConnectionParams":null,"writerConnectorType":null,"writerType":0,"writerTable":null,"writerConnectionParams":null,"statisticsValueConnectorType":null,"statisticsValueType":0,"statisticsValueTable":null,"statisticsValueWriterConnectionParams":null}}
[INFO] 2023-02-10 09:17:30.903 +0000 - raw script : echo "dt: 20230131"
echo "dt_1: 20230129"
echo "start_date: 230130000000"
[INFO] 2023-02-10 09:17:30.903 +0000 - task execute path : /tmp/dolphinscheduler/exec/process/8166510133248/8188354628864_16/2103/2551
[INFO] 2023-02-10 09:17:30.903 +0000 - tenantCode user:root, task dir:2103_2551
[INFO] 2023-02-10 09:17:30.903 +0000 - create command file:/tmp/dolphinscheduler/exec/process/8166510133248/8188354628864_16/2103/2551/2103_2551.command
[INFO] 2023-02-10 09:17:30.903 +0000 - command : #!/bin/bash
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
export JAVA_HOME=/usr/lib/java-1.8.0
export HADOOP_HOME=/opt/hadoop
export HIVE_HOME=/opt/hive
export DATAX_HOME=/opt/datax
export SPARK_HOME=/opt/spark2
export YARN_CONF_DIR=/etc/hadoop/conf
export PATH=$PATH:$HADOOP_HOME/bin:$SPARK_HOME/bin:$JAVA_HOME/bin:$HIVE_HOME/bin:$DATAX_HOME/bin
/tmp/dolphinscheduler/exec/process/8166510133248/8188354628864_16/2103/2551/2103_2551_node.sh
[INFO] 2023-02-10 09:17:30.905 +0000 - task run command: sudo -u root sh /tmp/dolphinscheduler/exec/process/8166510133248/8188354628864_16/2103/2551/2103_2551.command
[INFO] 2023-02-10 09:17:30.906 +0000 - process start, process id is: 18032
[INFO] 2023-02-10 09:17:30.919 +0000 - process has exited, execute path:/tmp/dolphinscheduler/exec/process/8166510133248/8188354628864_16/2103/2551, processId:18032 ,exitStatusCode:0 ,processWaitForStatus:true ,processExitValue:0
[INFO] 2023-02-10 09:17:31.907 +0000 - -> dt: 20230131
dt_1: 20230129
start_date: 230130000000
[INFO] 2023-02-10 09:17:31.908 +0000 - FINALIZE_SESSION
```
### Version
3.0.x
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS closed issue #13543: [Bug] [plugin] shell task plugin replace time variable with a wrong time
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS closed issue #13543: [Bug] [plugin] shell task plugin replace time variable with a wrong time
URL: https://github.com/apache/dolphinscheduler/issues/13543
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13543: [Bug] [plugin] shell task plugin replace time variable with a wrong time
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13543:
URL: https://github.com/apache/dolphinscheduler/issues/13543#issuecomment-1425536706
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information, version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] SbloodyS commented on issue #13543: [Bug] [plugin] shell task plugin replace time variable with a wrong time
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #13543:
URL: https://github.com/apache/dolphinscheduler/issues/13543#issuecomment-1429195773
This issue has been fixed in #10376.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org