You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/02/19 00:50:59 UTC
[GitHub] [dolphinscheduler] jiangjialiang918 opened a new issue #8436: [task execute fail] [shell task/python task] shell/python script execute,report (No such file or directory)
jiangjialiang918 opened a new issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
在我安装了分布式集群之后没有任何错误,所有节点和进程也都能正常起来,当我创建一个shell 脚本作为测试时却发现任务总是失败,已经确定不是权限问题,因为使用的部署用户(dolphinscheduler)作为租户,是拥有dolphinscheduler路径下所有权限的,而且已经创建了task execute path : ./jobs/exec/process/4577755613088/4577887155744_2/2/28 ,
执行文件 ./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28.command 却没看到
日志见重现步骤(ps.文件上传总是失败)
安装配置如下:
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# ---------------------------------------------------------
# INSTALL MACHINE
# ---------------------------------------------------------
# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudo-distributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
ips="data01,data02,data03"
# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort="22"
# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
masters="data01"
# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers="data01:default,data02:default,data03:default"
# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
alertServer="data03"
# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers="data01"
# A comma separated list of machine hostname or IP would be installed Python gateway server, it
# must be a subset of configuration `ips`.
# Example for hostname: pythonGatewayServers="ds1", Example for IP: pythonGatewayServers="192.168.8.1"
pythonGatewayServers="data01"
# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd)
installPath="/data1_1T/dolphinscheduler"
# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser="dolphinscheduler"
# The directory to store local data for all machine we config above. Make sure user `deployUser` have permissions to read and write this directory.
dataBasedirPath="~/dolphinscheduler"
# ---------------------------------------------------------
# DolphinScheduler ENV
# ---------------------------------------------------------
# JAVA_HOME, we recommend use same JAVA_HOME in all machine you going to install DolphinScheduler
# and this configuration only support one parameter so far.
javaHome="/opt/dev/jdk1.8.0_212"
# DolphinScheduler API service port, also this is your DolphinScheduler UI component's URL port, default value is 12345
apiServerPort="12345"
# ---------------------------------------------------------
# Database
# NOTICE: If database value has special characters, such as `.*[]^${}\+?|()@#&`, Please add prefix `\` for escaping.
# ---------------------------------------------------------
# The type for the metadata database
# Supported values: ``postgresql``, ``mysql`, `h2``.
# DATABASE_TYPE=${DATABASE_TYPE:-"h2"}
DATABASE_TYPE="mysql"
SPRING_DATASOURCE_URL="jdbc:mysql://os7:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8"
# Spring datasource url, following <HOST>:<PORT>/<database>?<parameter> format, If you using mysql, you could use jdbc
# string jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8 as example
SPRING_DATASOURCE_URL=${SPRING_DATASOURCE_URL:-"jdbc:h2:mem:dolphinscheduler;MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=true"}
# Spring datasource username
SPRING_DATASOURCE_USERNAME="op"
# Spring datasource password
SPRING_DATASOURCE_PASSWORD="123456"
# ---------------------------------------------------------
# Registry Server
# ---------------------------------------------------------
# Registry Server plugin name, should be a substring of `registryPluginDir`, DolphinScheduler use this for verifying configuration consistency
registryPluginName="zookeeper"
# Registry Server address.
registryServers="data01:2181,data02:2181,data03:2181"
# The root of zookeeper, for now DolphinScheduler default registry server is zookeeper.
zkRoot="/dolphinscheduler"
# ---------------------------------------------------------
# Worker Task Server
# ---------------------------------------------------------
# Worker Task Server plugin dir. DolphinScheduler will find and load the worker task plugin jar package from this dir.
taskPluginDir="lib/plugin/task"
# resource storage type: HDFS, S3, NONE
resourceStorageType="NONE"
# resource store on HDFS/S3 path, resource file will store to this hdfs path, self configuration, please make sure the directory exists on hdfs and has read write permissions. "/dolphinscheduler" is recommended
resourceUploadPath="/dolphinscheduler"
# if resourceStorageType is HDFS，defaultFS write namenode address，HA, you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3，write S3 address，HA，for example ：s3a://dolphinscheduler，
# Note，S3 be sure to create the root directory /dolphinscheduler
defaultFS="hdfs://mycluster:8020"
# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"
# resourcemanager port, the default value is 8088 if not specified
resourceManagerHttpAddressPort="8088"
# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single node, keep this value empty
yarnHaIps="192.168.xx.xx,192.168.xx.xx"
# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace 'yarnIp1' to actual resourcemanager hostname
singleYarnIp="yarnIp1"
# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="hdfs"
# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username,watch out the @ sign should followd by \\
keytabUserName="hdfs-mycluster\\@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"
# kerberos expire time, the unit is hour
kerberosExpireTime="2"
# use sudo or not
sudoEnable="true"
# worker tenant auto create
workerTenantAutoCreate="false"
### What you expected to happen
能够正常执行shell等任务
### How to reproduce
部署完集群之后,确定所有进程都能正常启动,
使用登录ds web管理页面,
创建一个普通的test用户,使用默认添加的dolphinscheduler(部署用户),为test创建令牌,提交令牌,确认有效期,
使用test用户登录ds web,
创建一个shell 工作流,只需要一个节点, 脚本内容: echo 'hello dolphin' ,
保存工作流,上线,手动运行,可以得到运行日志如下:
[LOG-PATH]: /data1_1T/dolphinscheduler/logs/4577887155744_2/2/28.log, [HOST]: 192.168.100.102
[INFO] 2022-02-19 00:29:00.378 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[83] - shell task params {"resourceList":[],"localParams":[],"rawScript":"echo 'hello dolphin'","dependence":{},"conditionResult":{"successNode":[],"failedNode":[]},"waitStartTimeout":{},"switchResult":{}}
[INFO] 2022-02-19 00:29:00.424 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[137] - raw script : echo 'hello dolphin'
[INFO] 2022-02-19 00:29:00.425 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[138] - task execute path : ./jobs/exec/process/4577755613088/4577887155744_2/2/28
[INFO] 2022-02-19 00:29:00.596 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[86] - tenantCode user:dolphinscheduler, task dir:2_28
[INFO] 2022-02-19 00:29:00.596 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[91] - create command file:./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28.command
[INFO] 2022-02-19 00:29:00.597 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[117] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /data1_1T/dolphinscheduler/conf/env/dolphinscheduler_env.sh
./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28_node.sh
[INFO] 2022-02-19 00:29:00.622 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[285] - task run command: sudo -u dolphinscheduler sh ./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28.command
[INFO] 2022-02-19 00:29:00.625 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[176] - process start, process id is: 101187
[INFO] 2022-02-19 00:29:00.647 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[200] - process has exited, execute path:./jobs/exec/process/4577755613088/4577887155744_2/2/28, processId:101187 ,exitStatusCode:127 ,processWaitForStatus:true ,processExitValue:127
[INFO] 2022-02-19 00:29:01.627 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[66] - -> welcome to use bigdata scheduling system...
sh: ./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28.command: No such file or directory
[INFO] 2022-02-19 00:29:01.628 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[60] - FINALIZE_SESSION
### Anything else
每次都能重现,重启所有进程还是一样
### Version
2.0.2
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jiangjialiang918 commented on issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
jiangjialiang918 commented on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1045832950
> @jiangjialiang918 Hi, please use English, and format the description with the right markdown grammar.
Ok,Has been modified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] removed a comment on issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] removed a comment on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1045456399
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
After I installed the distributed cluster without any errors, all nodes and processes can be normal, when I created a shell script as a test but found that the task always fails, it has been determined that it is not a permission problem, because the deployment user used ( dolphinscheduler) as a tenant, has all permissions under the dolphinscheduler path, and has created a task execute path: ./jobs/exec/process/4577755613088/4577887155744_2/2/28 ,
Execute the file ./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28.command but don't see it
See the log for the reproduction steps (ps. file upload always fails)
The installation configuration is as follows:
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# ------------------------------------------------- --------
# INSTALL MACHINE
# ------------------------------------------------- --------
# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudo-distributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
ips="data01,data02,data03"
# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort="22"
# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
masters="data01"
# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers="data01:default,data02:default,data03:default"
# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
alertServer="data03"
# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers="data01"
# A comma separated list of machine hostname or IP would be installed Python gateway server, it
# must be a subset of configuration `ips`.
# Example for hostname: pythonGatewayServers="ds1", Example for IP: pythonGatewayServers="192.168.8.1"
pythonGatewayServers="data01"
# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd)
installPath="/data1_1T/dolphinscheduler"
# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser="dolphinscheduler"
# The directory to store local data for all machine we config above. Make sure user `deployUser` have permissions to read and write this directory.
dataBasedirPath="~/dolphinscheduler"
# ------------------------------------------------- --------
#DolphinScheduler ENV
# ------------------------------------------------- --------
# JAVA_HOME, we recommend use same JAVA_HOME in all machine you going to install DolphinScheduler
# and this configuration only support one parameter so far.
javaHome="/opt/dev/jdk1.8.0_212"
# DolphinScheduler API service port, also this is your DolphinScheduler UI component's URL port, default value is 12345
apiServerPort="12345"
# ------------------------------------------------- --------
# Database
# NOTICE: If database value has special characters, such as `.*[]^${}\+?|()@#&`, Please add prefix `\` for escaping.
# ------------------------------------------------- --------
# The type for the metadata database
# Supported values: ``postgresql``, ``mysql`, `h2``.
# DATABASE_TYPE=${DATABASE_TYPE:-"h2"}
DATABASE_TYPE="mysql"
SPRING_DATASOURCE_URL="jdbc:mysql://os7:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8"
# Spring datasource url, following <HOST>:<PORT>/<database>?<parameter> format, If you using mysql, you could use jdbc
# string jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8 as example
SPRING_DATASOURCE_URL=${SPRING_DATASOURCE_URL:-"jdbc:h2:mem:dolphinscheduler;MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=true"}
# Spring datasource username
SPRING_DATASOURCE_USERNAME="op"
# Spring datasource password
SPRING_DATASOURCE_PASSWORD="123456"
# ------------------------------------------------- --------
# Registry Server
# ------------------------------------------------- --------
# Registry Server plugin name, should be a substring of `registryPluginDir`, DolphinScheduler use this for verifying configuration consistency
registryPluginName="zookeeper"
# Registry Server address.
registryServers="data01:2181,data02:2181,data03:2181"
# The root of zookeeper, for now DolphinScheduler default registry server is zookeeper.
zkRoot="/dolphinscheduler"
# ------------------------------------------------- --------
# Worker Task Server
# ------------------------------------------------- --------
# Worker Task Server plugin dir. DolphinScheduler will find and load the worker task plugin jar package from this dir.
taskPluginDir="lib/plugin/task"
# resource storage type: HDFS, S3, NONE
resourceStorageType="NONE"
# resource store on HDFS/S3 path, resource file will store to this hdfs path, self configuration, please make sure the directory exists on hdfs and has read write permissions. "/dolphinscheduler" is recommended
resourceUploadPath="/dolphinscheduler"
# if resourceStorageType is HDFS, defaultFS write namenode address, HA, you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3, write S3 address, HA, for example: s3a://dolphinscheduler,
# Note, S3 be sure to create the root directory /dolphinscheduler
defaultFS="hdfs://mycluster:8020"
# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"
# resourcemanager port, the default value is 8088 if not specified
resourceManagerHttpAddressPort="8088"
# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single node, keep this value empty
yarnHaIps="192.168.xx.xx,192.168.xx.xx"
# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace 'yarnIp1' to actual resourcemanager hostname
singleYarnIp="yarnIp1"
# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="hdfs"
# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username, watch out the @ sign should be followed by \\
keytabUserName="hdfs-mycluster\\@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"
# kerberos expire time, the unit is hour
kerberosExpireTime="2"
# use sudo or not
sudoEnable="true"
# tenant worker auto create
workerTenantAutoCreate="false"
### What you expected to happen
Can perform tasks such as shell normally
### How to reproduce
After deploying the cluster, make sure that all processes can start normally,
Use the login ds web management page,
Create a normal test user, use the dolphinscheduler added by default (deployment user), create a token for test, submit the token, confirm the validity period,
Use the test user to log in to ds web,
Create a shell workflow, only need one node, script content: echo 'hello dolphin' ,
Save the workflow, go online, run it manually, and you can get the running log as follows:
[LOG-PATH]: /data1_1T/dolphinscheduler/logs/4577887155744_2/2/28.log, [HOST]: 192.168.100.102
[INFO] 2022-02-19 00:29:00.378 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[83] - shell task params {"resourceList":[],"localParams":[ ],"rawScript":"echo 'hello dolphin'","dependence":{},"conditionResult":{"successNode":[],"failedNode":[]},"waitStartTimeout":{},"switchResult ":{}}
[INFO] 2022-02-19 00:29:00.424 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[137] - raw script : echo 'hello dolphin'
[INFO] 2022-02-19 00:29:00.425 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[138] - task execute path : ./jobs/exec/process/4577755613088/4577887155744_2/ 2/28
[INFO] 2022-02-19 00:29:00.596 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[86] - tenantCode user:dolphinscheduler, task dir:2_28
[INFO] 2022-02-19 00:29:00.596 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[91] - create command file:./jobs/exec/process/4577755613088/4577887155744_2/ 2/28/2_28.command
[INFO] 2022-02-19 00:29:00.597 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[117] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /data1_1T/dolphinscheduler/conf/env/dolphinscheduler_env.sh
./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28_node.sh
[INFO] 2022-02-19 00:29:00.622 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[285] - task run command: sudo -u dolphinscheduler sh ./jobs/exec/process /4577755613088/4577887155744_2/2/28/2_28.command
[INFO] 2022-02-19 00:29:00.625 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[176] - process start, process id is: 101187
[INFO] 2022-02-19 00:29:00.647 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[200] - process has exited, execute path:./jobs/exec/process/4577755613088 /4577887155744_2/2/28, processId:101187 ,exitStatusCode:127 ,processWaitForStatus:true ,processExitValue:127
[INFO] 2022-02-19 00:29:01.627 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[66] - -> welcome to use bigdata scheduling system...
sh: ./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28.command: No such file or directory
[INFO] 2022-02-19 00:29:01.628 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[60] - FINALIZE_SESSION
### Anything else
Reproduces every time, restarting all processes is still the same
### Version
2.0.2
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jiangjialiang918 removed a comment on issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
jiangjialiang918 removed a comment on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1046572255
No problem, but I think this problem should be a small one, I think they will not accept my PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1046526120
> > > dataBasedirPath = "./jobs" updated in ./conf/common.propertie after installed,Because no change as well as can't find the path ~ / dolphinscheduler/exec/process/XXXX
> >
> >
> > You mean `data.basedir.path`? Could you please change this to absolute path and restart the server? And then try again.
>
> Thank you, the problem has been solved, I feel this problem can be written on the document as a matter of attention
Hi, are you interested to optimize this in document? Welcome to submit a pr to fix the document, this might be an important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1046919939
@jiangjialiang918 Don't worry, I will help to review your pr, if you ping me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #8436: [task execute fail] [shell task/python task] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1045456773
Hi:
* Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/en-us/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1045695360
@jiangjialiang918 Hi, please use English, and format the description with the right markdown grammar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #8436: [task execute fail] [shell task/python task] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1045456399
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
After I installed the distributed cluster without any errors, all nodes and processes can be normal, when I created a shell script as a test but found that the task always fails, it has been determined that it is not a permission problem, because the deployment user used ( dolphinscheduler) as a tenant, has all permissions under the dolphinscheduler path, and has created a task execute path: ./jobs/exec/process/4577755613088/4577887155744_2/2/28 ,
Execute the file ./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28.command but don't see it
See the log for the reproduction steps (ps. file upload always fails)
The installation configuration is as follows:
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# ------------------------------------------------- --------
# INSTALL MACHINE
# ------------------------------------------------- --------
# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudo-distributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
ips="data01,data02,data03"
# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort="22"
# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
masters="data01"
# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers="data01:default,data02:default,data03:default"
# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
alertServer="data03"
# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers="data01"
# A comma separated list of machine hostname or IP would be installed Python gateway server, it
# must be a subset of configuration `ips`.
# Example for hostname: pythonGatewayServers="ds1", Example for IP: pythonGatewayServers="192.168.8.1"
pythonGatewayServers="data01"
# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd)
installPath="/data1_1T/dolphinscheduler"
# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser="dolphinscheduler"
# The directory to store local data for all machine we config above. Make sure user `deployUser` have permissions to read and write this directory.
dataBasedirPath="~/dolphinscheduler"
# ------------------------------------------------- --------
#DolphinScheduler ENV
# ------------------------------------------------- --------
# JAVA_HOME, we recommend use same JAVA_HOME in all machine you going to install DolphinScheduler
# and this configuration only support one parameter so far.
javaHome="/opt/dev/jdk1.8.0_212"
# DolphinScheduler API service port, also this is your DolphinScheduler UI component's URL port, default value is 12345
apiServerPort="12345"
# ------------------------------------------------- --------
# Database
# NOTICE: If database value has special characters, such as `.*[]^${}\+?|()@#&`, Please add prefix `\` for escaping.
# ------------------------------------------------- --------
# The type for the metadata database
# Supported values: ``postgresql``, ``mysql`, `h2``.
# DATABASE_TYPE=${DATABASE_TYPE:-"h2"}
DATABASE_TYPE="mysql"
SPRING_DATASOURCE_URL="jdbc:mysql://os7:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8"
# Spring datasource url, following <HOST>:<PORT>/<database>?<parameter> format, If you using mysql, you could use jdbc
# string jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8 as example
SPRING_DATASOURCE_URL=${SPRING_DATASOURCE_URL:-"jdbc:h2:mem:dolphinscheduler;MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=true"}
# Spring datasource username
SPRING_DATASOURCE_USERNAME="op"
# Spring datasource password
SPRING_DATASOURCE_PASSWORD="123456"
# ------------------------------------------------- --------
# Registry Server
# ------------------------------------------------- --------
# Registry Server plugin name, should be a substring of `registryPluginDir`, DolphinScheduler use this for verifying configuration consistency
registryPluginName="zookeeper"
# Registry Server address.
registryServers="data01:2181,data02:2181,data03:2181"
# The root of zookeeper, for now DolphinScheduler default registry server is zookeeper.
zkRoot="/dolphinscheduler"
# ------------------------------------------------- --------
# Worker Task Server
# ------------------------------------------------- --------
# Worker Task Server plugin dir. DolphinScheduler will find and load the worker task plugin jar package from this dir.
taskPluginDir="lib/plugin/task"
# resource storage type: HDFS, S3, NONE
resourceStorageType="NONE"
# resource store on HDFS/S3 path, resource file will store to this hdfs path, self configuration, please make sure the directory exists on hdfs and has read write permissions. "/dolphinscheduler" is recommended
resourceUploadPath="/dolphinscheduler"
# if resourceStorageType is HDFS, defaultFS write namenode address, HA, you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3, write S3 address, HA, for example: s3a://dolphinscheduler,
# Note, S3 be sure to create the root directory /dolphinscheduler
defaultFS="hdfs://mycluster:8020"
# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"
# resourcemanager port, the default value is 8088 if not specified
resourceManagerHttpAddressPort="8088"
# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single node, keep this value empty
yarnHaIps="192.168.xx.xx,192.168.xx.xx"
# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace 'yarnIp1' to actual resourcemanager hostname
singleYarnIp="yarnIp1"
# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="hdfs"
# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username, watch out the @ sign should be followed by \\
keytabUserName="hdfs-mycluster\\@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"
# kerberos expire time, the unit is hour
kerberosExpireTime="2"
# use sudo or not
sudoEnable="true"
# tenant worker auto create
workerTenantAutoCreate="false"
### What you expected to happen
Can perform tasks such as shell normally
### How to reproduce
After deploying the cluster, make sure that all processes can start normally,
Use the login ds web management page,
Create a normal test user, use the dolphinscheduler added by default (deployment user), create a token for test, submit the token, confirm the validity period,
Use the test user to log in to ds web,
Create a shell workflow, only need one node, script content: echo 'hello dolphin' ,
Save the workflow, go online, run it manually, and you can get the running log as follows:
[LOG-PATH]: /data1_1T/dolphinscheduler/logs/4577887155744_2/2/28.log, [HOST]: 192.168.100.102
[INFO] 2022-02-19 00:29:00.378 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[83] - shell task params {"resourceList":[],"localParams":[ ],"rawScript":"echo 'hello dolphin'","dependence":{},"conditionResult":{"successNode":[],"failedNode":[]},"waitStartTimeout":{},"switchResult ":{}}
[INFO] 2022-02-19 00:29:00.424 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[137] - raw script : echo 'hello dolphin'
[INFO] 2022-02-19 00:29:00.425 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[138] - task execute path : ./jobs/exec/process/4577755613088/4577887155744_2/ 2/28
[INFO] 2022-02-19 00:29:00.596 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[86] - tenantCode user:dolphinscheduler, task dir:2_28
[INFO] 2022-02-19 00:29:00.596 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[91] - create command file:./jobs/exec/process/4577755613088/4577887155744_2/ 2/28/2_28.command
[INFO] 2022-02-19 00:29:00.597 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[117] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /data1_1T/dolphinscheduler/conf/env/dolphinscheduler_env.sh
./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28_node.sh
[INFO] 2022-02-19 00:29:00.622 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[285] - task run command: sudo -u dolphinscheduler sh ./jobs/exec/process /4577755613088/4577887155744_2/2/28/2_28.command
[INFO] 2022-02-19 00:29:00.625 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[176] - process start, process id is: 101187
[INFO] 2022-02-19 00:29:00.647 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[200] - process has exited, execute path:./jobs/exec/process/4577755613088 /4577887155744_2/2/28, processId:101187 ,exitStatusCode:127 ,processWaitForStatus:true ,processExitValue:127
[INFO] 2022-02-19 00:29:01.627 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[66] - -> welcome to use bigdata scheduling system...
sh: ./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28.command: No such file or directory
[INFO] 2022-02-19 00:29:01.628 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[60] - FINALIZE_SESSION
### Anything else
Reproduces every time, restarting all processes is still the same
### Version
2.0.2
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jiangjialiang918 commented on issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
jiangjialiang918 commented on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1046572255
No problem, but I think this problem should be a small one, I think they will not accept my PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jiangjialiang918 commented on issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
jiangjialiang918 commented on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1046572514
> > > > dataBasedirPath = "./jobs" updated in ./conf/common.propertie after installed,Because no change as well as can't find the path ~ / dolphinscheduler/exec/process/XXXX
> > >
> > >
> > > You mean `data.basedir.path`? Could you please change this to absolute path and restart the server? And then try again.
> >
> >
> > Thank you, the problem has been solved, I feel this problem can be written on the document as a matter of attention
>
> Hi, are you interested to optimize this in document? Welcome to submit a pr to fix the document, this might be an important.
No problem, but I think this problem should be a small one, I think they will not accept my PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jiangjialiang918 commented on issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
jiangjialiang918 commented on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1045989940
dataBasedirPath = "./jobs" updated in ./conf/common.propertie after installed,Because no change as well as can't find the path ~ / dolphinscheduler/exec/process/XXXX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] ruanwenjun commented on issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
ruanwenjun commented on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1046007920
> dataBasedirPath = "./jobs" updated in ./conf/common.propertie after installed,Because no change as well as can't find the path ~ / dolphinscheduler/exec/process/XXXX
You mean `data.basedir.path`? Could you please change this to absolute path and restart the server? And then try again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jiangjialiang918 commented on issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
jiangjialiang918 commented on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1046017894
> > dataBasedirPath = "./jobs" updated in ./conf/common.propertie after installed,Because no change as well as can't find the path ~ / dolphinscheduler/exec/process/XXXX
>
> You mean `data.basedir.path`? Could you please change this to absolute path and restart the server? And then try again.
Thank you, the problem has been solved, I feel this problem can be written on the document as a matter of attention
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jiangjialiang918 closed issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
Posted by GitBox <gi...@apache.org>.
jiangjialiang918 closed issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org