You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/02/19 12:12:01 UTC
[GitHub] [dolphinscheduler] github-actions[bot] removed a comment on issue #8436: [BUG][task execute fail] shell/python script execute,report (No such file or directory)
github-actions[bot] removed a comment on issue #8436:
URL: https://github.com/apache/dolphinscheduler/issues/8436#issuecomment-1045456399
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
After I installed the distributed cluster without any errors, all nodes and processes can be normal, when I created a shell script as a test but found that the task always fails, it has been determined that it is not a permission problem, because the deployment user used ( dolphinscheduler) as a tenant, has all permissions under the dolphinscheduler path, and has created a task execute path: ./jobs/exec/process/4577755613088/4577887155744_2/2/28 ,
Execute the file ./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28.command but don't see it
See the log for the reproduction steps (ps. file upload always fails)
The installation configuration is as follows:
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# ------------------------------------------------- --------
# INSTALL MACHINE
# ------------------------------------------------- --------
# A comma separated list of machine hostname or IP would be installed DolphinScheduler,
# including master, worker, api, alert. If you want to deploy in pseudo-distributed
# mode, just write a pseudo-distributed hostname
# Example for hostnames: ips="ds1,ds2,ds3,ds4,ds5", Example for IPs: ips="192.168.8.1,192.168.8.2,192.168.8.3,192.168.8.4,192.168.8.5"
ips="data01,data02,data03"
# Port of SSH protocol, default value is 22. For now we only support same port in all `ips` machine
# modify it if you use different ssh port
sshPort="22"
# A comma separated list of machine hostname or IP would be installed Master server, it
# must be a subset of configuration `ips`.
# Example for hostnames: masters="ds1,ds2", Example for IPs: masters="192.168.8.1,192.168.8.2"
masters="data01"
# A comma separated list of machine <hostname>:<workerGroup> or <IP>:<workerGroup>.All hostname or IP must be a
# subset of configuration `ips`, And workerGroup have default value as `default`, but we recommend you declare behind the hosts
# Example for hostnames: workers="ds1:default,ds2:default,ds3:default", Example for IPs: workers="192.168.8.1:default,192.168.8.2:default,192.168.8.3:default"
workers="data01:default,data02:default,data03:default"
# A comma separated list of machine hostname or IP would be installed Alert server, it
# must be a subset of configuration `ips`.
# Example for hostname: alertServer="ds3", Example for IP: alertServer="192.168.8.3"
alertServer="data03"
# A comma separated list of machine hostname or IP would be installed API server, it
# must be a subset of configuration `ips`.
# Example for hostname: apiServers="ds1", Example for IP: apiServers="192.168.8.1"
apiServers="data01"
# A comma separated list of machine hostname or IP would be installed Python gateway server, it
# must be a subset of configuration `ips`.
# Example for hostname: pythonGatewayServers="ds1", Example for IP: pythonGatewayServers="192.168.8.1"
pythonGatewayServers="data01"
# The directory to install DolphinScheduler for all machine we config above. It will automatically be created by `install.sh` script if not exists.
# Do not set this configuration same as the current path (pwd)
installPath="/data1_1T/dolphinscheduler"
# The user to deploy DolphinScheduler for all machine we config above. For now user must create by yourself before running `install.sh`
# script. The user needs to have sudo privileges and permissions to operate hdfs. If hdfs is enabled than the root directory needs
# to be created by this user
deployUser="dolphinscheduler"
# The directory to store local data for all machine we config above. Make sure user `deployUser` have permissions to read and write this directory.
dataBasedirPath="~/dolphinscheduler"
# ------------------------------------------------- --------
#DolphinScheduler ENV
# ------------------------------------------------- --------
# JAVA_HOME, we recommend use same JAVA_HOME in all machine you going to install DolphinScheduler
# and this configuration only support one parameter so far.
javaHome="/opt/dev/jdk1.8.0_212"
# DolphinScheduler API service port, also this is your DolphinScheduler UI component's URL port, default value is 12345
apiServerPort="12345"
# ------------------------------------------------- --------
# Database
# NOTICE: If database value has special characters, such as `.*[]^${}\+?|()@#&`, Please add prefix `\` for escaping.
# ------------------------------------------------- --------
# The type for the metadata database
# Supported values: ``postgresql``, ``mysql`, `h2``.
# DATABASE_TYPE=${DATABASE_TYPE:-"h2"}
DATABASE_TYPE="mysql"
SPRING_DATASOURCE_URL="jdbc:mysql://os7:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8"
# Spring datasource url, following <HOST>:<PORT>/<database>?<parameter> format, If you using mysql, you could use jdbc
# string jdbc:mysql://127.0.0.1:3306/dolphinscheduler?useUnicode=true&characterEncoding=UTF-8 as example
SPRING_DATASOURCE_URL=${SPRING_DATASOURCE_URL:-"jdbc:h2:mem:dolphinscheduler;MODE=MySQL;DB_CLOSE_DELAY=-1;DATABASE_TO_LOWER=true"}
# Spring datasource username
SPRING_DATASOURCE_USERNAME="op"
# Spring datasource password
SPRING_DATASOURCE_PASSWORD="123456"
# ------------------------------------------------- --------
# Registry Server
# ------------------------------------------------- --------
# Registry Server plugin name, should be a substring of `registryPluginDir`, DolphinScheduler use this for verifying configuration consistency
registryPluginName="zookeeper"
# Registry Server address.
registryServers="data01:2181,data02:2181,data03:2181"
# The root of zookeeper, for now DolphinScheduler default registry server is zookeeper.
zkRoot="/dolphinscheduler"
# ------------------------------------------------- --------
# Worker Task Server
# ------------------------------------------------- --------
# Worker Task Server plugin dir. DolphinScheduler will find and load the worker task plugin jar package from this dir.
taskPluginDir="lib/plugin/task"
# resource storage type: HDFS, S3, NONE
resourceStorageType="NONE"
# resource store on HDFS/S3 path, resource file will store to this hdfs path, self configuration, please make sure the directory exists on hdfs and has read write permissions. "/dolphinscheduler" is recommended
resourceUploadPath="/dolphinscheduler"
# if resourceStorageType is HDFS, defaultFS write namenode address, HA, you need to put core-site.xml and hdfs-site.xml in the conf directory.
# if S3, write S3 address, HA, for example: s3a://dolphinscheduler,
# Note, S3 be sure to create the root directory /dolphinscheduler
defaultFS="hdfs://mycluster:8020"
# if resourceStorageType is S3, the following three configuration is required, otherwise please ignore
s3Endpoint="http://192.168.xx.xx:9010"
s3AccessKey="xxxxxxxxxx"
s3SecretKey="xxxxxxxxxx"
# resourcemanager port, the default value is 8088 if not specified
resourceManagerHttpAddressPort="8088"
# if resourcemanager HA is enabled, please set the HA IPs; if resourcemanager is single node, keep this value empty
yarnHaIps="192.168.xx.xx,192.168.xx.xx"
# if resourcemanager HA is enabled or not use resourcemanager, please keep the default value; If resourcemanager is single node, you only need to replace 'yarnIp1' to actual resourcemanager hostname
singleYarnIp="yarnIp1"
# who has permission to create directory under HDFS/S3 root path
# Note: if kerberos is enabled, please config hdfsRootUser=
hdfsRootUser="hdfs"
# kerberos config
# whether kerberos starts, if kerberos starts, following four items need to config, otherwise please ignore
kerberosStartUp="false"
# kdc krb5 config file path
krb5ConfPath="$installPath/conf/krb5.conf"
# keytab username, watch out the @ sign should be followed by \\
keytabUserName="hdfs-mycluster\\@ESZ.COM"
# username keytab path
keytabPath="$installPath/conf/hdfs.headless.keytab"
# kerberos expire time, the unit is hour
kerberosExpireTime="2"
# use sudo or not
sudoEnable="true"
# tenant worker auto create
workerTenantAutoCreate="false"
### What you expected to happen
Can perform tasks such as shell normally
### How to reproduce
After deploying the cluster, make sure that all processes can start normally,
Use the login ds web management page,
Create a normal test user, use the dolphinscheduler added by default (deployment user), create a token for test, submit the token, confirm the validity period,
Use the test user to log in to ds web,
Create a shell workflow, only need one node, script content: echo 'hello dolphin' ,
Save the workflow, go online, run it manually, and you can get the running log as follows:
[LOG-PATH]: /data1_1T/dolphinscheduler/logs/4577887155744_2/2/28.log, [HOST]: 192.168.100.102
[INFO] 2022-02-19 00:29:00.378 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[83] - shell task params {"resourceList":[],"localParams":[ ],"rawScript":"echo 'hello dolphin'","dependence":{},"conditionResult":{"successNode":[],"failedNode":[]},"waitStartTimeout":{},"switchResult ":{}}
[INFO] 2022-02-19 00:29:00.424 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[137] - raw script : echo 'hello dolphin'
[INFO] 2022-02-19 00:29:00.425 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[138] - task execute path : ./jobs/exec/process/4577755613088/4577887155744_2/ 2/28
[INFO] 2022-02-19 00:29:00.596 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[86] - tenantCode user:dolphinscheduler, task dir:2_28
[INFO] 2022-02-19 00:29:00.596 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[91] - create command file:./jobs/exec/process/4577755613088/4577887155744_2/ 2/28/2_28.command
[INFO] 2022-02-19 00:29:00.597 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[117] - command : #!/bin/sh
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
source /data1_1T/dolphinscheduler/conf/env/dolphinscheduler_env.sh
./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28_node.sh
[INFO] 2022-02-19 00:29:00.622 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[285] - task run command: sudo -u dolphinscheduler sh ./jobs/exec/process /4577755613088/4577887155744_2/2/28/2_28.command
[INFO] 2022-02-19 00:29:00.625 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[176] - process start, process id is: 101187
[INFO] 2022-02-19 00:29:00.647 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[200] - process has exited, execute path:./jobs/exec/process/4577755613088 /4577887155744_2/2/28, processId:101187 ,exitStatusCode:127 ,processWaitForStatus:true ,processExitValue:127
[INFO] 2022-02-19 00:29:01.627 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[66] - -> welcome to use bigdata scheduling system...
sh: ./jobs/exec/process/4577755613088/4577887155744_2/2/28/2_28.command: No such file or directory
[INFO] 2022-02-19 00:29:01.628 TaskLogLogger-class org.apache.dolphinscheduler.plugin.task.shell.ShellTask:[60] - FINALIZE_SESSION
### Anything else
Reproduces every time, restarting all processes is still the same
### Version
2.0.2
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org