You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "wangjiapengone (via GitHub)" <gi...@apache.org> on 2023/10/25 09:03:13 UTC
[I] [Bug] [Module Name] Shell tasks report "no such file" error when using resource files stored on HDFS [dolphinscheduler]
wangjiapengone opened a new issue, #15077:
URL: https://github.com/apache/dolphinscheduler/issues/15077
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
I use HDFS to store resource files. The resource file is: `hdfs://hdfs-cluster01/dolphinscheduler_test/default/resources/test/hello.sh`
Then I create a shell task to execute this resource file
The task report "No such file or directory" error.
### What you expected to happen
I think this task should run successfully. Because it's good in 3.1.4, but fails in 3.2.0.
### How to reproduce
1. Use HDFS as resource storage type
2. Create a shell script on Resource Center.
3. Create a task which execute this shell script. Then the error occurs
### Anything else
_No response_
### Version
3.2.x
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Shell tasks report "no such file" error when using resource files stored on HDFS [dolphinscheduler]
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on issue #15077:
URL: https://github.com/apache/dolphinscheduler/issues/15077#issuecomment-1780680148
Good catch, thanks, any suggestion about this scenario? @SbloodyS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Shell tasks report "no such file" error when using resource files stored on HDFS [dolphinscheduler]
Posted by "L-hamilton (via GitHub)" <gi...@apache.org>.
L-hamilton commented on issue #15077:
URL: https://github.com/apache/dolphinscheduler/issues/15077#issuecomment-1815834762
你好，我遇到了一样的问题，但是我在文件管理上传文件时在集群中只有api节点会出现这些文件，执行的worker节点并没有该资源，api节点并没有下发这些文件到worker节点中，在执行调度用了资源中心的文件，日志中会显示该文件下载失败，并不存在。
另外地，我也做了hello.sh的测试，日志报错是一样的，我的hello.sh在resources路径下，执行的命令是sh hello.sh，我的文件在本地，完整路径是/dolphinscheduler/default/resources/hello.sh，同时我的执行用户和文件使用者是一样的，这个错误依旧存在。
Hello, I have encountered the same problem, but when I upload files in file management, only the api node in the cluster will have these files. The executing worker node does not have the resource, and the api node does not deliver these files to the worker node. When the file from resource center is used in the execution scheduling, the log will show that the file download fails. It doesn't exist.
In addition, I also did the hello.sh test, the log error is the same as you, my hello.sh is in the resources path, the command is sh hello.sh, my file is in the local, Full path is /dolphinscheduler/default/resources/hello. sh, at the same time I perform the executing user and the resource owner are the same, the error remains.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Shell tasks report "no such file" error when using resource files stored on HDFS [dolphinscheduler]
Posted by "Radeity (via GitHub)" <gi...@apache.org>.
Radeity commented on issue #15077:
URL: https://github.com/apache/dolphinscheduler/issues/15077#issuecomment-1780535708
Would you like to help check whether this bug exist in dev?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Shell tasks report "no such file" error when using resource files stored on HDFS [dolphinscheduler]
Posted by "wangjiapengone (via GitHub)" <gi...@apache.org>.
wangjiapengone commented on issue #15077:
URL: https://github.com/apache/dolphinscheduler/issues/15077#issuecomment-1780669178
> Hi @wangjiapengone , I don't reproduce this bug, please check whether you can download files from HDFS in your env.
Hi, I can still reproduce this bug in dev.
This problem occurs when the tenant executing the task differs from the tenant owning the resource.
For example,
The tenant of the current user is "default",
I create a new file in Resource Center which called "hello.sh". The HDFS file path is: `hdfs://hdfs-cluster01/dolphinscheduler_test/default/resources/test/hello.sh`
Then I create a new shell task to execute this file,
The task will use the system user to execute this task, and the local file is missing.
If I look at the temporary local file, it looks like this
Apparently, the local file path is undesirable. But if the executing user and the resource owner are the same, the problem disappears.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Shell tasks report "no such file" error when using resource files stored on HDFS [dolphinscheduler]
Posted by "ruanwenjun (via GitHub)" <gi...@apache.org>.
ruanwenjun commented on issue #15077:
URL: https://github.com/apache/dolphinscheduler/issues/15077#issuecomment-1781043741
This is caused by the file is downloaded by bootstrap user, but the file will be used by tenant, if you set `sudo.enable=true`, we will set the file owner to tenant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Shell tasks report "no such file" error when using resource files stored on HDFS [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15077:
URL: https://github.com/apache/dolphinscheduler/issues/15077#issuecomment-1859344235
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Shell tasks report "no such file" error when using resource files stored on HDFS [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #15077:
URL: https://github.com/apache/dolphinscheduler/issues/15077#issuecomment-1869164182
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Shell tasks report "no such file" error when using resource files stored on HDFS [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #15077: [Bug] Shell tasks report "no such file" error when using resource files stored on HDFS
URL: https://github.com/apache/dolphinscheduler/issues/15077
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] Shell tasks report "no such file" error when using resource files stored on HDFS [dolphinscheduler]
Posted by "wangjiapengone (via GitHub)" <gi...@apache.org>.
wangjiapengone commented on issue #15077:
URL: https://github.com/apache/dolphinscheduler/issues/15077#issuecomment-1792036234
> This is caused by the file is downloaded by bootstrap user, but the file will be used by tenant, if you set `sudo.enable=true`, we will set the file owner to tenant.
Sorry, I can't catch your idea. The above problem still shows when I set `sudo.enable=true`. In fact, I never set `sudo.enable=false`. So, I don't think change file owner will solve this problem. The real problem is the downloaded file path.
When I start a standalone server in debug mode, I can see the `fileName` returned by `getResourceFileName` is incorrect.
Since `resDir` is not the prefix of `fullName`, the replacement does't work, `fileName` and `fullName` are the same. So the actual file downloaded is `hdfs:/hdfs-cluster01/dolphinscheduler_test/default/resources/test/hello.sh`
I think the correct `fileName` should be `test/hello.sh`.
Please correct me if there's any wrong with my statement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org