You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "lakako (via GitHub)" <gi...@apache.org> on 2023/11/28 08:32:53 UTC
[I] [Bug] [Resource/file-manange] Failed to get file on shell task [dolphinscheduler]
lakako opened a new issue, #15236:
URL: https://github.com/apache/dolphinscheduler/issues/15236
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
I have uploaded a file to file manage. When I want to use this file on shell task, happened "no such file or directory"
version: 3.2.0 standalone-server
### What you expected to happen
use file on shell task correctly.
### How to reproduce
upload file and create a shell task. select that file and use it in shell task.
### Anything else
_No response_
### Version
3.2.x
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Resource/file-manange] Failed to get file on shell task [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #15236:
URL: https://github.com/apache/dolphinscheduler/issues/15236#issuecomment-2230215347
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Resource/file-manange] Failed to get file on shell task [dolphinscheduler]
Posted by "lakako (via GitHub)" <gi...@apache.org>.
lakako commented on issue #15236:
URL: https://github.com/apache/dolphinscheduler/issues/15236#issuecomment-1835531226
```
[INFO] 2023-12-01 14:14:40.702 +0800 - ->
cat: /test.txt: No such file or directory
cat: test.txt: No such file or directory
root@295f2097dcc1:/opt/dolphinscheduler# cat /dolphinscheduler/default/resources/test.txt
hhhh
cxzcxz
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Resource/file-manange] Failed to get file on shell task [dolphinscheduler]
Posted by "git-blame (via GitHub)" <gi...@apache.org>.
git-blame commented on issue #15236:
URL: https://github.com/apache/dolphinscheduler/issues/15236#issuecomment-1924031313
This path problem also affects other task such as Spark Task including the main JAR file. However, since Dolphin will use the full JAR path when constructing the `spark-submit` command, the task still completes successfully.
But spark options like `--files` are still affected since specifying `--files myfile.txt` in the task definition will not work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Resource/file-manange] Failed to get file on shell task [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS closed issue #15236: [Bug] [Resource/file-manange] Failed to get file on shell task
URL: https://github.com/apache/dolphinscheduler/issues/15236
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Resource/file-manange] Failed to get file on shell task [dolphinscheduler]
Posted by "fuchanghai (via GitHub)" <gi...@apache.org>.
fuchanghai commented on issue #15236:
URL: https://github.com/apache/dolphinscheduler/issues/15236#issuecomment-1831724127
hi @lakako you can try ```cat /test .txt```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Resource/file-manange] Failed to get file on shell task [dolphinscheduler]
Posted by "git-blame (via GitHub)" <gi...@apache.org>.
git-blame commented on issue #15236:
URL: https://github.com/apache/dolphinscheduler/issues/15236#issuecomment-1919171845
I have the same problem. The issue seems to be the file is placed with the wrong path in the execution directory. The entire `file:/` URI is treated like a directory path.
```log
[INFO] 2024-01-30 20:06:41.110 +0000 - get resource file from path:file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default/resources/hello.sh
[INFO] 2024-01-30 20:06:41.130 +0000 - Download resources: {file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default/resources/hello.sh=file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default/resources/hello.sh} successfully
...
[INFO] 2024-01-30 20:06:41.162 +0000 - Final Shell file is :
#!/bin/bash
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
echo "What's here: $BASEDIR"
find .
bash hello.sh
[INFO] 2024-01-30 20:06:41.164 +0000 - Executing shell command : sudo -u root -i /tmp/dolphinscheduler/exec/process/default/12443475537632/12443490756832_1/1/1/1_1.sh
[INFO] 2024-01-30 20:06:41.206 +0000 - process start, process id is: 43710
[INFO] 2024-01-30 20:06:42.204 +0000 - ->
What's here: /tmp/dolphinscheduler/exec/process/default/12443475537632/12443490756832_1/1/1
.
./file:
./file:/opt
./file:/opt/apache-dolphinscheduler-3.2.0-bin
./file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server
./file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files
./file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default
./file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default/resources
./file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default/resources/hello.sh
./1_1.sh
bash: hello.sh: No such file or directory
```
[1706645301047.log](https://github.com/apache/dolphinscheduler/files/14112667/1706645301047.log)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org