You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by "njnu-seafish (via GitHub)" <gi...@apache.org> on 2025/09/10 01:59:50 UTC
[I] [Bug] [Resource Center] The resource file should not be deleted after it has been referenced by a task. [dolphinscheduler]
njnu-seafish opened a new issue, #17492:
URL: https://github.com/apache/dolphinscheduler/issues/17492
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar issues.
### What happened
1, Create a new shell file "suyc/testsh01.sh" in the Resource Center.
2, Create a shell task that references the resource file.
`bash suyc/testsh01.sh`
3, The shell task runs normally.
4, Delete the file from the Resource Center.
5, Run the shell task again, and the following exception message appears in the worker service log:
`[WI-0][TI-2902] - 2025-09-10 09:36:33.365 ERROR [exclusive-task-executor-container-worker-0] o.a.d.t.e.w.TaskExecutorWorker:[75] - PhysicalTaskExecutor{id=2902, name=sh_file, stat=INITIALIZED} execute failed
org.apache.dolphinscheduler.plugin.task.api.TaskException: Download resource file: hdfs://ctyunns/dolphinscheduler/default/resources/suyc/testsh01.sh error
at org.apache.dolphinscheduler.server.worker.utils.TaskExecutionContextUtils.downloadResourcesIfNeeded(TaskExecutionContextUtils.java:99)
at org.apache.dolphinscheduler.server.worker.executor.PhysicalTaskExecutor.initializeTaskContext(PhysicalTaskExecutor.java:119)
at org.apache.dolphinscheduler.task.executor.AbstractTaskExecutor.start(AbstractTaskExecutor.java:66)
at org.apache.dolphinscheduler.task.executor.worker.TaskExecutorWorker.start(TaskExecutorWorker.java:65)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.FileNotFoundException: File does not exist: hdfs://ctyunns/dolphinscheduler/default/resources/suyc/testsh01.sh
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1606)
at org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1599)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1614)
at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:483)
at org.apache.dolphinscheduler.plugin.storage.hdfs.HdfsStorageOperator.download(HdfsStorageOperator.java:191)
at org.apache.dolphinscheduler.server.worker.utils.TaskExecutionContextUtils.downloadResourcesIfNeeded(TaskExecutionContextUtils.java:87)
... 8 common frames omitted`
### What you expected to happen
If a resource file is referenced by a task, it cannot be deleted.
### How to reproduce
1, Create a new shell file "suyc/testsh01.sh" in the Resource Center.
2, Create a shell task that references the resource file.
`bash suyc/testsh01.sh`
3, The shell task runs normally.
4, Delete the file from the Resource Center.
5, Run the shell task again
### Anything else
_No response_
### Version
dev
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [Resource Center] The resource file should not be deleted after it has been referenced by a task. [dolphinscheduler]
Posted by "njnu-seafish (via GitHub)" <gi...@apache.org>.
njnu-seafish commented on issue #17492:
URL: https://github.com/apache/dolphinscheduler/issues/17492#issuecomment-3279330547
> Because the way to store resource files in table `t_ds_task_definition ` is JSON, and indexes can't be used, full table query will lead to a sharp increase in database load in production environment, which will affect the normal task operation.
>
> In the future, if we need to realize this function, we should first split the storage structure of resource files from json.
You're absolutely right — your approach is very well thought out.
Query tasks that use a specific resource file:
>
SELECT
id,
NAME AS task_name,
task_type,
task_params
FROM t_ds_task_definition
WHERE JSON_CONTAINS(task_params -> '$.resourceList',JSON_OBJECT('resourceName', 'hdfs://ctyunns/dolphinscheduler/default/resources/suyc/testsh01.sh')) = 1 LIMIT 1;
Search for tasks referencing files in a given directory:
>
SELECT
id,
NAME AS task_name,
task_type,
task_params
FROM t_ds_task_definition
WHERE JSON_SEARCH(a.task_params -> '$.resourceList', 'one', 'hdfs://ctyunns/dolphinscheduler/default/resources/suyc/%', NULL, '$[*].resourceName') IS NOT NULL LIMIT 1;
The query performance is acceptable under normal task volumes, as long as the number of tasks is not extremely large.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [Resource Center] Resource files and directories should not be deleted or modified once they are referenced by a task. [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17492:
URL: https://github.com/apache/dolphinscheduler/issues/17492#issuecomment-3413348555
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [Resource Center] The resource file should not be deleted after it has been referenced by a task. [dolphinscheduler]
Posted by "njnu-seafish (via GitHub)" <gi...@apache.org>.
njnu-seafish commented on issue #17492:
URL: https://github.com/apache/dolphinscheduler/issues/17492#issuecomment-3273493860
> This is the design, not a bug, because the resource center is a third-party component, and DS does not store the actual file relationship, we can't control the permissions, additions, deletions and modifications of third-party components.
the table t_ds_task_definition has a column task_params：
{"localParams":[],"rawScript":"bash suyc/testsh01.sh","resourceList":[{"resourceName":"hdfs://ctyunns/dolphinscheduler/default/resources/suyc/testsh01.sh"}]}
Can we add a verification check and a warning prompt before deleting a file resource?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [Resource Center] The resource file should not be deleted after it has been referenced by a task. [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17492:
URL: https://github.com/apache/dolphinscheduler/issues/17492#issuecomment-3273516809
Because the way to store resource files in table `t_ds_task_definition ` is JSON, and indexes can't be used, full table query will lead to a sharp increase in database load in production environment, which will affect the normal task operation.
In the future, if we need to realize this function, we should first split the storage structure of resource files from json.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug] [Resource Center] The resource file should not be deleted after it has been referenced by a task. [dolphinscheduler]
Posted by "SbloodyS (via GitHub)" <gi...@apache.org>.
SbloodyS commented on issue #17492:
URL: https://github.com/apache/dolphinscheduler/issues/17492#issuecomment-3272998038
This is the design, not a bug, because the resource center is a third-party component, and DS does not store the actual file relationship, we can't control the permissions, additions, deletions and modifications of third-party components.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [Resource Center] Resource files and directories should not be deleted or modified once they are referenced by a task. [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #17492: [Question] [Resource Center] Resource files and directories should not be deleted or modified once they are referenced by a task.
URL: https://github.com/apache/dolphinscheduler/issues/17492
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Question] [Resource Center] Resource files and directories should not be deleted or modified once they are referenced by a task. [dolphinscheduler]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #17492:
URL: https://github.com/apache/dolphinscheduler/issues/17492#issuecomment-3445333751
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org