You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@dolphinscheduler.apache.org by GitBox <gi...@apache.org> on 2022/12/19 04:54:35 UTC
[GitHub] [dolphinscheduler] Radeity opened a new issue, #13223: [Improvement][Master] Validate same content of input file when using task cache
Radeity opened a new issue, #13223:
URL: https://github.com/apache/dolphinscheduler/issues/13223
### Search before asking
- [X] I had searched in the [issues](https://github.com/apache/dolphinscheduler/issues?q=is%3Aissue) and found no similar feature requirement.
### Description
Task cache mechanism is introduced in https://github.com/apache/dolphinscheduler/pull/13194. I just found that maybe there's something wrong when task has file input parameter. Cache key can make sure tasks have the same value of property (store resource path in file paramter), but the same resource path doesn't mean the same file contents.
Maybe we can add checksum of input file when generating cache key. In addition, i think it's improper to download the whole file in master, we can save checksum in another file, download checksum file when generating cache key.
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] github-actions[bot] commented on issue #13223: [Improvement][Master] Validate same content of input file when using task cache
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #13223:
URL: https://github.com/apache/dolphinscheduler/issues/13223#issuecomment-1357093386
Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
* In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
* If you haven't received a reply for a long time, you can [join our slack](https://s.apache.org/dolphinscheduler-slack) and send your question to channel `#troubleshooting`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou commented on issue #13223: [Improvement][Master] Validate same content of input file when using task cache
Posted by GitBox <gi...@apache.org>.
jieguangzhou commented on issue #13223:
URL: https://github.com/apache/dolphinscheduler/issues/13223#issuecomment-1357130511
> @jieguangzhou PTAL.
The file output using `file parameter` will not change if the user does not manually modify the resource center.
Users should not modify the task output files in the file parameter feature, It's the mechanism inside ds.
But if we use the files parameters from SCP or s3 (If we support them in the future), I think testing is better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] Radeity commented on issue #13223: [Improvement][Master] Validate same content of input file when using task cache
Posted by GitBox <gi...@apache.org>.
Radeity commented on issue #13223:
URL: https://github.com/apache/dolphinscheduler/issues/13223#issuecomment-1357093275
@jieguangzhou PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [dolphinscheduler] jieguangzhou closed issue #13223: [Improvement][Master] Validate same content of input file when using task cache
Posted by GitBox <gi...@apache.org>.
jieguangzhou closed issue #13223: [Improvement][Master] Validate same content of input file when using task cache
URL: https://github.com/apache/dolphinscheduler/issues/13223
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@dolphinscheduler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org